Heterogeneous transaction system data near-real-time aggregation method and system

Abstract

The invention relates to the technical field of transaction data aggregation, and discloses a near-real-time aggregation method and system for heterogeneous transaction system data, and the method comprises the steps: obtaining real-time change data from a plurality of heterogeneous transaction systems through a change data capturing technology, and standardizing the change data into a uniform format; performing Hash routing calculation on the standardized change data based on the order number to generate a Hash value, and routing the change data of the same order to the same message queue according to the Hash value; an independent consumer thread is allocated to each message queue through a multi-queue parallel processing mechanism, and the number of the consumer threads is dynamically adjusted according to the message accumulation amount; according to the method, the processed change data is aggregated according to the preset core field template, the aggregated data is stored in the aggregation database, cross-system unified query and analysis are supported, and the problems that heterogeneous data query is difficult, order affiliation judgment is complex, cross-system data statistical analysis is obstructed, and data synchronization timeliness is not high are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of transaction data aggregation, and particularly to a method and system for nearly real-time aggregation of data in heterogeneous transaction systems. Background Art

[0002] Currently, with the rapid development of fintech, both software development technology and hardware storage technology are developing and iterating rapidly, which shortens the iteration cycle of transaction systems. During the system upgrade and iteration process, it is normal for new and old systems to run in parallel. However, this parallel architecture brings the following problems: First, transaction systems of different generations often adopt different data structure designs and storage media (such as Mysql, Oracle, PostgreSQL, etc.), resulting in inconsistent query interfaces and methods, and it is impossible to obtain all data through a single query method. Second, in a multi-parallel system environment, determining which transaction system a specific order belongs to requires traversing all systems for query. This "broadcast" query mode is not only inefficient but also increases the load pressure on each system. Third, due to data isolation between systems, it is impossible to directly perform cross-system data aggregation and analysis, making it difficult to form a global view required for business decisions, especially prominent in scenarios that require real-time or near-real-time data analysis. In addition, traditional relational databases highly rely on preset indexes to improve query efficiency. However, in a large-volume transaction system, due to performance and storage cost considerations, not many indexes are created, resulting in very low query efficiency for non-index fields. At the same time, even some existing data aggregation solutions either use a timed batch synchronization method, resulting in poor data timeliness, or use a real-time synchronization method but affect the performance of the main system due to the lack of an effective parallel processing mechanism.

[0003] Existing technologies can improve the processing efficiency of transaction data through algorithm models or improve the analysis ability of multi-source heterogeneous data through consistency analysis, but neither solves the problem of aggregation of heterogeneous data sources or optimization of message queues. Therefore, there is an urgent need for a method and system for real-time aggregation of heterogeneous data to fundamentally improve the synchronization processing, query, and analysis capabilities of heterogeneous data. Summary of the Invention

[0004] The purpose of the present invention is to solve the above-mentioned drawbacks existing in the prior art, and provides a method and system for nearly real-time aggregation of data in heterogeneous transaction systems. By designing a unified data aggregation module, obtaining changed data based on CDC technology, and designing an "order hash routing + multi-queue parallel processing" mechanism, it is possible to nearly real-time aggregate the transaction data of multiple heterogeneous transaction systems, and solve problems such as difficult query of heterogeneous data, complex determination of order attribution, obstacles to cross-system data statistical analysis, and low timeliness of data synchronization.

[0005] On the one hand, a method for nearly real-time aggregation of heterogeneous transaction system data is provided, including the following steps: S1: Obtain real-time change data from multiple heterogeneous transaction systems through change data capture technology, and standardize the change data into a unified format; S2: Perform hash routing calculation on the standardized change data based on the order number to generate a hash value, and route the change data of the same order to the same message queue according to the hash value; S3: Allocate independent consumer threads for each message queue through a multi-queue parallel processing mechanism, set a consumer thread scheduling mechanism, and dynamically adjust the number of consumer threads according to the message backlog; S4: Convert the processed change data according to a preset core field template, and store the converted data in an aggregation database to support unified query and analysis across systems.

[0006] Further, in step S1, the standardizing the change data into a unified format includes: Determine the data medium of heterogeneous transaction system data that needs to be processed according to the business; For each heterogeneous transaction system, use the corresponding change data capture technology to capture the transaction data with data changes; Process the change data obtained by different change data capture technologies into a unified standard JSON data structure, retaining the core information while avoiding complex nested structures.

[0007] Further, in step S2, routing the change data of the same order to the same message queue according to the hash value includes: S21: Perform hash calculation on the order number using the MurmurHash3 algorithm to obtain an initial hash value, and enhance the dispersion of the initial hash value through high-low bit mixing operation to obtain a new hash value; S22: Take the modulus of the new hash value with the number of message queues, determine the target message queue number according to the modulus result, and send the change data to the specified message queue to complete the hash routing based on the order number.

[0008] Further, in step S3, the setting of the consumer thread scheduling mechanism includes: S31: Set the minimum number of threads and the maximum number of threads , where the maximum number of threads is set based on the number of consumers during the business peak period and is less than or equal to the number of message queues; S32: Divide into four ladder levels L1 to L4 equally, where L1 is the minimum number of threads threads, and L4 is the maximum number of threads threads; S33: Determine whether to trigger stepped expansion or contraction based on the message backlog, and set a cooling period for expansion and contraction to achieve dynamic adjustment of the number of consumer threads.

[0009] Preferably, the stepped expansion includes: According to the consumption improvement rate of each level of expansion from L1 to L4 , the expected processing time of the backlogged messages and the safety expansion coefficient calculate the expansion threshold , and the formula is as follows: wherein, the consumption improvement rate of each level of expansion , represents the consumption rate of a single thread; Starting from L1, when the message backlog quantity reaches the expansion threshold , expand to the next level in the order of the ladder levels; If the messages continue to pile up and reach the expansion threshold when expanding to the next level, continue to expand to the next level.

[0010] More preferably, the contraction includes: If the message backlog is zero or lower than the preset threshold, and the message processing capacity of the previous level meets the current message new rate coverage, and the continuous stable low load reaches the preset duration, then contract to the previous level according to the ladder levels.

[0011] Further, in step S4, aggregating the processed change data according to the preset core field template further includes: Perform templatization processing on the processed change data, convert the data of different data sources according to the defined core field template, and use the document - type storage structure to retain the original non - core fields of each system. The core field template includes order number, transaction time, user id, transaction price, quantity, payment amount, business scenario, and transaction source; Write to the aggregation database ElasticSearch, and perform corresponding operations on the indexes in the aggregation database ElasticSearch according to the operation type of the change data to achieve the aggregation of heterogeneous data source data.

[0012] Further, the method further includes historical data processing, including: When initializing the aggregation system or adding a heterogeneous trading system, import historical data into the aggregation system through batch push and merge and store it with the real - time change data.

[0013] Preferably, the method further includes a monitoring and fault - tolerance mechanism, which further includes: When the data synchronization delay time exceeds the threshold and the consumer threads reach the maximum number for automatic expansion but still cannot process the piled-up messages, an alarm is triggered and manual intervention is notified. The synchronized data is verified through a timed task verification method. The aggregation system actively initiates a data query to the source system, processes the source data according to the aggregation template, compares the data consistency between the source system and the aggregation system, and makes corrections based on the source system data.

[0014] On the other hand, a heterogeneous transaction system data near-real-time aggregation system is also provided, including: A data capture and standardization module, which is used to obtain real-time change data from multiple heterogeneous transaction systems through change data capture technology and standardize the change data into a unified format; A hash routing and message distribution module, which is used to perform hash routing calculation on the standardized change data based on the order number, generate a hash value, and route the change data of the same order to the same message queue according to the hash value; A dynamic resource management module, which is used to allocate independent consumer threads for each message queue through a multi-queue parallel processing mechanism, set a consumer thread scheduling mechanism, and dynamically adjust the number of consumer threads according to the message accumulation volume; A data aggregation and storage module, which is used to convert the processed change data according to a preset core field template and store the converted data in an aggregation database to support unified query and analysis across systems.

[0015] In addition, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the heterogeneous transaction system data near-real-time aggregation method described in any one of the above is implemented.

[0016] Meanwhile, an electronic device is provided, including: one or more processors; a storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the heterogeneous transaction system data near-real-time aggregation method described in any one of the above.

[0017] Compared with the prior art, the beneficial effects of the present invention are: (1) The present invention integrates heterogeneous data sources by standardizing the JSON format, defines a core field template, shields the field and storage differences of different databases, provides a single query interface, reduces the complexity of cross-system data access, simplifies the aggregation logic through the standardization of core fields, reduces the data conversion time-consuming, and uses a document-type storage structure to retain the unique business fields of each system; (2) Through order hash routing and multi-queue parallel processing, the present invention routes the change data of the same order to the same queue, avoiding out-of-order updates of the order status, ensuring data consistency, and enhancing the dispersion of hash values through a combination of high and low bit mixed operations to achieve uniform distribution of the message queue; (3) Through stepwise expansion (L1-L4 levels) and safe contraction strategies, the present invention dynamically adjusts the number of consumer threads, solves the accumulation and latency during peak business periods and the thread idleness during low business periods, avoids unnecessary resource waste, and at the same time sets a safety factor and a cooling period to avoid system oscillations caused by frequent expansion and contraction; (4) The present invention introduces historical data processing and fault tolerance mechanisms. By seamlessly connecting historical data with real-time change data, it ensures a global view of business analysis, and at the same time automatically repairs data differences through verification tasks; (5) The present invention shortens the average repair time through delay threshold alarms and thread full-load alarms. If dynamic expansion fails, manual intervention is carried out to improve system availability. Brief Description of the Drawings

[0018] The drawings are used to provide a further understanding of the present invention, and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation to the present invention. In the drawings: Figure 1 is a flowchart of a method for nearly real-time aggregation of heterogeneous trading system data according to the present invention; Figure 2 is a schematic structural diagram of a system for nearly real-time aggregation of heterogeneous trading system data according to the present invention; Figure 3 is a schematic diagram of a hash routing algorithm according to the present invention; Figure 4 is a schematic diagram of the dynamic adjustment process of consumer threads according to the present invention. Detailed Embodiments

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Apparently, the described embodiments are some, but not all, of the embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts shall fall within the protection scope of the present application.

[0020] Through the unified design of the data aggregation layer, the present invention standardizes these core fields. Meanwhile, by adopting a document-based storage structure to retain the unique business fields of each system, the unity of the query method is achieved. At the same time, aiming at the problem of insufficient efficiency of the traditional single-threaded consumption model of the message queue, this solution innovatively designs a "order hash routing + multi-queue parallel processing" mechanism: by performing a hash calculation on the order number, it is ensured that all relevant messages of the same order are routed to the same message queue. While ensuring the sequential processing of single-order data, high-concurrency processing between different orders is achieved, significantly improving the data aggregation efficiency. On this basis, according to the number of message backlogs, the consumer thread number level is dynamically adjusted to reduce the synchronization delay of data aggregation during peak business hours and reduce resource waste during off-peak business hours.

[0021] The following will illustrate the specific implementation manners of the present invention in conjunction with the accompanying drawings and embodiments.

[0022] Embodiment 1 Please refer to Figure 1 , a method for near-real-time aggregation of heterogeneous transaction system data provided in this embodiment, the technical solution includes the following steps: S1: Obtain real-time change data from multiple heterogeneous transaction systems through change data capture technology, and standardize the change data into a unified format; S2: Perform a hash routing calculation on the standardized change data based on the order number to generate a hash value, and route the change data of the same order to the same message queue according to the hash value; S3: Allocate independent consumer threads to each message queue through a multi-queue parallel processing mechanism, set a consumer thread scheduling mechanism, and dynamically adjust the number of consumer threads according to the message backlog; S4: Convert the processed change data according to a preset core field template, and store the converted data in an aggregation database to support unified query and analysis across systems.

[0023] The specific data structure framework is as Figure 2 shown.

[0024] Among them, in step S1, the further standardizing the change data into a unified format further includes: Determine the data media of heterogeneous transaction system data that needs to be processed according to the business; For each heterogeneous transaction system, use the corresponding change data capture technology to capture the transaction data with data changes; Process the change data obtained by different change data capture technologies into a unified standard JSON data structure, retaining the core information while avoiding complex nested structures.

[0025] This part includes a CDC data capture layer and a change data processing layer respectively, see Figure 2 the relevant part. Specifically, in this embodiment, first, according to the business, determine the heterogeneous transaction system data media that need to be synchronized (such as the MySQL database storing transaction system A and the Oracle database storing transaction system B), and then use the change data capture technology of their respective databases to capture transaction data with data changes such as addition, deletion, and modification. Since the change data formats obtained by different change data capture technologies are not unified, it is necessary to first process the change data into a unified standard change data format. This embodiment designs a standard JSON data structure for formatting. The key information is as follows: sourceType: The source system, indicating that the data source is system A, system B, etc. dataBaseType: The database type, indicating that the data source is MySQL, Oracle, etc. eventTime: The data change time, represented by a timestamp dataBase: The database name table: The table name operationType: The operation type: INSERT, UPDATE, DELETE before: The JSON format of the data record before the change after: The JSON format of the data record after the change orderId: The order number Unify the formatting of the change data from different systems, retain the core information while avoiding complex nested structures, making subsequent processing and parsing simple and intuitive.

[0026] Next, in order to improve the processing speed of the change data, it is necessary to send the change data to a group of message queues. We implement it through the hash routing algorithm and message distribution of the order number in the basic step S2, including: S21: Use the MurmurHash3 algorithm to perform a hash calculation on the order number to obtain an initial hash value, and enhance the dispersion of the initial hash value through high and low bit mixing operations to obtain a new hash value; S22: Use the new hash value to take the modulus of the number of message queues, determine the target message queue number according to the modulus result, and send the change data to the specified message queue to complete the hash routing based on the order number.

[0027] Specifically, data disorder needs to be prevented during this process, that is, we need to ensure that the change data of the same order number is in order. Due to the FIFO (First In First Out) characteristic of the message queue, here we only need to let the change data of the same order enter the same message queue. And to ensure that the order numbers can be evenly distributed into the message queues, a hash algorithm with high dispersion is required. In this embodiment, the MurmurHash3 hash algorithm is used to calculate the order number to obtain a 32-bit hash value.

[0028] In this embodiment, we do not directly perform a modulo operation on the number of message queues using the hash value of the order number. Because the number of message queues is limited, the high bits of the 32-bit hash value may not necessarily participate in the modulo operation. To enhance the randomness of the low bits of the hash value, a mixed calculation of the high and low bits of the hash value is performed. , obtaining a new hash value. Finally, use the new hash value to perform a modulo operation on the number of message queues. The result after modulo is the number of the corresponding message queue, target_queue = hash_value % queue_count, and the change data is sent to the specified message queue, completing the hash routing based on the order number.

[0029] As Figure 3 shown, extract the order number: 250321MK003KGF323, calculate the hash value through the hash algorithm, hashfunction(250321MK003KGF323)=3634066 n 360, and the binary representation is: 11011000100110110111001110111000. Then perform a mixed operation of the high and low bits. Get: 1101 10001000111011010100011000 (original value hash) 0000 000000000000110110001000 0111 (shifted right hash>>>16) =1101100010001111011001010011111 (result) It can be seen that the result value is 3634177695. Take the modulo of this hash value with the number of message queues, 3634177695%8 = 7, that is, send the message to queue 7.

[0030] When it is monitored that there is a new message stored in the message queue, the consumer thread will pull and process the data. Based on the requirement for the order of order change data, a message queue can only be consumed by one consumer. At the same time, there are generally peaks and valleys in transaction services. The business peaks are from 9:00 to 11:00, 14:00 to 16:00, and 20:00 to 22:00 in a day, and the middle of the night is the business valley. If the same number of consumer threads is always maintained, when the number of consumer threads is set low, messages will continuously pile up during the business peak period, resulting in a longer delay in data synchronization; when the number of consumer threads is set too high, there will be many idle threads during the business low peak period, causing unnecessary resource waste.

[0031] Therefore, as described in step S3, we set up a consumer thread scheduler to control the number of consumers, including: S31: Set the minimum number of threads and the maximum number of threads , where the maximum number of threads is set based on the number of consumers during the business peak period and is less than or equal to the number of message queues; S32: Divide into four ladder levels L1 to L4 equally, where L1 is the minimum number of threads threads, and L4 is the maximum number of threads threads; S33: Judge to trigger stepped expansion or contraction according to the message backlog volume, and set the cooling period for expansion and contraction to dynamically adjust the number of consumer threads.

[0032] Among them, the stepped expansion includes: According to the consumption improvement rate of each level of expansion from L1 to L4 , the expected processing time of the backlogged messages and the safety expansion coefficient calculate the expansion threshold , and the formula is as follows: Among them, the consumption improvement rate of each level of expansion , represents the consumption rate of a single thread; Starting from L1, when the number of backlogged messages reaches the expansion threshold , expand to the next level in the order of ladder levels; If the messages still continue to pile up and reach the expansion threshold when expanding to the next level, continue to expand to the next level.

[0033] The contraction includes: If the message accumulation volume is zero or below the preset threshold, and the message processing capacity of the upper level meets the current message addition rate, and it continues to be stably under low load for the preset duration, then scale down to the previous level according to the ladder level.

[0034] Specifically, as Figure 4 shown, in this embodiment, first, it is necessary to clarify the minimum required number of consumer threads , and this part of the consumers will be created at the beginning and will never go offline. It is also necessary to clarify the maximum required number of consumer threads , and this number needs to meet the number of consumers during the business peak period with a certain redundancy, and cannot exceed the number of message queues. Then, set four ladder levels in the system. In this embodiment, the four levels are configured as follows: Level L1: threads (minimum configuration) Level L2: threads (about 33% expansion) Level L3: threads (about 66% expansion) Level L4: threads (maximum configuration) Then, estimate the improvement in consumer speed brought by adding one more consumer messages / s. Then, it can be known that the improvement in consumption speed from L1 to L2 is messages / s; set the expected processing time , indicating how long it is hoped to process the accumulated messages; set the safety expansion coefficient a. For example, 0.8 means starting to expand when reaching 80% of the theoretical processing limit, leaving a margin for the system. Then, when the message accumulation quantity reaches , start the expansion operation from L1 to L2. The same is true for the expansion operations from L2 to L3 and from L3 to L4. When expanding to the next level and the messages still continue to accumulate and reach the expansion threshold, then it is necessary to continue the expansion operation.

[0035] The scaling-down operation needs to be more conservative. When the following conditions are met simultaneously, start scaling down: The accumulation volume is zero or below the preset threshold; The message processing capacity of the upper level can meet the current message addition rate. Here, the safety scaling-down coefficient needs to be used. For example, when the processing rate of level L1 is greater than the current message addition rate, this condition is met; Continue to be under low load and for a certain period of time .

[0036] When controlling the number of dynamic threads, it is necessary to avoid frequent scaling operations. In this embodiment, a cooling period of 3 minutes is set after each scaling operation, and no scaling operations are triggered during this period.

[0037] After the aggregator system consumer obtains the changed data, it is necessary to perform templatization processing of step S4 on the after data (changed data) of each trading system, including: Perform templatization processing on the processed changed data. Data from different data sources is converted according to the defined core field template, and non-core fields remain unchanged. The core field template includes order number, transaction time, user ID, transaction price, quantity, payment amount, business scenario, and transaction source; Write to the aggregation database ElasticSearch, and perform corresponding operations on the indexes in the aggregation database ElasticSearch according to the operation type of the changed data to achieve the aggregation of data from heterogeneous data sources.

[0038] In addition, when the aggregator system is launched for the first time or a new heterogeneous trading system is added, since the CDC technology obtains real-time changed data and cannot obtain historical data, it is necessary for the source system to directly push the historical data to the aggregator system for storage. The historical data processing further includes: When the aggregator system is initialized or a new heterogeneous trading system is added, import the historical data into the aggregator system through batch push and merge it with the real-time changed data for storage.

[0039] Furthermore, it also includes a monitoring and fault tolerance mechanism, specifically including: Since the changed data structure has the eventTime data change time, if the data synchronization delay time exceeds the threshold and the consumer threads reach the maximum number of automatic scaling but still cannot process the accumulated messages, trigger an alarm and notify manual intervention; Verify the synchronized data through a timed task verification method. The aggregator system actively initiates a data query to the source system, processes the source data according to the aggregation template, compares the data consistency between the source system and the aggregator system, and corrects it based on the source system data. If data is missing, supplement the data.

[0040] Based on the above method, this embodiment also provides a near-real-time aggregation system for heterogeneous trading system data, including: A data capture and standardization module, which is used to obtain real-time changed data from multiple heterogeneous trading systems through change data capture technology and standardize the changed data into a unified format; A hash routing and message distribution module, which is used to perform hash routing calculation on the standardized changed data based on the order number, generate a hash value, and route the changed data of the same order to the same message queue according to the hash value; A dynamic resource management module, which is used to allocate independent consumer threads for each message queue through a multi-queue parallel processing mechanism, set a consumer thread scheduling mechanism, and dynamically adjust the number of consumer threads according to the message backlog volume; A data aggregation and storage module, which is used to convert the processed changed data according to a preset core field template, and store the converted data in an aggregation database to support unified query and analysis across systems.

[0041] Among them, the functional implementation of each module in the system corresponds to each step in the embodiment of the method for nearly real-time aggregation of heterogeneous transaction system data in this embodiment, and its functions and implementation processes will not be elaborated here one by one.

[0042] Finally, it should be noted that the above are only the preferred embodiments of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions falling within the idea of the present invention belong to the protection scope of the present invention. It should be pointed out that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements should also be regarded as the protection scope of the present invention.

[0043] The technical features of the above embodiments can be combined arbitrarily. For the sake of concise description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

Claims

1. A method for nearly real-time aggregation of heterogeneous trading system data, characterized in that, It includes the following steps: S1: Obtain real-time change data from multiple heterogeneous transaction systems through change data capture technology, and standardize the change data into a unified format; S2: Perform hash routing calculation on the standardized change data based on the order number to generate a hash value, and route the change data of the same order to the same message queue according to the hash value; S3: Allocate independent consumer threads for each message queue through a multi-queue parallel processing mechanism, set a consumer thread scheduling mechanism, and dynamically adjust the number of consumer threads according to the message backlog; S4: Convert the processed change data according to a preset core field template, and store the converted data in an aggregation database to support unified query and analysis across systems.

2. The method for near real-time aggregation of heterogeneous trading system data according to claim 1, wherein In step S1, further standardizing the change data into a unified format includes: Determine the data media of heterogeneous transaction systems that need to be aggregated according to the business; For each heterogeneous transaction system, use the corresponding change data capture technology to capture the transaction data with data changes; Process the change data obtained by different change data capture technologies into a unified standard JSON data structure, retaining the core information while avoiding complex nested structures.

3. The method for near real-time aggregation of heterogeneous trading system data according to claim 1, wherein, In step S2, further routing the change data of the same order to the same message queue according to the hash value includes: S21: Use the MurmurHash3 algorithm to perform hash calculation on the order number to obtain an initial hash value, and enhance the dispersion of the initial hash value through high-low bit mixing operation to obtain a new hash value; S22: Take the modulus of the new hash value by the number of message queues, determine the target message queue number according to the modulus result, and send the change data to the specified message queue to complete the hash routing based on the order number.

4. The method for near real-time aggregation of heterogeneous trading system data according to claim 1, wherein In step S3, further setting the consumer thread scheduling mechanism includes: S31: Set the minimum number of threads and the maximum number of threads , where the maximum number of threads is set based on the number of consumers during the business peak period and is less than or equal to the number of message queues; S32: Divide equally into four ladder levels L1 to L4, where L1 is the minimum number of threads threads, and L4 is the maximum number of threads threads; S33: Judge whether to trigger stepped expansion or contraction according to the message backlog, and set an expansion and contraction cooling period to dynamically adjust the number of consumer threads.

5. The method for near real-time aggregation of heterogeneous trading system data according to claim 4, wherein The stepped expansion includes: Consumption improvement rate for each-level expansion from L1 to L4 and expected processing time of piled-up messages as well as safety expansion factor to calculate the expansion threshold , the formula is as follows: Among them, the consumption improvement rate for each level of expansion , represents the consumption rate of a single thread; Starting from L1, when the message accumulation quantity reaches the expansion threshold it expands to the next level in the order of the ladder level; If the message still continues to pile up and reaches the expansion threshold when expanding to the next level, continue to expand to the next level.

6. The method for near real-time aggregation of heterogeneous trading system data according to claim 4, wherein The contraction includes: If the message backlog is zero or lower than the preset threshold, and the message processing capacity of the previous level meets the current message addition rate, and the continuous stable low load reaches the preset duration, then contract to the previous level according to the stepped level.

7. The method for near real-time aggregation of heterogeneous trading system data according to claim 1, wherein In step S4, further converting the processed change data according to a preset core field template includes: Perform templatization processing on the processed change data, convert the data from different data sources according to the defined core field template, and use a document-based storage structure to retain the original non-core fields of each system. The core field template includes order number, transaction time, user id, transaction price, quantity, payment amount, business scenario, and transaction source; Write to the aggregation database ElasticSearch, and perform corresponding operations on the indexes in the aggregation database ElasticSearch according to the operation type of the change data to achieve the aggregation of data from heterogeneous data sources.

8. The method for near real-time aggregation of heterogeneous trading system data according to claim 1, wherein The method further includes historical data processing, which further includes: When the aggregation system is initialized or a new heterogeneous trading system is added, import historical data into the aggregation system by means of batch push, and merge and store it with real-time change data.

9. The method for near real-time aggregation of heterogeneous transaction system data according to claim 8, wherein The method further includes a monitoring and fault tolerance mechanism, which further includes: If the data synchronization delay time exceeds the threshold and the consumer threads still cannot process the piled-up messages when reaching the maximum number of automatic expansion, trigger an alarm and notify manual intervention; Verify the synchronized data through a timed task verification method. The aggregation system actively initiates a data query to the source system, processes the source data according to the aggregation template, compares the data consistency between the source system and the aggregation system, and corrects it based on the source system data.

10. A data near-real-time aggregation system for a heterogeneous trading system, characterized in that, It includes: A data capture and standardization module, configured to obtain real-time change data from multiple heterogeneous trading systems through change data capture technology, and standardize the change data into a unified format; A hash routing and message distribution module, configured to perform hash routing calculation on the standardized change data based on the order number to generate a hash value, and route the change data of the same order to the same message queue according to the hash value; A dynamic resource management module, configured to allocate independent consumer threads for each message queue through a multi-queue parallel processing mechanism, set a consumer thread scheduling mechanism, and dynamically adjust the number of consumer threads according to the message backlog; A data aggregation and storage module, configured to convert the processed change data according to a preset core field template, and store the converted data in an aggregation database to support unified query and analysis across systems.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by a processor, it implements the method for near-real-time aggregation of heterogeneous trading system data as described in any one of claims 1-9.

12. An electronic device, characterized in that, It includes: One or more processors; A storage device for storing one or more programs, which when executed by the one or more processors, cause the one or more processors to implement the method for near-real-time aggregation of heterogeneous trading system data as described in any one of claims 1-9.

Images