Data synchronization method, apparatus, device, and product

Abstract

The application discloses a data synchronization method, device, equipment and product, and relates to the technical field of data processing. The method comprises the following steps: when a database change event is detected, the database change event is sent to a message queue; according to business correlation, the database change event is distributed to a target internal queue through the message queue; the data change event is consumed through the target internal queue, change data after consumption processing is obtained, and the change data is written into a target storage system; the data volume of a source database and the target storage system is compared, a comparison result is obtained, and in the case that the comparison result is that there is a data difference, synchronization is performed according to the data difference, and a data synchronization result is obtained. Thus, the problem that the ordered consumption performance is poor and the data consistency control is difficult in the existing data synchronization scheme is solved, and the efficiency of data synchronization is improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to data synchronization methods, apparatus, devices, and products. Background Technology

[0002] Ledger-type data query systems are responsible for retrieving massive amounts of historical business data. As the scale of enterprise business expands, the amount of data in a single table often exceeds hundreds of millions, and the related dimensions are complex. It is necessary to support multi-parameter compound queries. In order to improve the query response speed, the industry generally adopts Canal to listen to the database change log (binlog), forward it to the business processing module through the Kafka message queue, and finally write the data to Elasticsearch. It uses its inverted index and multi-field aggregation capabilities to achieve efficient multi-condition complex queries.

[0003] However, this solution has revealed significant shortcomings in practical applications. First, to ensure data consistency, Canal requires sequential consumption, but different business logics have varying complexities, and a fixed batch size can easily trigger slow consumption alarms, resulting in severe consumption delays. Second, tens of gigabytes of binlog messages are generated instantaneously and flood the queue, making it impossible for the consumer to distinguish between normal business processes and batch updates. Sequential consumption causes normal messages to be blocked for extended periods. Additionally, Canal is strongly tied to the number of Kafka partitions, making it impossible to improve throughput by expanding partitions. Furthermore, middleware forwarding anomalies or consumer-side logic errors can cause inconsistencies between the source database and Elasticsearch data. Traditional full resynchronization is extremely costly and lacks rapid location methods, resulting in a heavy operational burden.

[0004] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0005] The main purpose of this application is to provide a data synchronization method, apparatus, device, and product, which aims to solve the technical problems of poor orderly consumption performance and difficulty in data consistency management in existing data synchronization schemes.

[0006] To achieve the above objectives, this application proposes a data synchronization method, which includes: When a database change event is detected, the database change event is sent to a message queue; Based on business relevance, the database change event is distributed to the target internal queue via the message queue; The data change event is consumed through the target's internal queue to obtain the processed changed data, and the changed data is written to the target storage system. The data volume of the source database and the target storage system is compared to obtain the comparison result. If the comparison result shows that there is a data difference, synchronization is performed according to the data difference to obtain the data synchronization result.

[0007] In one embodiment, the step of allocating the database change event to a target internal queue through the message queue based on business relevance includes: Extract the business association key from the database change event, and calculate the hash value of the business association key using a hash algorithm; The target queue number is obtained by performing a modulo operation on the total number of several internal queues using the hash value; The database change events are assigned to the target internal queues corresponding to the target queue sequence numbers, so that events with the same business association key enter the same internal queue for sequential consumption, and events with different business association keys enter different internal queues for parallel consumption.

[0008] In one embodiment, the step of consuming the data change event through the target internal queue to obtain the processed changed data, and writing the changed data into the target storage system includes: Data change events are retrieved in batches from the target's internal queue, and consumption tasks are constructed. Monitor the interface response time of the consumption task to obtain the real-time response duration; The number of events to be fetched is adjusted based on the real-time response duration to obtain the adjusted batch size; Based on the adjusted batch size, batch retrieval is performed to obtain optimized consumption tasks, and the optimized consumption tasks are executed to obtain changed data; The changed data is written to the target storage system to obtain the write result, and the consumption offset of the target internal queue is recorded through the offset manager.

[0009] In one embodiment, before the steps of consuming the data change event through the target internal queue to obtain the consumed and processed changed data, and writing the changed data into the target storage system, the method further includes: The number of blocked events in the message queue is collected in real time to obtain blocking detection data; Determine whether the blockage detection data exceeds the blockage threshold; When the blocked detection data exceeds the blocked threshold, it is determined to be a consumption anomaly, and the change event in the message queue is forwarded to the acceleration queue for sequential consumption.

[0010] In one embodiment, the step of forwarding change events in the message queue to the acceleration queue includes: Event frequency data is obtained by counting the frequency of events in the message queue using a sliding time window. Determine whether the event frequency data exceeds the surge threshold; If the event frequency data exceeds the surge threshold, the change event in the message queue is marked as a surge traffic event, and the surge traffic event is forwarded to the acceleration queue for sequential consumption after taking the remainder of the number of partitions according to the original partition number. If the event frequency data does not exceed the surge threshold, then after taking the remainder of the number of partitions in the acceleration queue according to the business association key, the corresponding change event is forwarded to the acceleration queue for sequential consumption.

[0011] In one embodiment, the step of comparing the data volume of the source database and the target storage system to obtain the comparison result includes: The comparison task type is determined based on the comparison trigger timing, and task type information is obtained; The comparison task parameters are constructed based on the task type information, and the comparison task parameters include the tenant identifier and the time range; The source database is queried based on the tenant identifier and time range to obtain a first data volume, and the target storage system is queried based on the tenant identifier and time range to obtain a second data volume; If the first data volume is inconsistent with the second data volume, the comparison result indicates that there is a data difference.

[0012] In one embodiment, the step of synchronizing data based on the data differences to obtain a data synchronization result when the comparison result shows data discrepancies includes: The target tenant range is determined based on the data differences, and comparison parameters are extracted from the target tenant range. The range of values ​​for the comparison parameters is recursively divided using a binary search method to obtain the sub-parameter ranges. Call the business data reconciliation interface to compare the data volume of the sub-parameter range and obtain the specific parameter information of the data difference; A data compensation task is constructed based on the specific parameter information, and the data compensation task is executed to obtain the data synchronization result.

[0013] Furthermore, to achieve the above objectives, this application also proposes a data synchronization device, which includes: The sending module is used to send the database change event to the message queue when a database change event is detected; The allocation module is used to allocate the database change event to the target internal queue through the message queue according to the business relevance; The consumption module is used to consume the data change event through the target's internal queue, obtain the processed changed data, and write the changed data into the target storage system. The synchronization module is used to compare the data volume of the source database and the target storage system to obtain the comparison result. If the comparison result shows that there is a data difference, synchronization is performed according to the data difference to obtain the data synchronization result.

[0014] In addition, to achieve the above objectives, this application also proposes a data synchronization device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the data synchronization method as described above.

[0015] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and which, when executed by a processor, implements the steps of the data synchronization method described above.

[0016] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the data synchronization method described above.

[0017] One or more technical solutions proposed in this application have at least the following technical effects: This application proposes a data synchronization method, apparatus, device, and product. Upon detecting a database change event, the method sends the event to a message queue. Based on business relevance, the message queue allocates the database change event to a target internal queue. The target internal queue consumes the data change event to obtain processed changed data, which is then written to a target storage system. A data volume comparison is performed between the source database and the target storage system to obtain a comparison result. If data differences are found, synchronization is performed based on these differences to obtain a data synchronization result. Therefore, by allocating events to internal queues based on business relevance to balance sequential and parallel consumption, and by automatically detecting and compensating for data differences through data volume comparison between the source database and the target storage system, this method solves the problems of poor ordered consumption performance and difficulty in data consistency management in existing data synchronization schemes, thus improving the efficiency of data synchronization. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating an embodiment of the data synchronization method of this application. Figure 2 This is a schematic diagram illustrating the kinship queue involved in the data synchronization method of this application; Figure 3 This is a schematic diagram illustrating the queue execution involved in the data synchronization method of this application; Figure 4 This is a schematic diagram illustrating the batch consumption adaptive adjustment involved in the data synchronization method of this application; Figure 5 This is a schematic diagram illustrating the data reconciliation involved in the data synchronization method of this application; Figure 6 This is a flowchart illustrating Embodiment 2 of the data synchronization method of this application; Figure 7 This is a schematic diagram illustrating the acceleration queue involved in the data synchronization method of this application; Figure 8 This is a schematic diagram illustrating the forwarding acceleration queue logic involved in the data synchronization method of this application; Figure 9 A simplified flowchart illustrating the data synchronization method provided in Embodiment 2 of this application; Figure 10 This is a schematic diagram illustrating the relationship between response time and batch size in the data synchronization method of this application; Figure 11 This is a schematic diagram of the module structure of the data synchronization device according to an embodiment of this application; Figure 12 This is a schematic diagram of the device structure of the hardware operating environment involved in the data synchronization method in this application embodiment.

[0021] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0022] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0023] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0024] The main solution of this application embodiment is as follows: Extract the business association key from the database change event, and calculate the hash value using a hash algorithm; perform a modulo operation on the total number of several internal queues using the hash value to obtain the target queue number; allocate the database change event to the target internal queue corresponding to the target queue number, so that events with the same business association key enter the same internal queue for sequential consumption, and events with different business association keys enter different internal queues for parallel consumption. Batch pull data change events from the target internal queue and construct consumption tasks; monitor the interface response time of the consumption task to obtain the real-time response duration; adjust the number of events pulled based on the real-time response duration to obtain the adjusted batch size; perform batch pulling according to the adjusted batch size to obtain the optimized consumption task, and execute the optimized consumption task to obtain the changed data; write the changed data to the target storage system to obtain the write result, and record the consumption offset of the target internal queue using an offset manager. The system collects the number of blocked events in the message queue in real time to obtain blocking detection data; it then determines whether the blocking detection data exceeds a blocking threshold; if the blocking detection data exceeds the blocking threshold, it is determined to be a consumption anomaly, and the change events in the message queue are forwarded to the acceleration queue for sequential consumption. The system also uses a sliding time window to count the frequency of events in the message queue to obtain event frequency data; it then determines whether the event frequency data exceeds a surge threshold; if the event frequency data exceeds the surge threshold, the change events in the message queue are marked as surge traffic events, and the surge traffic events are forwarded to the acceleration queue for sequential consumption after taking the remainder of the number of partitions in the acceleration queue based on the original partition sequence number; if the event frequency data does not exceed the surge threshold, the corresponding change events are forwarded to the acceleration queue for sequential consumption after taking the remainder of the number of partitions in the acceleration queue based on the business association key. The comparison task type is determined based on the comparison trigger timing, and task type information is obtained. Comparison task parameters are constructed based on the task type information, including tenant identifier and time range. Data is queried in the source database based on the tenant identifier and time range to obtain a first data volume. Data is then queried in the target storage system based on the tenant identifier and time range to obtain a second data volume. The first data volume and the second data volume are compared. If the first data volume and the second data volume are inconsistent, the comparison result indicates a data difference.The target tenant range is determined based on the data differences, and comparison parameters are extracted from the target tenant range. The range of values ​​for the comparison parameters is recursively split using a binary search method to obtain the sub-parameter ranges. The business data reconciliation interface is called to compare the data volume of the sub-parameter ranges to obtain specific parameter information about the data differences. A data compensation task is constructed based on the specific parameter information and executed to obtain the data synchronization result. This solves the problems of poor ordered consumption performance and difficulty in data consistency management in existing data synchronization schemes, achieving data synchronization and improving data synchronization efficiency. Based on this invention, addressing the real-world problems of the consumer side being unable to distinguish between normal business and batch updates, sequential consumption causing normal messages to be blocked for extended periods, and the strong binding of Canal and Kafka partition numbers preventing throughput improvement through partition expansion, resulting in low efficiency, a data synchronization method is designed. The effectiveness of this data synchronization method is verified during data synchronization, and the efficiency of data synchronization using this method is significantly improved.

[0025] In this embodiment, for ease of description, the data synchronization device will be used as the execution subject in the following description.

[0026] Due to limitations in existing log synchronization technologies based on Canal and Kafka, the efficiency and consistency of data synchronization need improvement. One issue is poor performance in ordered consumption. Different business logics have varying complexity, and fixed batch sizes easily trigger slow consumption alarms, leading to severe consumption delays. Another issue is the blocking problem of batch updates. When tens of gigabytes of binlog messages flood the queue instantaneously, the consumer cannot distinguish between normal business processes and batch updates, causing normal messages to be blocked for extended periods due to sequential consumption. Additionally, Canal and Kafka are strongly bound to each other in terms of partition numbers, making it impossible to improve throughput by expanding partitions. Furthermore, data consistency management is difficult. Middleware forwarding anomalies or consumer-side logic errors can cause inconsistencies between the source and target databases. Traditional full resynchronization is extremely costly and lacks rapid troubleshooting methods, resulting in a heavy operational burden. Therefore, in current large-volume, high-concurrency scenarios such as ledger-type data query systems, system stability and operational efficiency are also challenged. Due to the lack of flexible consumption scheduling mechanisms and automated consistency assurance methods, the overall performance and reliability of data synchronization cannot meet business requirements.

[0027] This application provides a solution that allocates events to internal queues based on business relevance to balance sequential and parallel consumption. By comparing the data volume between the source database and the target storage system, it automatically detects and compensates for data differences, thus solving the problems of poor performance in ordered consumption and difficulty in data consistency management in existing data synchronization solutions and improving the efficiency of data synchronization.

[0028] Based on this, embodiments of this application provide a data synchronization method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the data synchronization method of this application.

[0029] In this embodiment, the data synchronization method includes steps S01 to S04: Step S01: When a database change event is detected, the database change event is sent to a message queue; Before the implementation of this embodiment, it should be clear that in the prior art, when tens of GB of binlog messages are generated instantaneously and flood the queue, the consumer side cannot distinguish between normal business and batch updates. Sequential consumption causes normal messages to be blocked for a long time. At the same time, Canal and Kafka partition numbers are strongly bound, and throughput cannot be improved by expanding partitions. In addition, middleware forwarding anomalies or consumer logic errors will cause inconsistencies between source database and ES data. Traditional full resynchronization is extremely costly and lacks quick location methods, resulting in a heavy maintenance burden.

[0030] Therefore, in order to solve the above problems, this embodiment relies on the Canal component to monitor the binlog of the source database in real time, capturing all data change behaviors such as adding, modifying, and deleting data in the database around the clock. Once a valid data change event is detected, the change event is immediately standardized and encapsulated, unifying the core fields such as event data format, event type, business identifier, and timestamp. Then, the encapsulated complete database change event is pushed to a preset Kafka message queue for temporary storage, realizing real-time collection and asynchronous flow of database change data, providing a reliable data source for subsequent diversion and consumption processing.

[0031] Step S02: Based on business relevance, the database change event is allocated to the target internal queue through the message queue; After receiving all database change events, the message queue no longer adopts a unified, unordered, or global single-queue consumption mode. Instead, it deeply integrates the affinity characteristics of business data, extracts the core business association identifier corresponding to each change event, and uses a hash grouping algorithm to complete the precise distribution of events. Change events with the same business association attributes are grouped into the same internal queue, while differentiated events without business association are distributed to different internal queues. While strictly ensuring the order of changes to single business data, it maximizes the release of the parallel consumption capability of multiple queues, solving the performance bottleneck of traditional ordered consumption from the architectural level.

[0032] Step S03: Consume the data change event through the target internal queue to obtain the processed changed data, and write the changed data into the target storage system; After each internal queue completes event aggregation, the system initiates an adaptive batch consumption mechanism. Based on the event type and business processing complexity within the queue, it dynamically matches the optimal consumption batch, monitors the interface response status of the consumption task in real time, and continuously iterates and optimizes the batch retrieval quantity. It also completes preprocessing operations such as compliance parsing, data cleaning, and field normalization of change events to generate standardized change data.

[0033] Once the compliant data has been processed, it will be written in batches to the Elasticsearch target storage system. At the same time, the consumption offset will be accurately recorded throughout the process to avoid data omissions and duplicate consumption, and to ensure the stability and accuracy of the data writing process.

[0034] Step S04: Compare the data volume of the source database and the target storage system to obtain the comparison result. If the comparison result shows that there is a data difference, synchronize the data according to the data difference to obtain the data synchronization result.

[0035] After completing the routine data synchronization and writing operation, the system starts an automated data consistency verification mechanism. It can automatically adapt to two trigger modes: sudden traffic surge verification and daily scheduled inspection verification, based on the business scenario. With tenant and time interval as the core verification dimensions, it retrieves the full corresponding data from the source database and the target storage system to complete a precise bidirectional data volume comparison.

[0036] Once abnormal differences such as discrepancies, missing values, or redundancy are detected between the two ends of the data, the range of abnormal data is immediately locked. Through refined recursive splitting and precise reconciliation to locate the root cause of the anomaly, a data compensation task is automatically generated and executed, ultimately achieving complete consistency between the source database and the target storage data.

[0037] This allows events to be allocated to internal queues based on business relevance, balancing sequential and parallel consumption. Furthermore, by comparing the data volume between the source database and the target storage system, data discrepancies are automatically detected and compensated, thus solving the problems of poor sequential consumption performance and difficulty in data consistency management in existing data synchronization solutions.

[0038] Specifically, step S02 above, which involves allocating the database change event to the target internal queue through the message queue based on business relevance, includes: Step S021: Extract the business association key from the database change event, and calculate the hash value by using a hash algorithm on the business association key; Step S022: Perform a modulo operation on the total number of several internal queues using the hash value to obtain the target queue number; Step S023: Assign the database change event to the target internal queue corresponding to the target queue sequence number, so that events with the same business association key enter the same internal queue for sequential consumption, and events with different business association keys enter different internal queues for parallel consumption.

[0039] like Figure 2 As shown, Figure 2 The diagram shows a comparison of the architecture before and after optimization of the kinship queue. In the traditional synchronous architecture, all database change events of all businesses are uniformly entered into a single message queue, adopting a first-in-first-out single-order consumption mode. Business events of different complexities block each other. If the consumption of complex business such as document B takes a long time, it will directly block the consumption process of simple and normal business such as document A, causing large-scale consumption delays and system alarms.

[0040] This embodiment first decouples the consumption logic of different businesses through multiple topic queues to avoid mutual interference between data synchronization events of various businesses. On this basis, it is further refined and optimized. Multiple independent kinship-based internal queues are built inside the consumption application. The system prioritizes extracting the core business association keys such as the document number in each database change event. The association keys are encrypted using a standard hash algorithm to obtain a unique hash value. Then, the hash value is moduloed with the preset total number of internal queues to accurately match the corresponding target internal queue number.

[0041] All change events with the same business association key will be aggregated into the same internal queue and consumed in strict order of their occurrence time, preserving the complete change sequence of single business data and preventing business data corruption. Meanwhile, differentiated events with different business association keys will be distributed to different internal queues for synchronous consumption tasks, completely realizing parallel decoupled consumption of unrelated businesses.

[0042] Simultaneously, it is equipped with a Kafka client batch retrieval and manual ACK confirmation mechanism, combined with the Offset manager to record the queue splitting mapping relationship. When a single message fails to be consumed locally, only the corresponding offset data is locked, which will not affect the overall queue consumption and offset commit, thus completely solving the performance bottleneck of traditional single-queue serial consumption.

[0043] More specifically, step S03 above, which involves consuming the data change event through the target's internal queue to obtain the processed changed data and writing the changed data into the target storage system, includes: Step S031: Batch pull data change events from the target's internal queue and construct a consumption task; Step S032: Monitor the interface response time of the consumption task to obtain the real-time response duration; Step S033: Adjust the number of events to be retrieved based on the real-time response duration to obtain the adjusted batch size; Step S034: Perform batch retrieval according to the adjusted batch size to obtain the optimized consumption task, and execute the optimized consumption task to obtain the changed data; Step S035: Write the changed data into the target storage system to obtain the write result, and record the consumption offset of the target internal queue through the offset manager.

[0044] Figure 3 This diagram illustrates the message distribution, offset control, and batch consumption execution of the kinship queue in this embodiment. After the system initializes multiple sets of bounded kinship internal queues, it uses a thread pool mechanism to batch pull data change events to be processed from each target internal queue, integrates and organizes them into standardized consumption tasks, and delivers them to the consumption thread for unified execution.

[0045] For the special case where a single original message is split into multiple internal queues in batch update scenarios, a dedicated Offset Manager will store and bind the original message offset and the association mapping relationship between the sub-messages of each queue after splitting in real time, accurately recording the real-time consumption offset position of each internal queue.

[0046] During the execution of the consumption task, the system parses and processes the change events in the queue one by one, completes preprocessing operations such as data regularization, field validation, and format unification, and then writes the compliant change data into the target storage system.

[0047] After consumption is completed, manual ACK commit is only performed on the offsets corresponding to consecutively successfully consumed messages. If the consumption of a business message corresponding to a certain offset fails, such as the abnormal scenario of message with offset=502 and business key 6 in the figure, the system will lock the offset and not perform the commit operation. It will only retry the abnormal message. The offsets of other normally consumed messages can be committed normally, which effectively avoids the problems of global task blocking, duplicate data consumption or missed consumption caused by local consumption anomalies, and ensures the accuracy and stability of the batch consumption process.

[0048] Figure 4 This is a schematic diagram of the adaptive dynamic adjustment mechanism for batch consumption in this application. In the traditional fixed batch consumption mode, the system uses a uniform batch consumption quantity parameter for all business events, which cannot adapt to the processing complexity of different businesses. Simple new events are processed quickly, but fixed small batches cannot give full play to the system's throughput capacity. Complex update events are processed in a long time, and fixed large batches are prone to triggering interface timeouts, system rate limiting and slow consumption alarms, resulting in frequent message backlog and decreased system stability.

[0049] This embodiment abandons the fixed batch configuration mode and establishes a dynamic optimization mechanism based on the real-time response time of the interface. The system classifies and distinguishes messages according to event type, data table, and changed field dimension, and presets a basic recommended batch size for different business events such as adding new data to the main table and updating fields.

[0050] The entire consumption task operation monitors the interface response time in real time, with 3 seconds as the core threshold standard. When the real-time response time of the interface exceeds 3000ms, it is determined that the current batch is too large and the system load is too high. The number of events pulled in a single batch is automatically reduced, with each adjustment step fixed at 10, to reduce the pressure on a single round of consumption tasks and avoid timeout alarms. When the interface response time is less than 500ms, it is determined that the system load is sufficient and the processing capacity is adequate. The consumption batch size is automatically increased, also in increments of 10, to maximize the data consumption throughput.

[0051] The system records the tuning parameters and execution results of various message types in memory throughout the process, and continuously iterates and optimizes the optimal batch parameters to adapt to various business scenarios. This enables personalized and adaptive tuning of batch consumption under different business scenarios, and completely solves the performance imbalance problem caused by fixed batch parameters.

[0052] Further, step S04 above, which involves comparing the data volume of the source database and the target storage system to obtain the comparison result, includes: Step S041: Determine the comparison task type based on the comparison trigger timing and obtain task type information; Step S042: Query the source database according to the tenant identifier and time range to obtain a first data volume; query the target storage system according to the tenant identifier and time range to obtain a second data volume. Step S043: Compare the first data volume with the second data volume. If the first data volume and the second data volume are inconsistent, the comparison result is that there is a data difference.

[0053] In conventional incremental synchronization scenarios, data synchronization based on Canal events can theoretically guarantee data consistency between the source and target ends. However, extreme cases such as abnormal consumption logic, message loss in the message queue, or inconsistencies in consumption progress between the original and accelerated queues after a sudden surge in traffic and subsequent cessation of forwarding can still lead to data discrepancies between the two ends. Therefore, the system is equipped with a fully automated data reconciliation and compensation mechanism. The system's data comparison process can adapt to two types of core business scenarios, generating different types of comparison tasks accordingly: special reconciliation tasks for sudden traffic surges and daily scheduled inspection reconciliation tasks. The system can automatically determine the task type and generate matching task parameters based on the real-time business triggering timing.

[0054] Among them, the surge traffic reconciliation is adapted to the scenario of batch message surge and the final verification after the accelerated queue forwarding and consumption. It automatically collects the scope of all tenants covered by the current surge traffic, accurately extracts the corresponding tenant identifier and traffic effective time interval as core comparison parameters, and daily inspection and reconciliation is a normal operation and maintenance mechanism. Reconciliation tasks are generated at fixed intervals, and the default time range is selected such as the past month. All business tenant identifiers are bound in batches to complete the parameter construction.

[0055] After the parameters are configured, the system starts bidirectional data query tasks in parallel. Based on the unified tenant identifier and time range parameters, it initiates precise data statistical queries to the source database and the Elasticsearch target storage system respectively, and obtains the first data volume of the source database and the second data volume of the target storage system respectively.

[0056] The system performs precise comparisons of the two sets of data on a tenant-by-tenant and time-by-time basis. If the values ​​of the two sets of data are completely consistent, it is determined that the data in this round of synchronization is completely consistent and there are no anomalies. If there are deviations, missing or redundant values, it is immediately determined that there are data synchronization differences in the current tenant and the current time interval, the scope of the anomaly is locked and the anomaly-related information is persistently stored to provide data support for subsequent accurate positioning and data compensation.

[0057] Furthermore, step S04 above, in the case that the comparison result shows data differences, involves synchronizing based on the data differences to obtain a data synchronization result, including: Step S044: Determine the target tenant range based on the data differences, and extract comparison parameters based on the target tenant range; Step S045: The range of values ​​for the comparison parameters is recursively split using a binary search method to obtain the sub-parameter ranges. Step S046: Call the business data reconciliation interface to compare the data volume of the sub-parameter range and obtain the specific parameter information of the data difference; Step S047: Construct a data compensation task based on the specific parameter information, and execute the data compensation task to obtain the data synchronization result.

[0058] Figure 5 This is a schematic diagram of the data reconciliation and accurate compensation process for this application. After the system identifies the target tenant and corresponding time range with data differences through two-way data comparison, it accurately extracts all comparison parameters under the abnormal scenario. Relying on the binary recursive splitting mechanism, the parameter value range is refined layer by layer. The large-scale, coarse-grained abnormal interval is continuously split into multiple small-grained, accurately verifiable sub-parameter ranges, which greatly narrows the scope of abnormal data screening.

[0059] After the splitting is completed, the system repeatedly calls the business-specific data reconciliation interface to conduct a detailed review and comparison of the data volume corresponding to each sub-parameter range. It continuously iterates the splitting and comparison until the smallest abnormal parameter range with the number of target data rows less than the preset threshold is found. Finally, it accurately locks the specific parameter information and root cause range of abnormal problems such as missing data, data redundancy, and data disorder.

[0060] After completing the anomaly location, the system automatically and structurally constructs a dedicated data compensation task based on the accurately identified anomaly parameters, tenant range, and time interval, avoiding the high resource consumption problem of traditional full data resynchronization.

[0061] Once the compensation task is generated, it is automatically executed to specifically repair abnormal data, fill in missing data, correct erroneous data, and clean up redundant data. Currently, the binary search method supports range parameters including enumeration parameters and date parameters. In the future, more types of parameters such as value range can be expanded according to business needs, demonstrating good architectural scalability.

[0062] After compensation is completed, a second reconciliation verification is triggered to ensure that the data in the source database and the target storage system are completely consistent, and finally the entire process of data synchronization and repair is completed, achieving accurate data compensation with zero human intervention.

[0063] This embodiment, through the above-described scheme, specifically sends a database change event to a message queue when a database change event is detected; based on business relevance, the message queue allocates the database change event to a target internal queue; the target internal queue consumes the data change event to obtain processed changed data, and writes the changed data to a target storage system; the data volume of the source database and the target storage system is compared to obtain a comparison result; if the comparison result shows data differences, synchronization is performed based on the data differences to obtain a data synchronization result. Therefore, by allocating events to internal queues based on business relevance to balance sequential and parallel consumption, and by automatically detecting and compensating for data differences by comparing the data volume of the source database and the target storage system, the poor performance of ordered consumption and the difficulty in data consistency management in existing data synchronization schemes are solved, thus improving the efficiency of data synchronization.

[0064] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to the first embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 6 In step S03, the data synchronization method further includes steps S0301 to S0303: (This refers to consuming the data change event through the target's internal queue to obtain the processed changed data and writing the changed data into the target storage system.) Step S0301: Collect the number of blocked events in the message queue in real time to obtain blocking detection data; Step S0302: Determine whether the blockage detection data exceeds the blockage threshold; Step S0303: When the blocking detection data exceeds the blocking threshold, it is determined to be a consumption anomaly, and the change event in the message queue is forwarded to the acceleration queue for sequential consumption.

[0065] Figure 7 This is a schematic diagram of the automated fault handling architecture for the accelerated queue in this application. During operation, the monitoring module collects core monitoring data such as the number of real-time event blocking, consumption delay, and message backlog in the message queue 24 / 7, continuously summarizes and generates standardized blocking detection data, and perceives the queue consumption operation status in real time.

[0066] The decision module has a built-in dynamic and static dual threshold management mechanism. It compares the monitoring data with the preset blocking threshold in real time. When the number of queue blockages or the backlog exceeds the threshold standard, it immediately determines that the current queue has a consumption abnormality or message blocking failure and triggers the accelerated queue scheduling mechanism.

[0067] The execution module then initiates message forwarding logic, forwarding batches of pending change events in the original blocked queue to the preset accelerated standby queue. The dedicated consumer in the accelerated queue then undertakes the consumption task, maintaining the original business sequence of the change events throughout the process and ensuring sequential consumption.

[0068] Meanwhile, the system has intelligent degradation and automatic recovery capabilities. When it detects severe consumption anomalies or when the system is on the verge of overload, it automatically triggers a function degradation strategy, retaining only the core capabilities of basic MySQL conditional queries to ensure that the basic operation of the business is not affected. After the queue blockage is cleared and consumption returns to normal, the system automatically closes the degradation strategy, stops message forwarding, and restores the normal queue consumption mode. No manual operation and maintenance intervention is required throughout the process, which completely solves the problems of normal business blockage and consumption delay caused by a surge in batch messages. At the same time, it breaks through the technical limitations of traditional Canal and Kafka partition strong binding that cannot be scaled up and improved.

[0069] Specifically, step S0303 above, which involves forwarding the change event in the message queue to the acceleration queue, includes: Step S03031: Count the frequency of events in the message queue by using a sliding time window to obtain event frequency data; Step S03032: Determine whether the event frequency data exceeds the surge threshold; Step S03033: If the event frequency data exceeds the surge threshold, the change event in the message queue is marked as a surge traffic event, and the surge traffic event is forwarded to the acceleration queue for sequential consumption after taking the remainder of the number of partitions according to the original partition number. Step S03034: If the event frequency data does not exceed the surge threshold, then after taking the remainder of the number of partitions in the acceleration queue according to the business association key, the corresponding change event is forwarded to the acceleration queue for sequential consumption.

[0070] Figure 8 This is a schematic diagram illustrating the execution logic of the dual forwarding strategy for the accelerated queue in this application. The system relies on a sliding time window algorithm to continuously count the frequency of events occurring in the message queue within a unit of time, outputting event frequency data in real time to accurately distinguish between regular business traffic and sudden batch traffic. The system presets a traffic surge threshold and continuously compares the real-time frequency data with the threshold standard to form two sets of differentiated forwarding logic: (1) When the frequency data of events exceeds the surge threshold, it is determined that the current surge traffic scenario is caused by batch updates. The corresponding batch change events are immediately marked as surge traffic events. In order to avoid disrupting the original time sequence of batch data, the distribution is no longer based on the business association key. Instead, the original partition number of the message is extracted, the remainder is calculated on the total number of partitions in the acceleration queue, and the surge traffic events are forwarded to the corresponding partitions in the acceleration queue in an orderly manner according to the original partition time sequence. The surge traffic is accurately isolated and prevented from blocking the normal business message consumption of the original queue.

[0071] (2) When the event frequency data does not reach the surge threshold, it is determined to be a regular queue blockage scenario. There is no need to isolate the traffic. Based solely on the business affinity rule, the event business association key is extracted and the remainder of the number of accelerated queue partitions is taken. Regular blocked messages are then forwarded to the accelerated queue for consumption in an orderly manner. The overall consumption capacity is improved by expanding the consumption partitions. Among them, Hash forwarding does not separately identify surge traffic. Batch update messages and normal business messages are forwarded to the accelerated queue in sequence. There is still a certain risk of blockage when the total message volume is too large. It is suitable for regular continuous queue blockage scenarios. Surge forwarding can accurately identify and isolate surge traffic only, ensuring the real-time consumption of normal business messages in the original queue to the greatest extent. It is suitable for scenarios with high sensitivity to business latency. The two strategies are precisely adapted to different fault scenarios, solving the regular queue blockage problem and specifically isolating sudden batch traffic, ensuring the real-time and orderly consumption of business in all aspects.

[0072] The system is equipped with a traffic splitting and dual-queue parallel consumption mechanism, and is configured with a debouncing strategy to avoid frequent start-stop of the splitting logic due to traffic fluctuations. Under normal operation, all database change events are received by the original queue and then distributed to the related internal queues for consumption processing. The acceleration queue is in a reserved standby state, and the Offset Manager submits the consumption offset of the original queue at a normal pace.

[0073] When the number of blocked events collected by the monitoring module exceeds the preset blocking threshold in multiple consecutive detection cycles, the system matches and identifies message features based on the statistical results of the sliding time window, and determines high-frequency events with the same data table, the same change event type, and the same set of change fields as sudden increase feature messages.

[0074] After identification, the system starts the diversion and forwarding logic. The original queue continues to receive and consume normal business messages that do not suddenly increase, maintaining the real-time consumption of normal business. Only the identified sudden increase characteristic messages are diverted to the acceleration queue according to the corresponding forwarding strategy. The acceleration queue independently completes the sequential consumption. The two queues process different types of messages in parallel without interfering with each other, thereby improving the overall consumption throughput and alleviating the blocking pressure on the original queue.

[0075] During the traffic splitting process, the original queue's offset manager independently submits the consumption offset for regular messages; the accelerated queue maintains its own consumption offset separately, and submits it uniformly after all the sudden surge messages in the corresponding batch have been consumed. The two types of offsets are managed independently to avoid duplicate or missing data consumption.

[0076] When the queue blockage is relieved and the frequency of sudden surge messages remains below the recovery threshold for a set duration, the system stops diverting new surge events to the acceleration queue. After all existing messages in the acceleration queue have been consumed and the corresponding offsets have been committed, the acceleration queue returns to standby mode. The original queue continues to consume messages normally without interruption, achieving a smooth and seamless transition between the dual-queue mode.

[0077] This embodiment, through the above-described scheme, specifically obtains blocking detection data by real-time collection of the number of blocked events in the message queue; determines whether the blocking detection data exceeds a blocking threshold; when the blocking detection data exceeds the blocking threshold, it is determined to be a consumption anomaly, and the changed events in the message queue are forwarded to an acceleration queue for sequential consumption. Thus, by allocating events to internal queues based on business relevance to balance sequential and parallel consumption, and by automatically detecting and compensating for data differences by comparing the data volume between the source database and the target storage system, the poor performance of ordered consumption and the difficulty in data consistency management in existing data synchronization schemes are solved, thereby improving the efficiency of data synchronization.

[0078] For example, to help understand the implementation flow of the data synchronization method obtained by combining this embodiment with the above embodiment one, please refer to... Figure 9 , Figure 9 A simplified flowchart of a data synchronization method is provided, specifically: The overall operation is divided into five core components: data collection, intelligent traffic distribution, adaptive consumption, anomaly self-healing, and consistency verification and compensation, all of which are automated and interconnected.

[0079] First, the Canal component is used to monitor the binlog logs of the source database in real time, collect all data change events, and push them to the message queue in a standardized manner to complete the accurate collection of data sources.

[0080] Subsequently, relying on the business affinity hash grouping mechanism, change events are intelligently diverted to the corresponding internal queues, achieving efficient distribution that is orderly within the same business and parallel across businesses.

[0081] After the queue receives an event, it starts an adaptive batch consumption mechanism, dynamically adjusts the consumption batch according to the interface response time, completes data preprocessing and target storage writing, and precisely controls the consumption position through the Offset Manager.

[0082] During operation, the system monitors queue congestion and traffic status in real time, identifies abnormal consumption and traffic surge scenarios, automatically triggers a self-healing mechanism to accelerate queue forwarding, and adapts to dual forwarding strategies to resolve consumption congestion issues.

[0083] Finally, the system routinely performs two-way data reconciliation, distinguishing between sudden surge final verification and routine inspection verification scenarios. It accurately locates data discrepancies through binary search and automatically executes data compensation tasks, ultimately achieving high-performance, high-consistency, and high-self-healing fully automated data synchronization. This completely solves the core pain points of traditional solutions, such as poor consumption performance, severe congestion, and difficulty in data consistency management.

[0084] Figure 10 The graph shows the relationship between the response time and batch size of the batch consumption adaptive adjustment mechanism in this application. In the actual operation of the system, the batch size and the interface response time show a dynamic negative correlation. The system continuously iterates and optimizes with a fixed adjustment step size of 10, using 3000ms as the upper limit threshold for response time and 500ms as the baseline threshold for efficient response.

[0085] When business processing is complex and interface pressure is high, excessively large single consumption batches will cause interface response time to increase. Once the 3000ms limit is exceeded, the system will automatically reduce the number of batches to reduce the data processing volume of a single consumption task, quickly compress the interface response time, avoid timeout alarms and system rate limiting issues, and ensure system stability.

[0086] When business processing is simple and system load is low, the interface response time is stable within 500ms. The system determines that resources are sufficient, automatically increases the number of consumption batches, increases the amount of data processed per unit time, and maximizes the release of system throughput capacity.

[0087] As can be seen from the curve trend, after adaptive dynamic optimization, the interface response time of the system in most scenarios can be stably controlled within the 3-second compliance range, with only a few small instantaneous fluctuations. At the same time, the batch size is dynamically adapted to the business load, perfectly achieving a dynamic balance between system throughput and operational stability. Compared with the traditional fixed batch consumption mode, it significantly improves the overall operational efficiency and fault tolerance of data synchronization.

[0088] In actual business environment testing, four message distribution instances with 1 core and 1.5G memory and four Kafka instances configured with four partitions were used to conduct a comparative test on the synchronization performance of 20,000 document change messages.

[0089] Test results show that the consumption rate of this solution is stable in the initial stage of traffic. After detecting a sudden increase in messages, the accelerated queue forwarding mechanism is automatically triggered. The original queue consumption logic is switched to direct forwarding mode, and all the sudden increase in traffic is transferred to the accelerated queue for sequential consumption, avoiding blocking the normal incremental messages in the original queue.

[0090] Ultimately, the peak consumption rate of the original queue can reach 1.5MB / s, which significantly improves the real-time consumption performance compared to traditional solutions. Combined with the dynamic batch consumption capability of the acceleration queue, the overall consumption throughput is significantly improved.

[0091] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the data synchronization method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0092] This application also provides a data synchronization device, please refer to... Figure 11 The data synchronization device includes: The sending module 10 is used to send the database change event to the message queue when a database change event is detected; The allocation module 20 is used to allocate the database change event to the target internal queue through the message queue according to the business relevance; The consumption module 30 is used to consume the data change event through the target internal queue, obtain the changed data after consumption processing, and write the changed data into the target storage system; The synchronization module 40 is used to compare the data volume of the source database and the target storage system to obtain the comparison result. If the comparison result shows that there is a data difference, synchronization is performed according to the data difference to obtain the data synchronization result.

[0093] The data synchronization device provided in this application, employing the data synchronization method described in the above embodiments, can solve the technical problems of poor ordered consumption performance and difficulty in data consistency management in existing data synchronization schemes. Compared with the prior art, the beneficial effects of the data synchronization device provided in this application are the same as those of the data synchronization method provided in the above embodiments, and other technical features in the data synchronization device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0094] This application provides a data synchronization device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the data synchronization method in Embodiment 1 above.

[0095] The following is for reference. Figure 12 The diagram illustrates a structural schematic of a data synchronization device suitable for implementing embodiments of this application. The data synchronization device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 12 The data synchronization device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0096] like Figure 12As shown, the data synchronization device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in the read-only memory 1002 or a program loaded from the storage device 1003 into the random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the data synchronization device. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the data synchronization device to communicate wirelessly or wiredly with other devices to exchange data. Although the figures show data synchronization devices with various systems, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0097] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0098] The data synchronization device provided in this application, employing the data synchronization method described in the above embodiments, can solve the technical problems of poor ordered consumption performance and difficulty in data consistency management in existing data synchronization schemes. Compared with the prior art, the beneficial effects of the data synchronization device provided in this application are the same as those of the data synchronization method provided in the above embodiments, and other technical features of this data synchronization device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0099] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0100] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0101] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the data synchronization method described in the above embodiments.

[0102] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0103] The aforementioned computer-readable storage medium may be included in the data synchronization device; or it may exist independently and not be assembled into the data synchronization device.

[0104] The aforementioned computer-readable storage medium carries one or more programs. When these programs are executed by a data synchronization device, the data synchronization device: upon detecting a database change event, sends the database change event to a message queue; based on business relevance, allocates the database change event to a target internal queue via the message queue; consumes the data change event via the target internal queue to obtain processed changed data, and writes the changed data to a target storage system; compares the data volume of the source database and the target storage system to obtain a comparison result; if the comparison result indicates a data difference, synchronizes the data based on the data difference to obtain a data synchronization result.

[0105] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0106] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0107] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0108] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described data synchronization method. This solves the technical problems of poor ordered consumption performance and difficulty in data consistency control in existing data synchronization schemes. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the data synchronization method provided in the above embodiments, and will not be repeated here.

[0109] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the data synchronization method described above.

[0110] The computer program product provided in this application can solve the technical problems of poor ordered consumption performance and difficulty in data consistency management in existing data synchronization schemes. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the data synchronization method provided in the above embodiments, and will not be repeated here.

[0111] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A data synchronization method, characterized in that, The data synchronization method includes: When a database change event is detected, the database change event is sent to a message queue; Based on business relevance, the database change event is distributed to the target internal queue via the message queue; The data change event is consumed through the target's internal queue to obtain the processed changed data, and the changed data is written to the target storage system. The data volume of the source database and the target storage system is compared to obtain the comparison result. If the comparison result shows that there is a data difference, synchronization is performed according to the data difference to obtain the data synchronization result.

2. The data synchronization method as described in claim 1, characterized in that, The step of allocating the database change event to the target internal queue through the message queue based on business relevance includes: Extract the business association key from the database change event, and calculate the hash value of the business association key using a hash algorithm; The target queue number is obtained by performing a modulo operation on the total number of several internal queues using the hash value; The database change events are assigned to the target internal queues corresponding to the target queue sequence numbers, so that events with the same business association key enter the same internal queue for sequential consumption, and events with different business association keys enter different internal queues for parallel consumption.

3. The data synchronization method as described in claim 1, characterized in that, The step of consuming the data change event through the target internal queue to obtain the processed changed data, and writing the changed data into the target storage system includes: Data change events are retrieved in batches from the target's internal queue, and consumption tasks are constructed. Monitor the interface response time of the consumption task to obtain the real-time response duration; The number of events to be fetched is adjusted based on the real-time response duration to obtain the adjusted batch size; Based on the adjusted batch size, batch retrieval is performed to obtain optimized consumption tasks, and the optimized consumption tasks are executed to obtain changed data; The changed data is written to the target storage system to obtain the write result, and the consumption offset of the target internal queue is recorded through the offset manager.

4. The data synchronization method as described in claim 1, characterized in that, Before the step of consuming the data change event through the target internal queue to obtain the processed changed data and writing the changed data into the target storage system, the method further includes: The number of blocked events in the message queue is collected in real time to obtain blocking detection data; Determine whether the blockage detection data exceeds the blockage threshold; When the blocked detection data exceeds the blocked threshold, it is determined to be a consumption anomaly, and the change event in the message queue is forwarded to the acceleration queue for sequential consumption.

5. The data synchronization method as described in claim 4, characterized in that, The step of forwarding the change event in the message queue to the acceleration queue includes: Event frequency data is obtained by counting the frequency of events in the message queue using a sliding time window. Determine whether the event frequency data exceeds the surge threshold; If the event frequency data exceeds the surge threshold, the change event in the message queue is marked as a surge traffic event, and the surge traffic event is forwarded to the acceleration queue for sequential consumption after taking the remainder of the number of partitions according to the original partition number. If the event frequency data does not exceed the surge threshold, then after taking the remainder of the number of partitions in the acceleration queue according to the business association key, the corresponding change event is forwarded to the acceleration queue for sequential consumption.

6. The data synchronization method as described in claim 1, characterized in that, The step of comparing the data volume of the source database and the target storage system to obtain the comparison result includes: The comparison task type is determined based on the comparison trigger timing, and task type information is obtained; The comparison task parameters are constructed based on the task type information, and the comparison task parameters include the tenant identifier and the time range; The source database is queried based on the tenant identifier and time range to obtain a first data volume, and the target storage system is queried based on the tenant identifier and time range to obtain a second data volume; If the first data volume is inconsistent with the second data volume, the comparison result indicates that there is a data difference.

7. The data synchronization method as described in claim 1, characterized in that, The step of synchronizing data based on the data differences when the comparison result shows a data difference, and obtaining the data synchronization result, includes: The target tenant range is determined based on the data differences, and comparison parameters are extracted from the target tenant range. The range of values ​​for the comparison parameters is recursively divided using a binary search method to obtain the sub-parameter ranges. Call the business data reconciliation interface to compare the data volume of the sub-parameter range and obtain the specific parameter information of the data difference; A data compensation task is constructed based on the specific parameter information, and the data compensation task is executed to obtain the data synchronization result.

8. A data synchronization device, characterized in that, The data synchronization device includes: The sending module is used to send the database change event to the message queue when a database change event is detected; The allocation module is used to allocate the database change event to the target internal queue through the message queue according to the business relevance; The consumption module is used to consume the data change event through the target's internal queue, obtain the processed changed data, and write the changed data into the target storage system. The synchronization module is used to compare the data volume of the source database and the target storage system to obtain the comparison result. If the comparison result shows that there is a data difference, synchronization is performed according to the data difference to obtain the data synchronization result.

9. A data synchronization device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the data synchronization method as described in any one of claims 1 to 7.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the data synchronization method as described in any one of claims 1 to 7.

Images