A distributed task data real-time synchronization method and system

Abstract

The application relates to a distributed task data real-time synchronization method and system. The method is applied to an intermediate database based on Flink and comprises the following steps: acquiring data records corresponding to source system transactions; generating corresponding enhanced data sets based on the data records; confirming whether there is a transaction boundary point when the number of data records in the enhanced data sets reaches a maximum value, if yes, splitting the data records in the enhanced data sets based on the transaction boundary point, and sending data records corresponding to the end of the transaction to Kafka, otherwise, sending all the data records in the enhanced data sets to Kafka; acquiring complete transaction data of the data records in Kafka, performing task merging on the complete transaction data to obtain task data; updating the archived data according to the task data, and acquiring change data of the archived data to update a target system. The application can reduce data synchronization delay and guarantee the integrity of transactions in the synchronization process.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and more specifically, to a method and system for real-time synchronization of distributed task data. Background Technology

[0002] Currently, there are two main methods for real-time synchronization of enterprise-level task data. The first method is periodic polling, where the source system writes the generated task data to a database view, and the target system periodically queries the database view for task data and updates its database accordingly. This method has poor real-time performance, long data collection intervals, and the scheduled task executes even when no data has changed, wasting system resources. The second method is application code-level push, which embeds data push logic into the application code. When data changes, the application actively calls the target system's interface or sends a message to a message queue to achieve real-time data push. This method is highly intrusive to the application, increases system complexity, and can negatively impact system performance. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to provide a method and system for real-time synchronization of distributed task data, addressing the aforementioned technical deficiencies of the prior art.

[0004] The technical solution adopted by this invention to solve its technical problem is: to construct a distributed task data real-time synchronization method, applied to an intermediate database based on Flink, the method comprising the following steps: S1. Obtain the data record corresponding to the source system transaction, wherein the data record contains the transaction ID of the source system transaction; S2. During the process of acquiring the data record, a corresponding enhanced data set is generated based on the data record; S3. When the number of data records in the enhanced data set reaches the maximum value, confirm whether there is a transaction boundary point based on the data records in the enhanced data set. If so, proceed to step S4; otherwise, proceed to step S5. S4. Based on the transaction boundary point, split the data records in the enhanced data set to obtain the data records corresponding to each transaction ID, send all data records corresponding to the ended transactions to Kafka, retain the data records corresponding to the unfinished transactions in the enhanced data set, and execute step S6. S5. Send all data records in the enhanced data set to Kafka; S6. Obtain the complete transaction data of the data record in Kafka by consuming messages in Kafka, and merge the complete transaction data to obtain the merged task data; S7. Update the archived data in the intermediate database according to the task data, and obtain the changed data of the archived data, so as to update the target system according to the changed data, thereby realizing data synchronization between the target system and the source system.

[0005] In one embodiment of the distributed task data real-time synchronization method of the present invention, in step S2, the step of generating a corresponding enhanced data set based on the data record during the acquisition of the data record includes: S21. When receiving the data record, determine whether there is a data record in the enhanced data set. If so, proceed to step S22; otherwise, proceed to step S23. S22. Obtain the transaction ID of the data record and the end data record in the enhanced data set, so as to add the transaction ID of the data record as the enhancement tag of the end data record in the end data record; S23. Add the data record to the enhanced data set.

[0006] In one embodiment of the distributed task data real-time synchronization method of the present invention, in step S3, when the number of data records in the enhanced data set reaches the maximum value, confirming whether there is a transaction boundary point based on the data records in the enhanced data set includes: S31. Obtain the transaction ID of the data record in the enhanced data set and the enhancement tag of the data record; S32. When the transaction ID of the data record is different from the enhanced tag of the data record, the data record is taken as the transaction boundary point corresponding to the transaction ID.

[0007] In one embodiment of the distributed task data real-time synchronization method of the present invention, in step S4, the step of splitting the data records in the enhanced data set based on the transaction boundary point to obtain the data records corresponding to each transaction ID includes: S41. Segment the data records in the enhanced data set using the transaction boundary point as the endpoint; S42. For each segment, obtain the data record before the tail endpoint and the data record corresponding to the tail endpoint, and the transaction ID corresponding to the data record, so as to obtain the data record corresponding to the transaction ID. S43. Obtain all data records after the end endpoint and the transaction ID corresponding to the data record, so as to obtain the data record corresponding to the transaction ID.

[0008] In one embodiment of the distributed task data real-time synchronization method of the present invention, in step S6, obtaining the complete transaction data of the data record in Kafka by consuming messages from Kafka includes: S61. Obtain the complete transaction data in the Kafka based on the transaction group; S62. When the Kafka contains data records that have not been grouped by transaction, the data records are segmented using the transaction boundary point as the endpoint. S63. For each segment, obtain the data record before the tail endpoint and the data record corresponding to the tail endpoint, as well as the transaction ID corresponding to the data record, so as to obtain the data record corresponding to the transaction ID as the corresponding complete transaction data.

[0009] In one embodiment of the distributed task data real-time synchronization method of the present invention, it further includes: S11. After acquiring the data record, timing is performed, and if the timing duration exceeds the threshold value and the next data record has not been received, the data record is marked with an end marker and added to the enhanced data set. S12. The data record corresponding to the end marker is taken as a transaction boundary point; S13. Based on the transaction boundary points, split the data records in the enhanced data set to obtain the data records corresponding to each transaction ID respectively; S14. Group all data records corresponding to transaction IDs into transaction groups and send them to Kafka, then execute step S6.

[0010] In one embodiment of the distributed task data real-time synchronization method of the present invention, in step S6, the step of merging the complete transaction data to obtain merged task data includes: Obtain the operation categories corresponding to all data records in the complete transaction data, merge adjacent data records corresponding to the same operation category to obtain the corresponding merged data, and obtain the task data based on all the merged data of the complete transaction data.

[0011] In one embodiment of the distributed task data real-time synchronization method of the present invention, it further includes: When the intermediate database synchronizes with the target system for the first time, it directly synchronizes with the target system based on the archived data in the intermediate database.

[0012] In one embodiment of the distributed task data real-time synchronization method of the present invention, it further includes: Within a preset time period before the task expiration time, first task data corresponding to a preset transaction is obtained, and second task data corresponding to the preset transaction is obtained at the task expiration time. Change data between the first task data and the second task data is obtained, so as to update the target system according to the change data.

[0013] The present invention also provides a distributed task data real-time synchronization system, including a memory and a processor; The memory is used to store computer programs; The processor is used to execute the computer program to implement the method described above.

[0014] The distributed task data real-time synchronization method and system of the present invention have the following beneficial effects: they can reduce data synchronization delay and ensure the integrity of transactions during the synchronization process. Attached Figure Description

[0015] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings: Figure 1 This is a flowchart of an embodiment of a distributed task data real-time synchronization method of the present invention; Figure 2 This is a flowchart of another embodiment of a distributed task data real-time synchronization method of the present invention; Figure 3 This is a flowchart of another embodiment of a distributed task data real-time synchronization method of the present invention; Figure 4 This is a flowchart of another embodiment of a distributed task data real-time synchronization method of the present invention; Figure 5 This is a flowchart of another embodiment of a distributed task data real-time synchronization method of the present invention; Figure 6 This is a flowchart of another embodiment of a distributed task data real-time synchronization method of the present invention; Figure 7 This is a schematic diagram of task merging, representing an embodiment of a distributed task data real-time synchronization method according to the present invention. Figure 8 This is a schematic diagram illustrating the acquisition of change data in an embodiment of a distributed task data real-time synchronization method of the present invention. Detailed Implementation

[0016] To provide a clearer understanding of the technical features, objectives, and effects of the present invention, specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0017] like Figure 1The figure shows an embodiment of a distributed task data real-time synchronization method according to the present invention. Figure 1 In the embodiment of the distributed task data real-time synchronization method of the present invention shown, the method is applied to a Flink-based intermediate database. That is, during data processing, it runs as a Flink job.

[0018] The method includes the following steps: S1, obtaining data records corresponding to the source system transactions, wherein the data records contain the transaction IDs of the source system transactions; S2, during the process of obtaining data records, generating corresponding enhanced data sets based on the data records; S3, when the number of data records in the enhanced data set reaches the maximum value, confirming whether there are transaction boundary points based on the data records in the enhanced data set; if so, proceeding to step S4, otherwise proceeding to step S5; S4, splitting the data records in the enhanced data set based on the transaction boundary points to obtain the data records corresponding to each transaction ID, sending all data records corresponding to completed transactions to Kafka, retaining the data records corresponding to uncompleted transactions in the enhanced data set, and proceeding to step S6; S5, sending all data records in the enhanced data set to Kafka; S6, obtaining the complete transaction data of the data records in Kafka by consuming messages from Kafka, merging the complete transaction data into tasks to obtain merged task data; S7, updating the archived data in the intermediate database based on the task data, and obtaining the changed data of the archived data to update the target system based on the changed data, thereby achieving data synchronization between the target system and the source system.

[0019] Based on step S1, the source system can be understood as a data generation system. The type of source system can include, but is not limited to, systems with MySQL or DM databases. The source system database has an enabled logging function, enabling the intermediate database to acquire data records from the source system database in real time. The data records corresponding to source system transactions can be the log data corresponding to those transactions. It is important to note that during data acquisition, data records must be acquired sequentially according to the data generation order to ensure the orderliness of the task data. This is achieved by setting the parallelism during data acquisition to 1, thus ensuring global orderliness. Because data records are generated by source system transactions, the corresponding data records will contain the transaction IDs corresponding to those source system transactions. Therefore, it can be understood that the source system's log data contains log data corresponding to each transaction ID, and the data records acquired from the log data during the data acquisition process will necessarily also contain the corresponding transaction IDs.

[0020] Based on step S2, Apache Flink, as a stream processing engine, performs event-driven computation record by record, which contradicts the characteristic of a complete transaction in a database, which typically consists of multiple change records. To address this issue, this embodiment employs a Stateful Stream Processing mechanism. During data acquisition, instead of processing each arriving data record in isolation, Flink utilizes its persistent state storage as a highly reliable context buffer to save and backtrack previous data records. That is, during data record acquisition, data is added to the obtained data records to obtain corresponding enhanced data sets.

[0021] Based on step S3, a threshold value is set for the number of data records that the enhanced data set can contain. During the data record acquisition process, when the number of data records in the enhanced data set reaches this threshold (i.e., the maximum value), no more data records can be added to the enhanced data set. At this point, it is necessary to confirm whether there are transaction boundary points in the data records in the enhanced data set. Transaction boundary points are used to distinguish whether data records correspond to transactions from the same source system. When transaction boundary points exist, it means that the data records in the enhanced data set belong to transactions from different source systems. When no transaction boundary points exist, it means that the data records in the enhanced data set belong to transactions from the same source system. It can be understood that each time a data record is added to the enhanced data set, it is confirmed whether the number of data records in the enhanced data set has reached the maximum value. At this point, the data records in the enhanced data set have actually completed data enhancement based on the acquired data records.

[0022] Based on step S4, when a transaction boundary point is determined to exist, the data records in the enhanced data set are split according to the transaction boundary point to obtain the data record corresponding to each transaction ID. In the enhanced data set, only the data record corresponding to the last unfinished transaction is retained, and the data records corresponding to other completed transactions are grouped and sent to Kafka. This process can be understood as sending all data records corresponding to completed source system transactions to Kafka, with each source system transaction's data record being processed in an independent group.

[0023] Because the process of acquiring data records is continuous, the generation of the enhanced data set is also performed in real time. By retaining data records corresponding to unfinished transactions in the enhanced data set, data updates can be performed based on existing data records when acquiring data records corresponding to that transaction ID, ensuring that complete data records of the source system transactions are obtained through the enhanced data set. That is, while executing step S6 after step S4, step S2 will continue to be executed.

[0024] Based on step S5, when it is determined that no transaction boundary point exists, it is equivalent to all data records in the enhanced data set belonging to the same source system transaction, and the complete source system transaction data record cannot be obtained. At this time, all data records in the enhanced data set are sent to Kafka. This process is equivalent to clearing the enhanced data set so that the corresponding enhanced data set can continue to be generated based on the data records. That is, while executing step S6 after step S5, step S2 will continue to be executed.

[0025] Based on step S6, and using transaction groups in Kafka, complete transaction data for data records in Kafka is obtained by consuming messages from Kafka. This complete transaction data is then used for task merging to obtain the task data corresponding to the transaction. This process reduces the amount of task data during transaction processing and improves transaction synchronization speed. The complete transaction data can be understood as all data records corresponding to each transaction. Each transaction can be processed individually, or multiple transactions can be processed simultaneously.

[0026] Based on step S7, after obtaining the task data, the archived data in the intermediate database can be updated based on the task data. The archived data in the intermediate database actually corresponds to the target system. Obtaining the changes in the archived data before and after updating it based on the task data can be understood as the changes that need to be processed in the target system. By responding to the publicly available change data based on the target system, data synchronization between the target system and the source system can be achieved.

[0027] In one embodiment, such as Figure 2 As shown, in step S2, during the process of acquiring data records, a corresponding enhanced data set is generated based on the data records; including: S21, when receiving data records, determining whether there are data records in the enhanced data set; if so, proceeding to step S22; otherwise, proceeding to step S23; S22, acquiring the transaction ID of the data record and the end data record in the enhanced data set, using the transaction ID of the data record added to the end data record as the enhancement marker of the end data record; S23, adding the data record to the enhanced data set.

[0028] Specifically, during the data record acquisition process, each time a data record is received, it is determined whether there is a data record in the enhanced data set. If there is a data record in the enhanced data set, the transaction ID of the current data record is added to the last data record in the enhanced data set, i.e., the end data record. Then, the current data record is added to the enhanced data set and used as a new end data record in the enhanced data set.

[0029] It is understandable that steps S21 to S23 are performed cyclically during the data record acquisition process. That is, during the data record acquisition process, if the previous data record exists, the transaction ID of the current data record is extracted and used as the "next transaction ID" of the previous data record, and dynamically added to the previous data record. At this time, the previous record has been enhanced and can be archived into the group set. To ensure the continuity of the processing flow, after completing the association enhancement of the previous data, the currently received data record is stored in the Flink state storage, making it the "previous data" of the next processing cycle, forming a continuously scrolling processing window.

[0030] In one embodiment, such as Figure 3 As shown, in step S3, when the number of data records in the enhanced data set reaches the maximum value, the existence of a transaction boundary point is confirmed based on the data records in the enhanced data set, including: S31, obtaining the transaction ID and enhancement mark of the data record in the enhanced data set; S32, when the transaction ID and enhancement mark of the data record are different, the data record is used as the transaction boundary point corresponding to the transaction ID.

[0031] Specifically, transaction boundary points of data records can be obtained based on the transaction ID and enhancement markers within the data record. For example, based on the enhancement markers obtained through the above process, if the transaction ID and enhancement marker of a data record are the same, it indicates that the next data record belongs to the same transaction. That is, no transaction boundary point has been obtained. If the transaction ID and enhancement marker of a data record are different, it indicates that the next data record does not belong to the same transaction as the current data record. In this case, it can be understood that the transaction boundary point within the data record has been obtained.

[0032] In one embodiment, such as Figure 4 As shown, in step S4, the data records in the enhanced data set are split based on the transaction boundary points to obtain the data records corresponding to each transaction ID, including: S41, segmenting the data records in the enhanced data set with the transaction boundary points as endpoints; S42, obtaining the data records before and corresponding to the tail endpoint of each segment, as well as the transaction ID corresponding to the data record, to obtain the data record corresponding to the transaction ID; S43, obtaining all data records after the tail endpoint and the transaction ID corresponding to the data record, to obtain the data record corresponding to the transaction ID.

[0033] Specifically, after obtaining the transaction boundary points, transaction splitting can be performed based on these boundary points to obtain complete transaction data records. Based on the transaction boundary points obtained in the above process, data is segmented using the transaction boundary points as endpoints during data splitting. Within each segment, the data records before the tail endpoint and the data record corresponding to that tail endpoint are obtained and combined to obtain the data record corresponding to the transaction ID. This transaction ID can be understood as the transaction ID of all obtained data records. The transaction ID is then used to obtain the transaction ID of any data record within the obtained data records.

[0034] There may be further data records after the endpoint. Therefore, we can combine all the data records after the endpoint to obtain the data record corresponding to the transaction ID. This transaction ID can also be understood as the transaction ID of all the obtained data records. It is obtained by taking the transaction ID of any data record among the obtained data records.

[0035] In one embodiment, such as Figure 5 As shown, in step S6, the complete transaction data of data records in Kafka is obtained by consuming messages from Kafka, including: S61, obtaining the complete transaction data in Kafka based on transaction groups; S62, when Kafka contains data records that have not been grouped by transaction, segmenting the data records with transaction boundary points as endpoints; S63, for each segment, obtaining the data records before the tail endpoint and the data records corresponding to the tail endpoint, as well as the transaction ID corresponding to the data record, so as to obtain the data record corresponding to the transaction ID as the corresponding complete transaction data.

[0036] Specifically, when consuming messages from Kafka, complete transaction data can be obtained. In some scenarios, when sending data to Kafka, the data is grouped into transaction groups, allowing the complete transaction data to be obtained based on these groups. However, in other scenarios, some data records may be sent directly without being grouped. To retrieve these records, transaction splitting based on transaction boundary points is necessary to obtain the complete transaction data. Based on the transaction boundary points obtained above, data is segmented using these boundary points as endpoints. Within each segment, the data records before the tail endpoint and the data record corresponding to that tail endpoint are combined to obtain the data record with the corresponding transaction ID. This transaction ID can be understood as the transaction ID of all the obtained data records. The complete transaction data is then obtained from the transaction ID of any data record within the obtained data records. For data records from which complete transaction data cannot be obtained, they can be retained in Kafka to await the arrival of new data records corresponding to that transaction.

[0037] In one embodiment, such as Figure 6 As shown, the distributed task data real-time synchronization method of the present invention further includes: S11, timing after acquiring data records, and when no next data record is received when the timing duration exceeds a threshold value, adding an end marker to the data record and adding it to the enhanced data set; S12, using the data record corresponding to the end marker as a transaction boundary point; S13, splitting the data records in the enhanced data set based on the transaction boundary point to obtain the data records corresponding to each transaction ID respectively; S14, grouping all data records corresponding to transaction IDs into transaction groups and sending them to Kafka, and executing step S6.

[0038] Specifically, during the data acquisition process, a timer is used to time each data record acquisition. This timer checks if the next data record can be received before the timer reaches its maximum value. If it is received, the data acquisition process continues. If not, the data acquisition process is considered complete. At this point, an end marker is added to the last received data record. This can also be understood as adding an end marker to the last received data record in the augmented dataset when the data record reception is deemed finished. This end marker can also serve as a transaction boundary point to indicate the end of the transaction.

[0039] The data records in the enhanced data set are split based on all transaction boundary points to obtain the data record corresponding to each transaction ID. All data records corresponding to each transaction ID are then grouped by transaction and sent to Kafka. The process of splitting data records based on transaction boundary points can be referred to step S41. Since there are no data records after the end endpoint at this point, it is unnecessary to retain data records in the enhanced data set.

[0040] In one embodiment, in step S6, the complete transaction data is merged to obtain merged task data; this includes: obtaining the operation categories corresponding to all data records in the complete transaction data, merging adjacent data records corresponding to the same operation categories to obtain the corresponding merged data, and obtaining task data based on all merged data of the complete transaction data.

[0041] Specifically, after obtaining complete transaction data, the data records within the complete transaction data are categorized and confirmed for operation. A single consumption operation can contain data from one or more transactions, each transaction containing change records for multiple data tables. The change records of all transactions in the current batch are parsed, using table names as grouping keys. For each table, adjacent log records with the same operation type are merged to reduce subsequent operation batches. When inserting data, the existence of records with the same primary key value in the data table is checked. If a record with the same primary key value exists, it is filtered out to avoid exceptions due to violations of uniqueness constraints. Update and delete operations do not need to consider the existence of duplicate primary key values; they can be implemented directly by calling the jOOQ API. In other words, the obtained task data can correspond to one or more complete transaction data sets. The merging process can be based on a single complete transaction data set or multiple complete transaction data sets.

[0042] like Figure 7 The diagram illustrates how data merging of transaction data yields task data. Specifically, the operation process for a table derived from data records is sequentially A, B, B, C, C, A, A, A… where A, B, and C correspond to different operation categories. Consecutive B operations are merged, consecutive C operations are merged, and consecutive A operations are merged to obtain the corresponding merged data.

[0043] like Figure 8As shown, a specific embodiment of changing data acquisition is presented, illustrating the acquisition of changed data before synchronization of archived data in an intermediate database. In one specific embodiment, a unique identifier is configured for the task data. During data parsing, the key fields that can identify the task object, namely "Task ID" and "Task Owner," are extracted and concatenated into a new field as the identity identifier of the task object. Before the changed data is archived, the archived task data is queried to obtain a data snapshot set C1 before the archived data is synchronized. After the changed data is archived, the archived task data is queried to obtain a data snapshot set C2 after the archived data is synchronized. The difference between the two data snapshot sets C1 and C2 represents the changed task data within this time period. The specific difference comparison process is as follows: The new field formed by concatenating "Task ID" and "Task Owner" can uniquely identify a record. Compare all identity identifiers in the two data snapshot sets; data that exists in data snapshot set C1 but not in data snapshot set C2 is deleted (corresponding to SC3) and marked as "delete"; data that does not exist in data snapshot set C1 but exists in data snapshot set C2 is deleted (corresponding to SC1) and marked as "create"; data that exists in both data snapshot sets C1 and C2 indicates that it has always existed. To determine whether the data content has changed, a hash value needs to be calculated for the data record. If the identity identifier is the same but the hash value is different, it indicates that the data content has changed (corresponding to SC2) and marked as "update".

[0044] In one embodiment, the distributed task data real-time synchronization method of the present invention further includes: when the intermediate database is synchronizing with the target system for the first time, directly synchronizing the target system based on the archived data in the intermediate database. Specifically, during the first synchronization process of the target system, for example, the "first startup identifier" in a preset configuration file is read. If it is true, it indicates the first synchronization. All task data in the intermediate database is read, and all task data is written to the message queue of the target system through threads. At this time, the task data of the source system can be fully synchronized to the intermediate database using a third-party tool such as the DM migration tool.

[0045] The full data synchronization process occurs during the initialization phase, and the amount of full data is very large. The synchronization process takes a lot of time. If the synchronization process is executed in the main thread of initialization, the program will pause for a long time. The pause time exceeds the checkpoint trigger interval and the maximum interval for Kafka to commit offsets. Therefore, the full initialization logic can be executed in a sub-thread, and the main thread does not need to wait for the result of the full data synchronization.

[0046] In one embodiment, the distributed task data real-time synchronization method of the present invention further includes: acquiring first task data corresponding to a preset transaction within a preset time period before the task expiration time, acquiring second task data corresponding to the preset transaction at the task expiration time, acquiring change data between the first task data and the second task data, and updating the target system according to the change data.

[0047] Specifically, in some business systems, tasks are managed by agents, meaning that one person can delegate a task to multiple people, allowing others to approve the task. However, the delegation relationship has a time limit; if the delegation relationship expires, the task to be done by the delegate becomes an expired task.

[0048] Since there is no actual data change when the proxy relationship expires, in order to ensure data consistency, the task data transformation tool customizes a scheduled task that is triggered at 12 o'clock every night to perform two queries. The first query queries the proxy tasks from 30 minutes ago, and the second query queries the proxy tasks at the current time. The result sets of the two queries are compared, and the result set of differences is the expired proxy tasks, which are marked as "delete". These expired tasks are then pushed to the message queue of the target system.

[0049] In one embodiment, exception handling during synchronization can also be implemented through Checkpointing. Specifically, the JobManager's Checkpoint coordinator initiates a Checkpoint (e.g., the Nth Checkpoint). Upon receiving the instruction, the source operation operator injects a Checkpoint Barrier with a unique identifier (ID N) at its current position in the output data stream. The Checkpoint Barrier propagates downstream to various operation operators along with the data stream. When a Checkpoint Barrier is received, processing of the stream records after that Barrier is paused; a snapshot of the operation operator's current internal state (e.g., aggregation results, intermediate variables, etc.) is generated; and after completing its own state snapshot, the Checkpoint Barrier is forwarded to all its downstream operation operators.

[0050] During data processing, the Checkpoint can be restarted three times, with each restart interval being 10 seconds. After a system restart, the Checkpoint coordinator locates and selects the latest successfully completed Checkpoint (e.g., the Nth Checkpoint) from persistent storage as the recovery baseline. All operation operators in the system read the state snapshot associated with the Nth Checkpoint from persistent storage and load it into their respective memory, thereby restoring the consistent state before the failure. Based on the Kafka offset recorded in the Nth Checkpoint, the system instructs the source operation operators to restart reading the data stream from the position corresponding to that offset. This ensures that the system will continue computation after processing all the data corresponding to the Nth Checkpoint, without losing data or reprocessing confirmed data.

[0051] Furthermore, the distributed task data real-time synchronization system of the present invention may further include a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program to implement the above-described method. Specifically, according to embodiments of the present invention, the processes described above with reference to the flowchart can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowchart. In such embodiments, when the computer program is downloaded and installed by an electronic device and executed, it performs the functions defined in the methods of the embodiments of the present invention.

[0052] It is understood that the above embodiments only illustrate preferred embodiments of the present invention, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can freely combine the above technical features without departing from the concept of the present invention, and can also make several modifications and improvements, all of which fall within the protection scope of the present invention. Therefore, all equivalent transformations and modifications made with respect to the scope of the claims of the present invention should fall within the scope of the claims of the present invention.

Claims

1. A method for real-time synchronization of distributed task data, characterized in that, Applied to a Flink-based intermediate database, the method includes the following steps: S1. Obtain the data record corresponding to the source system transaction, wherein the data record contains the transaction ID of the source system transaction; S2. During the process of acquiring the data record, a corresponding enhanced data set is generated based on the data record; S3. When the number of data records in the enhanced data set reaches the maximum value, confirm whether there is a transaction boundary point based on the data records in the enhanced data set. If so, proceed to step S4; otherwise, proceed to step S5. S4. Based on the transaction boundary point, split the data records in the enhanced data set to obtain the data records corresponding to each transaction ID, send all data records corresponding to the ended transactions to Kafka, retain the data records corresponding to the unfinished transactions in the enhanced data set, and execute step S6. S5. Send all data records in the enhanced data set to Kafka; S6. Obtain the complete transaction data of the data record in Kafka by consuming messages in Kafka, and merge the complete transaction data to obtain the merged task data; S7. Update the archived data in the intermediate database according to the task data, and obtain the changed data of the archived data, so as to update the target system according to the changed data, thereby realizing data synchronization between the target system and the source system.

2. The distributed task data real-time synchronization method according to claim 1, characterized in that, In step S2, the step of generating a corresponding enhanced data set based on the data record during the acquisition of the data record includes: S21. When receiving the data record, determine whether there is a data record in the enhanced data set. If so, proceed to step S22; otherwise, proceed to step S23. S22. Obtain the transaction ID of the data record and the end data record in the enhanced data set, so as to add the transaction ID of the data record as the enhancement tag of the end data record in the end data record; S23. Add the data record to the enhanced data set.

3. The distributed task data real-time synchronization method according to claim 2, characterized in that, In step S3, when the number of data records in the enhanced data set reaches its maximum value, the existence of a transaction boundary point is confirmed based on the data records in the enhanced data set, including: S31. Obtain the transaction ID of the data record in the enhanced data set and the enhancement tag of the data record; S32. When the transaction ID of the data record is different from the enhanced tag of the data record, the data record is taken as the transaction boundary point corresponding to the transaction ID.

4. The distributed task data real-time synchronization method according to claim 1, characterized in that, In step S4, splitting the data records in the enhanced data set based on the transaction boundary points to obtain the data records corresponding to each transaction ID includes: S41. Segment the data records in the enhanced data set using the transaction boundary point as the endpoint; S42. For each segment, obtain the data record before the tail endpoint and the data record corresponding to the tail endpoint, and the transaction ID corresponding to the data record, so as to obtain the data record corresponding to the transaction ID. S43. Obtain all data records after the end endpoint and the transaction ID corresponding to the data record, so as to obtain the data record corresponding to the transaction ID.

5. The distributed task data real-time synchronization method according to claim 1, characterized in that, In step S6, obtaining the complete transaction data of the data record in Kafka by consuming messages from Kafka includes: S61. Obtain the complete transaction data in the Kafka based on the transaction group; S62. When the Kafka contains data records that have not been grouped by transaction, the data records are segmented using the transaction boundary point as the endpoint. S63. For each segment, obtain the data record before the tail endpoint and the data record corresponding to the tail endpoint, as well as the transaction ID corresponding to the data record, so as to obtain the data record corresponding to the transaction ID as the corresponding complete transaction data.

6. The distributed task data real-time synchronization method according to claim 1, characterized in that, The method further includes: S11. After acquiring the data record, timing is performed, and if the timing duration exceeds the threshold value and the next data record has not been received, the data record is marked with an end marker and added to the enhanced data set. S12. The data record corresponding to the end marker is taken as a transaction boundary point; S13. Based on the transaction boundary points, split the data records in the enhanced data set to obtain the data records corresponding to each transaction ID respectively; S14. Group all data records corresponding to transaction IDs into transaction groups and send them to Kafka, then execute step S6.

7. The distributed task data real-time synchronization method according to claim 1, characterized in that, In step S6, merging the complete transaction data to obtain merged task data includes: Obtain the operation categories corresponding to all data records in the complete transaction data, merge adjacent data records corresponding to the same operation category to obtain the corresponding merged data, and obtain the task data based on all the merged data of the complete transaction data.

8. The distributed task data real-time synchronization method according to claim 1, characterized in that, The method further includes: When the intermediate database synchronizes with the target system for the first time, it directly synchronizes with the target system based on the archived data in the intermediate database.

9. The distributed task data real-time synchronization method according to claim 1, characterized in that, The method further includes: Within a preset time period before the task expiration time, first task data corresponding to a preset transaction is obtained, and second task data corresponding to the preset transaction is obtained at the task expiration time. Change data between the first task data and the second task data is obtained, so as to update the target system according to the change data.

10. A distributed task data real-time synchronization system, characterized in that, Including memory and processor; The memory is used to store computer programs; The processor is used to execute the computer program to implement the method as described in any one of claims 1 to 9.

Images