Multi-data-source data synchronization method, system, device, storage medium and product

Abstract

The application discloses a multi-data-source data synchronization method, system, device, storage medium and product, relates to the technical field of data synchronization, and comprises the following steps: in response to a write request of a source data source, determining whether double writing needs to be performed according to a double writing switch state and a double writing condition, and generating a double writing signal if the double writing needs to be performed; recording a double writing guarantee record containing a unique serial number and an initial state in a guarantee data table of the source data source; sending the double writing signal for double writing processing of a target data source client; receiving a double writing processing result confirmation message returned by the target data source client, wherein the confirmation message contains the unique serial number and a processing state; and updating the state of the corresponding record in the guarantee data table according to the confirmation message. Therefore, the data synchronization consistency and integrity in a high-concurrency scenario are guaranteed, the performance influence on a business system is reduced or avoided, and the problem that data inconsistency cannot be automatically supplemented and timely corrected when data is synchronized between multiple data sources is solved.

Description

Technical Field

[0001] This application relates to the field of data synchronization technology, and in particular to a method, system, device, storage medium and product for synchronizing data from multiple data sources. Background Technology

[0002] In multi-data source scenarios, data synchronization between different data sources is typically required to ensure data consistency and integrity. Currently, common synchronization methods include asynchronous dual-write, message queue-based data replication, and log parsing-based synchronization tools. Among these, asynchronous dual-write is widely adopted due to its relatively small impact on business performance; it involves asynchronously sending write requests to the target data source while simultaneously performing write operations on the source data source. Furthermore, some solutions also introduce data verification mechanisms or offset-based replication methods to maximize data consistency across multiple data sources.

[0003] However, existing technologies still have significant shortcomings in handling data synchronization in high-concurrency scenarios. On the one hand, traditional asynchronous dual-write cannot guarantee that the order in which the target data source receives operations is consistent with the actual order in which they occur in the source data source. When the same data is added, updated, or deleted sequentially, network latency and other factors may cause the target data source to receive the data out of order, leading to data loss or incorrect status. On the other hand, to improve consistency, some solutions require sacrificing system performance (e.g., frequent offset comparisons or locking), making it difficult to balance high performance and strong consistency. Furthermore, when data loss or errors occur during synchronization, existing solutions often lack automatic data replenishment and error correction capabilities, requiring manual intervention for investigation and repair, resulting in high maintenance costs and untimely responses. Summary of the Invention

[0004] The main purpose of this application is to provide a method, system, device, storage medium and product for synchronizing data from multiple data sources, which aims to ensure the consistency and integrity of data synchronization in high-concurrency scenarios and reduce or avoid the performance impact on business systems.

[0005] To achieve the above objectives, this application proposes a multi-data source data synchronization method, applied to the source data source client, the method comprising: In response to a write request to the source data source, determine whether to perform a dual write based on the dual write switch status and dual write conditions. If necessary, generate a dual write signal. Record a double-write protection record containing a unique sequence number and initial state in the protection data table of the source data source; Send a double-write signal to the target data source client to perform double-write processing; Receive a confirmation message of the double-write processing result returned by the target data source client, the confirmation message containing the unique sequence number and processing status; Based on the confirmation message, update the status of the corresponding record in the protection data table.

[0006] In one embodiment, the step of responding to a write request to the source data source, determining whether a dual write operation needs to be performed based on the dual write switch state and dual write conditions, and generating a dual write signal if necessary, includes: In response to a write request to the source data source, read the dual-write switch configuration parameters and determine whether the dual-write switch is enabled. If the dual-write switch is enabled, the dual-write object to be synchronized is obtained through the dual-write aspect; The dual-write objects are filtered according to preset dual-write conditions. If the dual-write objects meet the dual-write conditions, a dual-write signal is generated. The dual-write signal includes operation type, data primary key, data content, timestamp or version number.

[0007] In one embodiment, sending a dual-write signal to enable the target data source client to perform dual-write processing includes: According to the sending timing configured in the dual-write aspect, the dual-write signal is sent to the message middleware or directly to the target data source client so that the target data source client can perform dual-write processing.

[0008] In one embodiment, the sending timing includes sending immediately or delaying until after the source data source transaction is committed. The step of sending the dual-write signal to the message middleware or directly to the target data source client according to the sending timing configured in the dual-write aspect, so that the target data source client can perform dual-write processing, includes: Based on the transmission timing configured in the dual-write section, determine whether to immediately send the dual-write signal; If the configured sending timing is to send immediately, the dual-write signal will be pushed to the message middleware or sent to the target data source client. If the configured sending timing is to delay until the source data source transaction is committed, the double-write signal is temporarily stored in the local queue and a transaction synchronization callback is registered. After the source data source transaction is successfully committed, the double-write signal is retrieved from the local queue and sent. If the source data source transaction is rolled back, the double write signal in the local queue is discarded.

[0009] In one embodiment, before sending the dual-write signal for the target data source client to perform dual-write processing, the method further includes: The dual-write aspect can be configured through annotations or configuration files. The configuration items include: the expression for the dual-write switch, the SpEL extraction expression for the dual-write object, the Boolean expression for dual-write condition filtering, and the enumeration value for the timing of sending.

[0010] In one embodiment, the protection data table is located in the source data source, and the fields of the protection data table include: serial number, primary key, double-write content summary, creation timestamp, status, failure reason, and number of retries.

[0011] In one embodiment, updating the status of the corresponding record in the protection data table based on the confirmation message includes: If the confirmation message indicates successful processing, the status of the corresponding record in the security data table will be updated to "processed". If the confirmation message indicates a processing failure, the number of retries is incremented, and an alarm is triggered when the number of retries exceeds a preset threshold.

[0012] To achieve the above objectives, this application also proposes a multi-data source data synchronization method, applied to a target data source client, the method comprising: Receive a dual-write signal from the source data source client. The dual-write signal is generated by the source data source client in response to a write request to the source data source, based on the dual-write switch status and dual-write conditions. Perform dual-write processing based on the dual-write signal; A confirmation message for the dual-write processing result is returned to the source data source client. The confirmation message includes the unique sequence number and processing status. The source data source client updates the status of the corresponding record in the guarantee data table of the source data source according to the confirmation message. Before sending the dual-write signal, the source data source client records a dual-write guarantee record in the guarantee data table that includes a unique sequence number and an initial status.

[0013] In one embodiment, the dual-write processing based on the dual-write signal includes: The operation type of the dual write signal is analyzed, and the operation type includes write operation, update operation, or delete operation; Write, update, or delete operations are executed according to the operation type, and dual-write synchronization is performed using a delete record table and a delay processing record table. The delete record table is used to store the primary key of the data that has been successfully deleted and the deletion timestamp. The delay processing record table is used to temporarily store update or delete operations that cannot be executed because the data to be operated on does not exist in the target data source.

[0014] In one embodiment, the dual-write signal includes an operation type, a data primary key, data content, a timestamp, or a version number. The step of performing a write operation, update operation, or delete operation according to the operation type, and using a delete record table and a delay processing record table for dual-write synchronization, includes: If the operation type is a write operation, then query whether there is a deleted record with the same primary key in the local deleted record table of the target data source; if not, then perform the write operation; if it exists, then compare the write timestamp in the double write signal with the deletion timestamp in the deleted record. If the write timestamp is later than the deletion timestamp, then perform the write operation; otherwise, discard the write signal. If the operation type is an update operation, then query whether there is a data record in the target data source that has the same primary key as the data; if not, then store this update operation in the local delayed processing record table; if it exists, then compare the version number or timestamp in the double write signal with the version number or timestamp of the existing record in the target data source, wherein, when the version number or timestamp in the double write signal is updated, optimistic locking is used to perform the update, otherwise the update is abandoned; If the operation type is a deletion operation, then query whether there is a data record in the target data source that has the same data primary key; if not, then store this deletion operation in the delayed processing record table; if it exists, then compare the timestamp in the double write signal with the write timestamp of the record in the target data source, and perform deletion when the deletion timestamp is later than the write timestamp, and insert a record containing the data primary key and the deletion timestamp into the deletion record table after successful deletion.

[0015] In one embodiment, the double-write processing based on the double-write signal further includes at least one of the following: Initiate the first background cycle task, scan the delayed processing record table, and compensate for any unprocessed update or deletion operations. Initiate the second background cycle task to scan the protection data table in the source data source and perform compensatory double writes for double write protection records that have timed out and have not been confirmed. The received double-write signal is recorded in the verification table, and the third timed task is started to perform consistency verification and automatic error correction between the source data source and the target data source at the top of the hour.

[0016] In one embodiment, the step of initiating a first background cycle task, scanning a delayed processing record table, and compensating for any unprocessed update or deletion operations includes: Start the first background cycle task to scan the delayed processing record table for unprocessed update or delete operations; If it is an update operation, the latest data is retrieved from the source data source based on the data primary key; if the data does not exist in the source data source, the delayed record is deleted and the process ends; if the data exists in the source data source, the target data source is queried; if the data does not exist in the target data source, a write operation is performed; if the data exists in the target data source, an update is performed after version comparison; the delayed record is deleted after processing is complete. If it is a deletion operation, the deletion record table is queried. If there is a record with the same primary key and the deletion time is not earlier than the time of the current deletion operation, the processing ends. Otherwise, the target data source is queried. If the data to be deleted does not exist in the target data source, the process ends. If the data to be deleted exists, the deletion time is compared. If the current deletion time is later than the write time in the target data source, the deletion is performed. The delayed record is deleted after the processing is completed.

[0017] In one embodiment, the step of initiating a second background cycle task to scan the protection data table in the source data source and perform compensatory double writes for double write protection records that have timed out and not been confirmed includes: Start the second background cycle task to periodically scan the protection data table in the source data source and find double-write protection records whose status is pending and whose creation time exceeds the preset threshold. For each record found, reconstruct the double-write signal based on the double-write content recorded therein; The local client's double-write processing logic is invoked to re-execute the double-write operation for compensation. If the compensation process is successful, a confirmation message is sent to the source data source client so that the source data source client updates the status of the corresponding protection record to "processed". If multiple compensation attempts fail, an alarm is triggered.

[0018] In one embodiment, the step of recording the received double-write signal to the verification table and starting a third timed task to perform consistency verification and automatic error correction between the source data source and the target data source at the top of the hour includes: For each received double-write signal, record a verification message in the local verification table. The verification message includes at least the data primary key, operation timestamp, and data content hash value. The third scheduled task is initiated to scan all verification records in the verification table at each preset hour, and to perform the following verification operations on each verification record: Query the data content and timestamp of the same primary key in the source data source and the target data source respectively, and compare whether the two are consistent; If they match, delete the verification record. If the data content is inconsistent or the timestamp difference exceeds the allowable range, the error correction operation will be performed based on the data from the source data source.

[0019] In one embodiment, the error correction operation includes: If the source data source exists but the target data source does not exist, then write; If the source data source does not exist but the target data source does exist, then delete it. If both the source data source and the target data source exist but their data content is different, then update the target data source to the content of the source data source; After the error correction is completed, delete the verification record and record the error correction log.

[0020] Furthermore, to achieve the above objectives, this application also proposes a multi-data source synchronization system, the system comprising: a source data source client and a target data source client, wherein: The source data source client is used to respond to write requests to the source data source, determine whether dual write is required based on the dual write switch status and dual write conditions, and if so, generate a dual write signal; record a dual write guarantee record containing a unique sequence number and initial state in the guarantee data table of the source data source; send a dual write signal for the target data source client to perform dual write processing; receive a dual write processing result confirmation message returned by the target data source client, the confirmation message containing the unique sequence number and processing status; and update the status of the corresponding record in the guarantee data table according to the confirmation message. The target data source client is configured to receive a double-write signal from the source data source client, perform double-write processing based on the double-write signal, and return a double-write processing result confirmation message to the source data source client.

[0021] In addition, to achieve the above objectives, this application also proposes a multi-source data synchronization device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the multi-source data synchronization method described above.

[0022] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the multi-data source data synchronization method described above.

[0023] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the multi-data source data synchronization method described above.

[0024] One or more technical solutions proposed in this application have at least the following technical effects: The source data source client responds to a write request to the source data source by determining whether a dual write operation is needed based on the dual write switch status and dual write conditions. If needed, a dual write signal is generated. A dual write guarantee record containing a unique sequence number and initial state is recorded in the guarantee data table of the source data source. The dual write signal is sent to the target data source client for dual write processing. A confirmation message of the dual write processing result returned by the target data source client is received, containing the unique sequence number and processing status. Based on the confirmation message, the status of the corresponding record in the guarantee data table is updated. This solution addresses the performance issues arising from ensuring consistency during data synchronization across multiple data sources by adding a guarantee data table during data synchronization. Simultaneously, it guarantees data synchronization consistency and integrity between data sources in high-concurrency scenarios. Attached Figure Description

[0025] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0026] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a flowchart illustrating an embodiment of the multi-source data synchronization method of this application. Figure 2 This is a schematic diagram illustrating the process of dual-writing of the source data source client in an embodiment of the multi-source data synchronization method of this application; Figure 3 This is a flowchart illustrating another embodiment of the multi-source data synchronization method of this application. Figure 4 A schematic diagram illustrating the process of a target data source client handling write operations, provided in an embodiment of the multi-data source data synchronization method of this application; Figure 5 A schematic diagram illustrating the process of the target data source client handling update operations provided in an embodiment of the multi-data source data synchronization method of this application; Figure 6 A schematic diagram illustrating the process of a target data source client handling a deletion operation, provided in an embodiment of the multi-data source data synchronization method of this application; Figure 7 This is a flowchart illustrating the process of the target data source client handling the update operation of the delay queue, as provided in the embodiment of the multi-data source data synchronization method of this application. Figure 8This is a flowchart illustrating the process of deleting a delayed queue by the target data source client, as provided in an embodiment of the multi-data source data synchronization method of this application. Figure 9 This is a schematic diagram of the module structure of the multi-source data synchronization system according to an embodiment of this application; Figure 10 This is a schematic diagram of the device structure of the hardware operating environment involved in the multi-data source data synchronization method in this application embodiment.

[0028] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0029] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0030] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0031] The main solution of this application embodiment is as follows: The source data source client responds to a write request to the source data source by determining whether a dual write operation is needed based on the dual write switch status and dual write conditions. If needed, a dual write signal is generated. A dual write guarantee record containing a unique sequence number and initial state is recorded in the guarantee data table of the source data source. A dual write signal is sent for the target data source client to perform dual write processing. A confirmation message of the dual write processing result returned by the target data source client is received, the confirmation message containing the unique sequence number and processing status. Based on the confirmation message, the status of the corresponding record in the guarantee data table is updated. This solution solves the performance problem caused by ensuring consistency during data synchronization between multiple data sources by adding a guarantee data table during data synchronization. Simultaneously, it ensures the consistency and integrity of data synchronization between data sources in high-concurrency scenarios. Furthermore, this solution solves the problem of difficulty in guaranteeing consistency during data synchronization between multiple data sources by adding a delayed processing record table and a deletion record table during data synchronization. This solution also adds a verification table during data synchronization for hourly verification, solving the problem of not being able to automatically fill in and correct data inconsistencies in a timely manner during data synchronization between multiple data sources.

[0032] This application's embodiments take into account that, due to existing technologies, achieving data consistency through data synchronization between multiple data sources is often a cumbersome and time-consuming process. Traditional solutions require significant performance sacrifices to achieve real-time consistency. For example, Redis uses a "replication offset" when implementing master-slave data replication, and this offset must be compared for each synchronization. Furthermore, in concurrent scenarios, multiple operations on the same data can easily lead to data loss if the receiving order cannot be guaranteed, and data cannot be automatically replenished after loss.

[0033] For multi-data source data synchronization scenarios, this application provides a multi-data source data synchronization method, defined from both the source data source client side and the target data source client side. On the source data source client side, dual-write triggering and tracking are achieved through steps such as dual-write switch judgment, dual-write signal generation, ensuring data table records are maintained, signal sending, and status confirmation. On the target data source client side, a complete closed loop of data synchronization is achieved through steps such as receiving dual-write signals, processing them according to operation type (using record deletion and delayed record processing to resolve out-of-order issues), background periodic task compensation, guaranteed dual-write retries, and hourly verification and error correction.

[0034] This solution adds numerous data synchronization safeguards to the traditional asynchronous dual-write mechanism, ensuring data integrity and consistency while minimizing or avoiding performance impact on business systems. The main features of this solution are: (1) Add a delay processing mechanism: To address the out-of-order data reception issue during data synchronization, a "delay processing record table" and a "deletion record table" are added to the receiving data source to ensure eventual data consistency. For example, in the following scenarios: 1. The same data is added first and then updated in the source data. However, due to network latency and other issues, during data synchronization, the receiving data source receives the updated data first and then the added data. Since the receiving data has not yet been written, the update fails. The solution in this application is to use a delayed record table to first write the data to be updated into the delayed record table. After the data is written, the update operation of the delayed record table is processed to achieve eventual data consistency. Second, in some cases, the same data is added first and then deleted in the source data. However, due to network latency and other issues, during data synchronization, the receiving data source receives the data in the order of deletion first and then addition. Since the receiving data has not yet been written, the deletion fails, while the subsequent addition operation succeeds. This application's solution uses a deletion record table to record the deletion operation when deleting data and first checks whether the same data has been deleted when adding data, thus achieving eventual data consistency. (2) Add a data synchronization guarantee mechanism: In order not to affect the performance of dual writing, it is often necessary to perform asynchronous operations of multiple threads. The dual writing result of each thread is often unknowable. The solution of this application introduces a "guarantee data table" in the data source, records the dual writing data and initializes the state before dual writing, and modifies the state of the same dual writing data when receiving the data source for successful or failed dual writing to achieve the purpose of monitoring the dual writing result. (3) Add a timed verification mechanism: After receiving the double-written data, the receiver puts it into the "to be verified table" and scans the data in the verification data table at the next hour to verify and correct it.

[0035] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device, multi-data source data synchronization device, or system capable of realizing the above functions. The following description uses a multi-data source data synchronization system as an example to illustrate the various embodiments.

[0036] Based on this, the embodiments of this application provide a method for synchronizing data from multiple data sources, referring to... Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the multi-data source data synchronization method of this application.

[0037] In this embodiment, the multi-data source data synchronization method includes steps S10 to S50. The following provides a detailed explanation of each step.

[0038] like Figure 1 As shown, one embodiment of this application proposes a multi-data source data synchronization method, applied to the source data source client, the method comprising: Step S10: In response to the write request to the source data source, determine whether to perform a dual write based on the dual write switch status and dual write conditions. If necessary, generate a dual write signal. Specifically, dual writing between multiple data sources involves both the source data source and the target data source.

[0039] In this context, the source data source refers to the primary data source that must be written to, while the target data source refers to the data source to which the content written by the source data source needs to be copied. To achieve synchronous dual-write for both types of data sources, the clients connecting to these data sources (i.e., the source data source client and the target data source client) need to be modified separately.

[0040] This embodiment mainly describes an improvement scheme for the source data source client.

[0041] In this context, the source data source client refers to the application or server that connects to the source data source, responsible for receiving business write requests and performing data write operations on the source data source. For example, in an e-commerce system, the source data source can be the main database (MySQL), and the source data source client is the business application service.

[0042] A target data source client refers to an application or server that connects to the target data source. It is responsible for receiving write signals from the source data source client and performing corresponding write, update, or delete operations on the target data source. For example, the target data source can be a cache (Redis) or a search engine (Elasticsearch), and the target data source client is the corresponding data synchronization service.

[0043] Double writing refers to the operation of writing the same data to the target data source at the same time or after writing data to the source data source.

[0044] As a feasible implementation method, the dual-write signal contains a data structure containing information such as data primary key, operation type (write / update / delete), data content, timestamp or version number, which is used to notify the target data source client to perform synchronization.

[0045] As a feasible implementation, the dual-write switch is a configuration parameter (e.g., `enable.dual.write=true`) used to control whether the dual-write function is enabled. When the dual-write switch is off, the source data source client does not execute any dual-write logic, but only performs normal source data source write operations. The purpose of this dual-write switch is to quickly disable dual-write in case of system maintenance or failure of the dual-write function, so as to avoid affecting the main business.

[0046] As a feasible implementation method, double-write conditions are a set of filtering rules used to determine whether a specific write request needs to undergo double-write. For example, it can be configured to only perform double-write on specific table names or keys starting with a specific prefix, and requests that do not meet the conditions can skip the double-write logic directly. The purpose of these double-write conditions is to finely control the scope of double-write and avoid unnecessary resource consumption.

[0047] Specifically, as a feasible implementation method, the source data source client responds to the write request to the source data source by determining whether to perform a dual write based on the dual write switch status and dual write conditions. If necessary, a dual write signal is generated.

[0048] In this step, the source data source client continuously listens for write requests to the source data source initiated by the business layer. When a write request is received, the source data source client first reads the dual-write switch in the configuration parameters. This dual-write switch can be a boolean value in a static configuration file or a configuration center property that supports dynamic refresh. If the dual-write switch is off, the source data source client skips all subsequent dual-write logic and only performs normal source data source write operations.

[0049] If the dual-write switch is enabled, the source data source client obtains the dual-write object to be synchronized through the dual-write aspect. The dual-write aspect is an implementation based on AOP (Aspect-Oriented Programming) technology, inserting additional logic before and after the data write method execution. For example, in the Spring framework, an around advice can be defined using the "@Around" annotation to obtain method parameters or return values ​​as dual-write objects before and after method execution. The dual-write object is typically a POJO (Plain Old Java Object) containing all fields of the data that needs to be synchronized, such as order number, user ID, amount, status, etc.

[0050] After obtaining the dual-write object, the source data source client filters the object according to preset dual-write conditions. Dual-write conditions can be rules based on data content, such as "only synchronize orders with an order amount greater than 100 yuan" or "only synchronize user data whose user ID is in the graylist." These conditions can be defined using expressions (such as SpEL expressions) in the configuration file, allowing for flexible adjustment of filtering rules without modifying the code.

[0051] If a dual-write object meets the dual-write condition, the source data source client generates a dual-write signal. A dual-write signal is a structured data object containing at least the following fields: operation type (identifying whether the operation is write, update, or delete), primary key (used to locate the data in the target data source), data content (complete entity data; can be empty for delete operations), timestamp (the time the operation occurred in the source data source, accurate to milliseconds), and version number (used for optimistic locking control, incrementing with each update). This dual-write signal will be used for subsequent sending and synchronization processing.

[0052] Through the above steps, the source data client achieves precise control over dual-write operations, triggering dual-write only when the switch is on and the conditions are met, thus avoiding unnecessary performance overhead.

[0053] Step S20: Record a double-write guarantee record containing a unique sequence number and initial state in the guarantee data table of the source data source; The assurance data table is a database table located in the source data source, used to record the initial state and final result of each double-write operation. Its fields include: sequence number (uniquely identifying a double-write operation), primary key (the primary key of the data being operated on), double-write content summary (a summary or complete content of the double-write signal), creation timestamp (recording the generation time), status (pending / processed / failed), reason for failure, number of retries, etc. The purpose of this assurance data table is to ensure full traceability of double-write operations, providing a data foundation for subsequent assurance compensation.

[0054] The unique serial number can be a globally unique identifier generated using a UUID (Universally Unique Identifier), used to uniquely identify a single double-write operation. This serial number is used throughout the entire lifecycle of the double-write operation, from its generation on the source data source client, to the return of a confirmation message after processing by the target data source client, and then to the update of the status on the source data source client; the same serial number is used for association throughout the entire process.

[0055] Specifically, as a feasible implementation method, the source data source client records a double-write guarantee record containing a unique sequence number and initial state in the guarantee data table of the source data source.

[0056] In this step, after generating the dual-write signal but before sending it, the source data source client first inserts a dual-write guarantee record into the guarantee data table of the source data source. This guarantee data table is located in the source data source and resides in the same database as the business data table. This leverages the local transaction features of the source data source to ensure consistency between the guarantee record and the business data writes.

[0057] The data table is designed with the following fields: The sequence number field uses a UUID to generate a globally unique string, serving as the primary key of the record; the primary key field records the primary key value of the data being operated on; the double-write summary field stores a summary of the double-write signal or the complete JSON content, used to reconstruct the double-write signal when compensation is needed; the creation timestamp field records the current time; the status field is initially set to pending; the failure reason field is initially empty; and the retries count field is initially 0.

[0058] In this embodiment, the dual-write guarantee record serves the following purposes: First, it provides a traceable credential for each dual-write operation, allowing system administrators to query which data was attempted to be dual-written and its current status within any given time period. Second, it provides a data source for subsequent guarantee compensation mechanisms; when the target data source client fails to process a dual-write signal in a timely manner, it can re-trigger the dual-write by scanning records in the guarantee data table that are still in the "pending processing" state and have timed out. Finally, it ensures the persistence of dual-write operations, so that even if the source data source client restarts after sending the dual-write signal, the records in the guarantee data table will not be lost.

[0059] Through the above steps, the source data client establishes a persistent tracing mechanism for dual-write operations, laying the foundation for ensuring data consistency across the entire chain.

[0060] Step S30: Send a dual-write signal to allow the target data source client to perform dual-write processing; In this step, the source data source client sends out the generated double-write signal so that the target data source client can receive it and perform the corresponding data synchronization processing. The sending method can be either directly via network calls (such as REST API or RPC) to the target data source client, or asynchronous transmission via message brokers (such as Kafka or RocketMQ). This embodiment preferably uses Kafka as the message broker, encapsulating the double-write signal in JSON format and sending it to the specified topic in Kafka.

[0061] After transmission is complete, the double-write signal enters the message middleware or directly reaches the target data source client. Upon receiving the double-write signal, the target data source client will execute the corresponding write, update, or delete operation (see the subsequent target data source client-side implementation for specific processing methods). After the target data source client completes processing, it will return an acknowledgment message to the source data source client. This acknowledgment message contains a unique sequence number and processing status (success or failure).

[0062] Through the above steps, the source data source client reliably delivers the dual-write signal, asynchronously handing over the data synchronization task to the target data source client for processing, without affecting the main process performance of the source data source write operation.

[0063] Step S40: Receive a confirmation message of the double-write processing result returned by the target data source client. The confirmation message includes the unique sequence number and the processing status. In this step, the source data source client waits for and receives an acknowledgment message from the target data source client. The method of transmitting the acknowledgment message depends on the communication method used in the preceding steps. If Kafka is used, the target data source client can send the acknowledgment message through another Kafka topic or return it directly via an HTTP callback interface; if direct RPC calls are used, the acknowledgment message can be returned via a synchronous or asynchronous RPC response.

[0064] The confirmation message contains two key pieces of information: a unique sequence number and a processing status. The unique sequence number is used to associate the confirmation message with the previously recorded dual-write guarantee record, while the processing status indicates whether the target data source client's processing of the dual-write signal was successful or failed. If processing fails, the confirmation message may also include the reason for the failure (such as "target data source connection timed out" or "data version conflict") so that the source data source client can perform subsequent processing.

[0065] Through the above steps, the source data client receives feedback on the results of the dual-write operation, forming a complete closed loop from initiation to confirmation.

[0066] Step S50: Update the status of the corresponding record in the security data table according to the confirmation message.

[0067] In this step, the source data source client updates the status of the corresponding record in the protection data table based on the received confirmation message. Specifically, the source data source client extracts a unique sequence number from the confirmation message, uses this sequence number to query the corresponding protection record in the protection data table, and then updates the status field of that record according to the processing status in the confirmation message.

[0068] If the confirmation message indicates successful processing, the status of the corresponding record in the data table will be updated to "processed". This means that the double-write operation has been successfully completed, and the data in the source data source and the target data source are consistent.

[0069] If the confirmation message indicates processing failure, the following logic is executed: First, the retry count field in the protection record is incremented by 1; then, it is determined whether the retry count exceeds a preset threshold (e.g., 3 or 5 times). If it does not exceed the threshold, the status is kept as "pending," waiting for the subsequent protection compensation mechanism to retry sending the double-write signal; if the retry count exceeds the preset threshold, an alarm is triggered, for example, by sending an email, SMS, or writing to the alarm log to notify system maintenance personnel, informing them that the double-write operation has failed multiple times and manual intervention is required for investigation.

[0070] Through the above steps, the source data client implements closed-loop management and failure retry mechanism for dual-write state, which not only ensures the observability of dual-write operation, but also ensures that abnormal situations can be detected and handled in a timely manner through the alarm mechanism.

[0071] This embodiment's technical solution achieves a complete closed loop on the source data source client side, from dual-write triggering, guarantee record, signal sending, result confirmation, and status update. This ensures that every dual-write operation is traceable, retryable, and alertable, providing a solid foundation for the reliability of data synchronization between multiple data sources. Specifically, in response to a write request to the source data source, the source data source client determines whether dual-write is necessary based on the dual-write switch status and dual-write conditions. If necessary, it generates a dual-write signal; records a dual-write guarantee record containing a unique sequence number and initial state in the source data source's guarantee data table; sends the dual-write signal for the target data source client to perform dual-write processing; receives a dual-write processing result confirmation message from the target data source client, the confirmation message containing the unique sequence number and processing status; and updates the status of the corresponding record in the guarantee data table based on the confirmation message. This solution addresses performance issues arising from ensuring consistency during data synchronization between multiple data sources by adding a guarantee data table during data synchronization; simultaneously, it guarantees the consistency and integrity of data synchronization between data sources in high-concurrency scenarios.

[0072] In one feasible implementation, another embodiment of this application is proposed. In this embodiment, for step S10, in response to a write request to the source data source, it is determined whether a dual write operation needs to be performed based on the dual write switch state and dual write conditions. If so, a dual write signal is generated for detailed analysis.

[0073] Step S10, in response to a write request to the source data source, determines whether a dual write operation is needed based on the dual write switch status and dual write conditions. If needed, a dual write signal is generated, including: In response to a write request to the source data source, read the dual-write switch configuration parameters and determine whether the dual-write switch is enabled. If the dual-write switch is enabled, the dual-write object to be synchronized is obtained through the dual-write aspect; The dual-write objects are filtered according to preset dual-write conditions. If the dual-write objects meet the dual-write conditions, a dual-write signal is generated. The dual-write signal includes operation type, data primary key, data content, timestamp or version number.

[0074] Specifically, in response to a write request to the source data source, the source data source client first reads the dual-write switch configuration parameter. This configuration parameter can be stored in various configuration sources, such as the "dual.write.enabled" property in the "application.yml" configuration file of Spring Boot; or it can be a dynamic configuration item in a distributed configuration center (such as Apollo, Nacos), which supports dynamic refresh at runtime without restarting the service.

[0075] After reading the configuration parameters, the source data source client determines whether the dual-write switch is enabled. If the switch is disabled, the dual-write logic is skipped, and only the normal source data source write operation is performed. This ensures that the main business process is not affected during dual-write function failures or system maintenance.

[0076] If the dual-write switch is enabled, the source data source client obtains the dual-write object to be synchronized through the dual-write aspect. The dual-write aspect is an implementation method based on AOP technology. For example, in the Java Spring framework, an aspect class annotated with "@Around" can be defined. This aspect class intercepts all methods annotated with "@DualWrite" or matching a specific package name. When the intercepted method is executed, the aspect logic is triggered before and after the method execution. After the method execution, the aspect can extract the dual-write object from the method's return value or from the method's parameters (e.g., method parameters containing data objects to be saved). This dual-write object contains all the field data that needs to be synchronized to the target data source.

[0077] After obtaining the dual-write object, the source data source client filters the object according to preset dual-write conditions. The dual-write condition can be an expression returning a Boolean value, which can be written using SpEL (Spring Expression Language), making the filtering rules flexible and configurable. The purpose of filtering is to finely control the scope of dual-writes, avoiding unnecessary storage and performance overhead on the target data source caused by dual-writing all data. For example, in a canary release scenario, dual-writes can be configured to only apply to data within a specific range of user IDs, enabling a gradual switch in traffic.

[0078] If a dual-write object meets the dual-write conditions, a dual-write signal is generated. A dual-write signal is a structured data object with fields including: operation type (enumerated values: INSERT, UPDATE, DELETE), used to inform the target data source client what operation needs to be performed; primary key (e.g., order number or user ID), used to locate the specific data record in the target data source; data content (complete entity data for INSERT and UPDATE operations, and can be empty or contain only the primary key for DELETE operations); timestamp (the current system time in milliseconds), used for time comparison in out-of-order scenarios; and version number (initially 1, incrementing with each update), used for optimistic locking control.

[0079] Through the above steps, this embodiment clarifies the complete process from write request to dual write signal generation, covering key links such as switch judgment, aspect acquisition, condition filtering and signal construction, making the logic of dual write triggering clear and controllable.

[0080] The technical solution in this embodiment achieves fine-grained control of dual-write triggering. Through multi-level management of switching, aspect, and conditional filtering, it not only ensures the flexible switching of dual-write function, but also achieves precise filtering of the dual-write data range, avoiding unnecessary performance loss.

[0081] In one feasible implementation, another embodiment of this application is proposed. In this embodiment, step S30, sending a dual-write signal for the target data source client to perform dual-write processing, is explained in detail, particularly the control of the timing of the sending.

[0082] S30 involves sending a dual-write signal to the target data source client for dual-write processing, including: According to the sending timing configured in the dual-write aspect, the dual-write signal is sent to the message middleware or directly to the target data source client so that the target data source client can perform dual-write processing.

[0083] Specifically, the source data source client sends the dual-write signal to the message middleware or directly to the target data source client, depending on the sending timing configured in the dual-write aspect. The sending timing is a configuration item of the dual-write aspect and can take one of the following two values: send immediately or delay until the source data source transaction is committed.

[0084] If the configured sending timing is immediate, the source data source client will push the double-write signal to the message middleware or send it to the target data source client via a direct network call immediately after generating the double-write signal. The advantage of this approach is that the double-write operation has minimal latency, and the target data source client can receive and process the double-write signal as quickly as possible. Immediate sending is suitable for scenarios with high real-time requirements and a low probability of transaction rollback in the source data source.

[0085] If the configured sending timing is delayed until after the source data source transaction is committed, the source data source client does not immediately send the double-write signal, but instead temporarily stores the double-write signal in a local queue. This local queue can be a thread-safe queue implementation such as a "ThreadLocal" variable (thread-local variable) or "ConcurrentLinkedQueue". Simultaneously, the source data source client registers a transaction synchronization callback.

[0086] Transaction synchronization callback is a programming mechanism that allows specific callback logic to be executed after a database transaction is committed. In this embodiment, when configured for delayed sending, the source data source client temporarily stores the double-write signal in a local queue and registers a transaction synchronization callback. This callback is triggered after the source data source transaction is successfully committed, retrieving the double-write signal from the queue and sending it.

[0087] Specifically, during the execution of the transaction in the source data source, the double-write signal is kept in a local queue and is not sent. Only after the source data source transaction is successfully committed, and the "afterCommit()" method of the transaction synchronization callback is triggered, will the source data source client retrieve the double-write signal from the local queue and send it out. The advantage of this approach is that it guarantees the atomicity of the source data source transaction, meaning that only successfully committed data is synchronized to the target data source, avoiding the synchronization of dirty data.

[0088] If a transaction in the source data source is rolled back (e.g., due to business exceptions or database constraint conflicts), the "afterCompletion(int status)" method of the transaction synchronization callback will be triggered, where the status parameter is "STATUS_ROLLED_BACK". At this time, the source data source client will discard the double-write signal in its local queue and will not send it to the target data source. This ensures data consistency between the source and target data sources; that is, if the source data source fails to write data, the target data source will not write it either.

[0089] Through the above steps, this embodiment provides flexible control over the timing of transmission, enabling developers to select the most suitable transmission strategy based on specific business scenarios and achieve a balance between data consistency and real-time performance.

[0090] The technical solution of this embodiment realizes flexible configuration of the timing of dual write signal transmission. It supports real-time transmission to meet low latency requirements, as well as transmission after transaction commit to ensure data consistency. Furthermore, it avoids the synchronization of dirty data to the target data source through the signal discarding mechanism during transaction rollback.

[0091] In one feasible implementation, another embodiment of this application is proposed. In this embodiment, the configuration of the dual-write facets is described in detail.

[0092] Before sending the dual-write signal in step S30 to allow the target data source client to perform dual-write processing, the following steps are also included: The dual-write aspect can be configured through annotations or configuration files. The configuration items include: the expression for the dual-write switch, the SpEL extraction expression for the dual-write object, the Boolean expression for dual-write condition filtering, and the enumeration value for the timing of sending.

[0093] Specifically, before sending the dual-write signal, developers need to configure the dual-write aspect using annotations or configuration files. The configuration options for the dual-write aspect include the following four aspects: The expression for the dual-write switch: Specifies an expression that returns a Boolean value to control the overall on / off state of the dual-write function. This expression supports dynamic refreshing; when the value in the configuration center changes, the aspect can use the new switch value in the next interception.

[0094] SpEL extraction expression for double-write objects: Specifies a SpEL expression to extract double-write objects from the parameters or return values ​​of the intercepted method. The flexibility of SpEL expressions allows double-write aspects to adapt to various method signatures.

[0095] Boolean expression for double-write condition filtering: Specifies a SpEL expression that returns a boolean value to determine whether the current double-write object meets the double-write condition. This expression can use any field of the double-write object for logical combination to achieve fine-grained filtering control.

[0096] The enumerated value for the timing of sending: Specifies when the double-write signal is sent, and can be either "IMMEDIATE" (send immediately) or "AFTER_TRANSACTION" (send after transaction commit). This configuration determines when the double-write signal is sent to the target data source client.

[0097] In actual development, these configurations can be implemented in two ways: one is annotation-based, where developers annotate methods that require double-writing; the other is XML or YAML configuration file-based, where double-writing rules are specified for specific classes or methods in the configuration file. Each method has its advantages: annotation-based methods are more intuitive and have better code cohesion, while configuration file-based methods are more flexible and require no code modification.

[0098] Through the above steps, this embodiment provides a complete dual-write aspect configuration scheme, which allows developers to flexibly define the dual-write switch, object source, filtering conditions and sending timing, without having to write duplicate code for each method that requires dual-write.

[0099] The technical solution of this embodiment realizes flexible configuration of dual-write aspects, and supports the customization of switch expressions, object extraction, conditional filtering and sending timing through annotations or configuration files, which greatly reduces the access cost of dual-write function and improves the maintainability and scalability of code.

[0100] Therefore, in order to reduce the workload of the transformation, a dual-write aspect is added at the data writing point. This aspect can specify whether to perform dual writing, whether to send the dual-write signal before or after writing to the source data source, how to generate the dual-write object, and whether to send the dual-write signal immediately.

[0101] The processing flow of the source data client in this embodiment can be referred to Figure 2 As shown.

[0102] like Figure 2 As shown, the process of dual writing between the source and data source clients is as follows: First, determine whether the dual-write switch is turned on; this switch can be a configuration parameter.

[0103] Then, obtain the dual-write object, which is specified by the configuration of the dual-write aspect and is the main content of the dual-write signal.

[0104] Next, it is determined whether the conditions for dual writing are met. This step mainly filters the content to be written to the target data source. Content that does not meet the conditions cannot be written to the target data source.

[0105] Next, check whether the double-write content is immediately sent to the target data source client. This check can be specified in the double-write aspect. The purpose of this step is to implement database transaction operations. When transaction control is required, you can choose not to send it immediately, but to add it to a write queue. After the transaction of the source data source is completed, manually send the data of the source data source to the target data source client.

[0106] The process involves pushing the double-written data to Kafka, which is then consumed by the target data source. This step primarily serves to record the received double-written content. Since data loss is possible during the double-write process, this additional Kafka data push allows for data verification.

[0107] The technical solution of this embodiment realizes the flexible configuration of dual-write aspects. This embodiment clarifies the complete process from write request to dual-write signal generation, covering key links such as switch judgment, aspect acquisition, condition filtering and signal construction, making the logic of dual-write triggering clear and controllable.

[0108] The technical solution in this embodiment achieves fine-grained control of dual-write triggering. Through multi-level management of switching, aspect, and conditional filtering, it not only ensures the flexible switching of dual-write function, but also achieves precise filtering of the dual-write data range, avoiding unnecessary performance loss.

[0109] like Figure 3 As shown, another embodiment of this application proposes a multi-data source data synchronization method, applied to a target data source client, the method comprising: Step S100: Receive a dual-write signal from the source data source client. The dual-write signal is generated by the source data source client in response to a write request to the source data source, based on the dual-write switch status and dual-write conditions. Specifically, the target data source client refers to the application or server that connects to the target data source. It is responsible for receiving double-write signals from the source data source client and performing corresponding data operations on the target data source. For example, the target data source can be a Redis cache or an Elasticsearch search engine, and the target data source client is the corresponding data synchronization consumer service.

[0110] In this step, the target data source client continuously listens for double-write signals from the source data source client. The receiving method differs depending on how the source data source client sends the signal: if the source data source client sends the double-write signal via Kafka, the target data source client, acting as a Kafka consumer, pulls messages from the specified topic; if the source data source client sends the signal via direct RPC calls, the target data source client exposes a REST API or RPC interface for the source data source client to call.

[0111] The dual-write signal is generated by the source data source client in response to a write request to the source data source, based on the dual-write switch status and dual-write conditions. The dual-write signal contains all the information required for synchronization: operation type (write / update / delete), data primary key, data content, timestamp, and version number.

[0112] Before the source data source client sends the dual-write signal, a dual-write guarantee record containing a unique sequence number and initial state has already been recorded in the source data source's guarantee data table. This guarantee record is associated with the current dual-write signal through a unique sequence number and persists throughout the entire dual-write lifecycle.

[0113] Through the above steps, the target data source client successfully received the double-write signal from the source data source client, thus preparing for subsequent data synchronization processing.

[0114] Step S200: Perform dual-write processing according to the dual-write signal; In this step, the target data source client performs corresponding data operations on the target data source based on the received double-write signal. The specific processing method varies depending on the type of operation; please refer to the subsequent embodiments for details.

[0115] Through the above steps, the target data source client has completed the actual processing of the dual-write signal, synchronizing the data changes from the source data source to the target data source.

[0116] Step S300: Return a double-write processing result confirmation message to the source data source client. The confirmation message includes the unique sequence number and processing status. The source data source client updates the status of the corresponding record in the guarantee data table of the source data source according to the confirmation message. Before sending the double-write signal, the source data source client records a double-write guarantee record containing a unique sequence number and initial status in the guarantee data table.

[0117] In this step, after completing the double-write process, the target data source client returns a confirmation message to the source data source client. The transmission method of the confirmation message corresponds to the reception method in step S100: if the double-write signal is received through Kafka, the confirmation message can be sent through another Kafka topic; if it is received through a REST API, the processing result can be returned directly in the HTTP response.

[0118] The confirmation message contains two key pieces of information: a unique sequence number and a processing status. The unique sequence number is extracted from the double-write signal and is used to associate it with a record in the source data source's security data table; the processing status, whether successful or failed, indicates the result of the target data source client's processing of the double-write signal.

[0119] After receiving the confirmation message, the source data client updates the status of the corresponding record in the security data table according to the confirmation message (see the source data client side embodiment for details), forming a complete closed loop from initiation to confirmation.

[0120] Through the above steps, the target data source client feeds back the processing results to the source data source client, enabling the source data source client to track the final state of each double-write operation.

[0121] The technical solution of this embodiment realizes the complete process from receiving and processing dual-write signals to feedback of results on the target data source client side, forming a closed-loop tracking of the entire chain with the guarantee recording mechanism on the source data source client side, ensuring that every dual-write operation can be monitored and traced.

[0122] In one feasible implementation, another embodiment of this application is proposed. In this embodiment, the operation type branching process in step S200, which involves performing dual-write processing based on the dual-write signal, is elaborated in detail.

[0123] Step S200, which involves performing a dual-write process based on the dual-write signal, includes: The operation type of the dual write signal is analyzed, and the operation type includes write operation, update operation, or delete operation; Write, update, or delete operations are executed according to the operation type, and dual-write synchronization is performed using a delete record table and a delay processing record table. The delete record table is used to store the primary key of the data that has been successfully deleted and the deletion timestamp. The delay processing record table is used to temporarily store update or delete operations that cannot be executed because the data to be operated on does not exist in the target data source.

[0124] Specifically, the target data source client needs to be modified to receive dual-write signals, and the consistency issues that arise when concurrent operations on the same data need to be resolved. These consistency issues include: ① The same data in the source data was written and then deleted. Due to network issues, the target data source received the data out of order, receiving the deletion operation first and then the write operation. ② The same data in the source data was written first and then updated. Due to network issues, the target data source received the data out of order, receiving the update operation first and then the write operation. ③ The same data in the source data source is updated multiple times, such as first updating to A and then updating to B. Due to network issues, the target data source receives the data out of order, receiving the update to B operation first and then the update to A operation.

[0125] To address the above issues, the target data source client needs to handle write, update, and delete operations separately during the modification process.

[0126] Specifically, upon receiving the double-write signal, the target data source client first parses the operation type field within the signal. There are three operation types: write, update, and delete. Depending on the operation type, the target data source client executes different processing logic and utilizes a delete record table and a delayed record table processing mechanism to address data out-of-order issues in concurrent scenarios.

[0127] The deletion record table is located in the target data source and its fields include: auto-incrementing primary key, data primary key (with a unique index), deletion timestamp (milliseconds), and record creation time. This table stores the data primary key and deletion timestamp of successfully executed deletion operations, persistently recording successful deletions and resolving the out-of-order issue of "delete-then-insert" operations. When a subsequent write operation targeting the same primary key is received, this table can be queried to determine if a deletion operation has been performed previously, and the timestamp will determine whether the write operation is allowed. In other words, if the target data source client receives a deletion signal before a write signal, it can determine whether to allow the write operation by querying the deletion record table.

[0128] The delayed processing record table is located in the target data source. Its fields include: auto-incrementing primary key, data primary key, operation type, operation content (stored in JSON format), original timestamp, version number, status (pending / processed), and number of retries. This table temporarily stores update or delete operations that cannot be executed immediately due to the absence of the target data in the target data source. These operations are then processed later through a background task to compensate for the delay. This solves the "increment first, update later" out-of-order problem. When the target data source client receives an update signal before a write signal, the update operation is temporarily stored in the delayed table and executed later, after the write operation arrives.

[0129] Background periodic tasks: Scheduled tasks triggered by the scheduling framework, executed at fixed time intervals. This embodiment involves multiple background periodic tasks, used for delay compensation, guaranteed retries, and hourly verification, respectively.

[0130] As a feasible implementation method, the process for the target data source client to handle the dual write signals of the write operation can be referred to Figure 4 As shown.

[0131] When the operation type is a write operation, the target data source client first queries the local deletion record table of the target data source to check if there is a deletion record with the same primary key as the current data.

[0132] If the query result is empty (i.e., no deleted record exists), it means that the data has not been deleted before. The target data source client will directly perform a write operation to insert the data content into the target data source.

[0133] If the query result exists (i.e., a deleted record exists), the write timestamp in the double-write signal is compared with the deletion timestamp in the deleted record. The write operation is only executed if the write timestamp is later than the deletion timestamp; otherwise, the write signal is discarded. This logic solves the out-of-order problem of "delete first, add later". Suppose that in the source data source, a piece of data is first deleted (timestamp 100) and then added (timestamp 200). However, due to network latency, the target data source client receives the add signal (timestamp 200) first and then the delete signal (timestamp 100). If the write operation is executed directly, data that should have been deleted will be incorrectly retained in the target data source. By querying the deleted record table and comparing the timestamps, the target data source client finds that the timestamp 100 of the deleted record is less than the write timestamp 200, and therefore allows the write. When processing the deletion signal later (see the deletion operation handling below), the timestamps are compared again, and the deletion is executed correctly. Conversely, if the write timestamp is earlier than the deletion timestamp, it means that the data has already been deleted in the source data source, and this write operation is outdated and should be discarded.

[0134] As a feasible implementation method, the process for the target data source client to handle the dual write signal of the update operation can be referred to Figure 5 As shown.

[0135] When the operation type is an update operation, the target data source client first queries whether there is a data record in the target data source that has the same primary key as the current data.

[0136] If the data record does not exist in the target data source, it means that the data may have been added first and then updated in the source data source, but the addition signal has not yet reached the target data source client due to network latency. In this case, the target data source client does not discard the update signal, but instead stores this update operation in a local delay processing record table, waiting for the addition signal to arrive before performing a compensatory update through a background task. This solves the out-of-order problem of "added first, updated later".

[0137] If the data record exists in the target data source, the version number or timestamp in the double-write signal is compared with the version number or timestamp of the existing record in the target data source. The update operation is only performed if the version number or timestamp in the double-write signal is updated (i.e., the version number in the double-write signal is greater than the existing version number, or the timestamp in the double-write signal is later than the existing timestamp). An optimistic locking mechanism is used during the update. If the number of rows affected by the update is 0, it means the version has already been updated by other operations; the update fails and can be discarded or retried. This mechanism solves the problem of "out-of-order updates".

[0138] As a feasible implementation method, the process for the target data source client to handle the double write signal for the deletion operation can be referred to Figure 6 As shown.

[0139] When the operation type is a delete operation, the target data source client first queries whether there is a data record in the target data source that has the same primary key as the current data.

[0140] If the data record does not exist in the target data source, it means that the data may not have been written to the target data source yet (the new signal has not arrived). In this case, the deletion operation is stored in the delayed processing record table, and the deletion is performed through a background task after subsequent data is written.

[0141] If the data record exists in the target data source, the deletion timestamp in the double-write signal is compared with the write timestamp of the record in the target data source. The deletion operation is only performed if the deletion timestamp is later than the write timestamp. After successful deletion, a record containing the primary key and deletion timestamp is inserted into the deleted record table. This mechanism ensures that only data that has actually been deleted in the source data source is deleted in the target data source, avoiding erroneous deletions due to out-of-order delivery.

[0142] Through the above steps, this embodiment designs different processing logics for the three operation types, and makes full use of the deletion record table and the delayed processing record table to solve various out-of-order problems in high-concurrency scenarios.

[0143] This embodiment's technical solution enables the target data source client to perform typified processing of write, update, and delete operations. It solves the "delete first, add later" out-of-order problem by deleting the record table, solves the "add first, update later" out-of-order problem by delaying the processing of the record table, and solves the "multiple update out-of-order" problem by version number comparison and optimistic locking, thus comprehensively ensuring the eventual consistency of data synchronization in high-concurrency scenarios.

[0144] In one feasible implementation, another embodiment of this application is proposed. In this embodiment, step S200, performing dual-write processing based on the dual-write signal, further includes at least one of the following: Initiate the first background cycle task, scan the delayed processing record table, and compensate for any unprocessed update or deletion operations. Initiate the second background cycle task to scan the protection data table in the source data source and perform compensatory double writes for double write protection records that have timed out and have not been confirmed. The received double-write signal is recorded in the verification table, and the third timed task is started to perform consistency verification and automatic error correction between the source data source and the target data source at the top of the hour.

[0145] In one feasible implementation, the background compensation mechanism of the delay processing record table in the step "perform dual-write processing according to the dual-write signal" is elaborated in detail.

[0146] The first background cycle task is initiated, which scans the delayed processing record table and performs compensation processing on any unprocessed update or deletion operations, including: Start the first background cycle task to scan the delayed processing record table for unprocessed update or delete operations; If it is an update operation, the latest data is retrieved from the source data source based on the data primary key; if the data does not exist in the source data source, the delayed record is deleted and the process ends; if the data exists in the source data source, the target data source is queried; if the data does not exist in the target data source, a write operation is performed; if the data exists in the target data source, an update is performed after version comparison; the delayed record is deleted after processing is complete. If it is a deletion operation, the deletion record table is queried. If there is a record with the same primary key and the deletion time is not earlier than the time of the current deletion operation, the processing ends. Otherwise, the target data source is queried. If the data to be deleted does not exist in the target data source, the process ends. If the data to be deleted exists, the deletion time is compared. If the current deletion time is later than the write time in the target data source, the deletion is performed. The delayed record is deleted after the processing is completed.

[0147] Specifically, the target data source client initiates a first background periodic task to periodically scan the delayed processing record table and compensate for any unprocessed update or deletion operations. This background periodic task can be implemented using Java's "ScheduledExecutorService".

[0148] Each time a task is executed, it first queries the delayed processing record table for all records with a status of "pending processing". To avoid excessive load on a single task, the maximum number of records processed per session can be limited to 100. For each retrieved record, compensation processing is performed according to its operation type: As one implementation method, for update operations in delayed records, the target data source client first queries the source data source based on the primary key of the data in the record to obtain the latest complete state of the data. The source data source can be queried via RPC call to the query interface provided by the source data source service, or by directly querying the source database.

[0149] If the data no longer exists in the source data source (the query result is empty), it means that the data has been deleted from the source data source. There is no need to update it. Just delete the delayed record and end the processing.

[0150] If the data exists in the source data source, continue querying the target data source to see if a record with the same primary key exists. If the data does not exist in the target data source, it means the add signal has arrived and the write operation has been completed. In this case, the latest data from the source data source is directly written to the target data source. If the data exists in the target data source, perform an update based on version comparison. Compare the version number of the source data with the version number of the existing data in the target data source; the update is only performed if the version number of the source data source is higher.

[0151] After processing is complete, delete the record from the delayed processing log table (or update the status to "processed") to prevent duplicate execution.

[0152] As one implementation method, for deletion operations in delayed records, the target data source client first queries the deletion record table to check if there is a record with the same primary key and a deletion time no earlier than the time of the current pending deletion operation. If such a record exists, it means that the data has already been deleted, the current deletion operation is outdated, and the processing ends directly.

[0153] If the record to be deleted does not exist in the record table, or if it exists but its deletion time is earlier than the time of this operation, then continue to query whether the data exists in the target data source. If the data does not exist in the target data source, it means that the data has not been written or has been deleted, and the process ends directly.

[0154] If the data exists in the target data source, compare the timestamp of this deletion operation with the write timestamp of the data in the target data source. The deletion operation is only performed if the timestamp of this deletion operation is later than the write timestamp in the target data source. After processing, the record is deleted from the delayed processing record table.

[0155] During the compensation process, if a record fails to process (e.g., the source data connection times out), the record is not deleted. Instead, the retry count field is incremented by 1, and the record is allowed to try again in the next cycle. If the retry count reaches a preset maximum value (e.g., 5 times), the record is marked as "failed" and an alarm is triggered, and automatic retrying stops.

[0156] Through the above steps, this embodiment implements an automatic compensation mechanism for the delayed processing record table, ensuring that update and deletion operations temporarily stored due to out-of-order processing can ultimately be executed correctly.

[0157] The technical solution of this embodiment realizes automatic compensation of the delayed processing record table through the first background periodic task, so that the operations temporarily stored due to the disorder of "add first and then update" or "add first and then delete" can be correctly executed after the data arrives, ensuring the final consistency of the data. At the same time, the retry and alarm mechanism ensures the controllability of abnormal situations.

[0158] In one feasible implementation, the protective double-write compensation mechanism in the step "perform double-write processing according to the double-write signal" is elaborated in detail.

[0159] The initiation of the second background cycle task involves scanning the protection data table in the source data source and performing compensatory double writes on the double write protection records that have timed out and not been confirmed. Start the second background cycle task to periodically scan the protection data table in the source data source and find double-write protection records whose status is pending and whose creation time exceeds the preset threshold. For each record found, reconstruct the double-write signal based on the double-write content recorded therein; The local client's double-write processing logic is invoked to re-execute the double-write operation for compensation. If the compensation process is successful, a confirmation message is sent to the source data source client so that the source data source client updates the status of the corresponding protection record to "processed". If multiple compensation attempts fail, an alarm is triggered.

[0160] Specifically, the target data source client initiates a second background periodic task to periodically scan the protection data table in the source data source and perform compensatory double writes on expired and unconfirmed double-write protection records. The execution interval of this background periodic task can be set to 1 minute, which ensures timely detection of expired records while avoiding excessive query pressure on the source data source.

[0161] Each time a task is executed, the target data source client connects to the source data source and scans the data table for records whose status is "pending" and whose creation time exceeds a preset threshold (e.g., 30 seconds). If the double-write signal corresponding to these records is not acknowledged by the target data source client within 30 seconds after being sent, it may be due to network interruptions, Kafka message loss, target data source client crashes, or other reasons.

[0162] For each timeout record found, the target data source client performs the following compensation operation: First, the double-write content digest field is extracted from the assurance record, and the double-write signal is reconstructed based on this field. The double-write content digest in the assurance record stores complete information about the original double-write signal (including operation type, data primary key, data content, timestamp, and version number), so the original double-write signal can be completely recovered.

[0163] Secondly, the target data source client calls its local double-write processing logic (i.e., the processing flow described in Examples Six and Seven) to re-execute the double write. Since the target data source client's double-write processing logic is idempotent (ensuring consistent results from multiple processing of the same double-write signal through timestamp comparison and version number comparison), repeated execution will not lead to data errors.

[0164] If the compensation process is successful, the target data source client sends a confirmation message to the source data source client (or directly updates the protection data table in the source data source), updating the status of the corresponding protection record to "processed". If the compensation process fails, the record status is retained as "pending", waiting for the next cycle to try again. If multiple compensation failures occur (e.g., 3 consecutive times), an alarm is triggered, notifying system maintenance personnel to conduct manual investigation.

[0165] It's important to note that while the second background cycle task runs on the target data source client, it requires access to the guaranteed data tables in the source data source. This necessitates that the target data source client have read and update permissions for the source data source. In actual deployments, cross-data source access can be achieved through database connections or API calls.

[0166] Through the above steps, this embodiment implements a protective dual-write compensation mechanism, which can automatically detect and compensate for dual-write omissions caused by various abnormal reasons, greatly improving the reliability of data synchronization.

[0167] The technical solution in this embodiment achieves automatic compensation for timeout and unconfirmed double writes by scanning the data table through a second background periodic task. This effectively addresses abnormal scenarios such as network interruptions and message loss, ensuring the high reliability of the double write operation. At the same time, the alarm mechanism ensures that serious anomalies can be detected and handled in a timely manner.

[0168] In one feasible implementation, the hourly verification and automatic error correction mechanism in the step "dual write processing according to the dual write signal" is elaborated in detail.

[0169] The process of recording the received dual-write signal to the verification table and starting the third timed task to perform consistency verification and automatic error correction between the source data source and the target data source at the top of each hour includes: For each received double-write signal, record a verification message in the local verification table. The verification message includes at least the data primary key, operation timestamp, and data content hash value. The third scheduled task is initiated to scan all verification records in the verification table at each preset hour, and to perform the following verification operations on each verification record: Query the data content and timestamp of the same primary key in the source data source and the target data source respectively, and compare whether the two are consistent. If they are consistent, delete the verification record. If the data content is inconsistent or the timestamp difference exceeds the allowable range, the data in the source data source shall prevail and the error correction operation shall be performed.

[0170] Specifically, the target data source client records each received double-write signal to the verification table and starts a third timed task to perform consistency verification and automatic error correction between the source data source and the target data source at the top of the hour.

[0171] In this process, the source data source client sends a double-write signal to the main Kafka topic and simultaneously pushes the same double-write signal content to another Kafka topic—the verification topic. The target data source client starts a separate consumer group to consume messages from this verification topic. When a verification message is consumed, the target data source client, in addition to performing normal double-write processing, also inserts a verification record into its local table to be verified.

[0172] The fields in the table to be verified include: auto-incrementing primary key, data primary key (the primary key of the data being operated on), operation timestamp (the time the operation occurred in the source data source), data content hash value, and reception time (the time the target data source client received the verification message). This table stores metadata for all double-write operations that require consistency verification.

[0173] The target data source client starts a third scheduled task, which is configured using a Cron expression to trigger every hour on the hour. During task execution, it scans all verification records in the table to be verified.

[0174] For each verification record, perform the following verification operations: First, query the data content and timestamp of the same primary key in both the source and target data sources. Querying the source data source can be done by calling the query interface provided by the source data source service via RPC, or by directly querying the source database; querying the target data source involves directly querying the target database or its cache.

[0175] Secondly, compare whether the two are consistent. The comparison includes two aspects: whether the data content is consistent (by comparing the hash value of the data content or comparing field by field) and whether the timestamp difference is within the allowable range (e.g., the difference is less than 5 seconds). If the hash values ​​of the data content are the same and the timestamp difference is within the allowable range, the data is considered consistent, and the verification record is deleted directly.

[0176] If the data content is inconsistent or the timestamp difference exceeds the allowable range, the error correction operation will be performed based on the data from the source data source.

[0177] Error correction operations fall into three categories: In the first scenario: if the data exists in the source data source (the query result is not empty) but the data does not exist in the target data source, then a write operation is performed to completely write the data from the source data source to the target data source.

[0178] The second scenario is: if the data does not exist in the source data source (the query result is empty) but the data exists in the target data source, then a deletion operation is performed to remove the data record from the target data source.

[0179] The third scenario: If the data exists in both the source and target data sources but the data content is different, then an update operation is performed to update the data in the target data source with the data content of the source data source.

[0180] After the error correction operation is completed, delete the corresponding verification record in the table to be verified to prevent duplicate verification. At the same time, record an error correction log, including the data primary key, inconsistency details, error correction action (write / delete / update), and error correction time, for subsequent auditing and problem tracing.

[0181] Through the above steps, this embodiment achieves hourly-level automatic consistency verification and error correction, and can detect and automatically repair data inconsistencies caused by various anomalies (such as program bugs, human error, network partitions, and message loss).

[0182] The technical solution in this embodiment realizes automatic consistency verification and error correction between multiple data sources through the verification table and the hourly scheduled task. It can detect and automatically repair data inconsistencies in a timely manner, which greatly reduces the manual investigation and repair costs for operation and maintenance personnel and ensures long-term data consistency between multiple data sources.

[0183] Furthermore, to achieve the above objectives, this application also proposes a multi-source data synchronization system, such as... Figure 9 As shown, the system includes: a source data source client and a target data source client, wherein: The source data source client is used to respond to write requests to the source data source, determine whether dual write is required based on the dual write switch status and dual write conditions, and if so, generate a dual write signal; record a dual write guarantee record containing a unique sequence number and initial state in the guarantee data table of the source data source; send a dual write signal for the target data source client to perform dual write processing; receive a dual write processing result confirmation message returned by the target data source client, the confirmation message containing the unique sequence number and processing status; and update the status of the corresponding record in the guarantee data table according to the confirmation message. The target data source client is configured to receive a double-write signal from the source data source client, perform double-write processing based on the double-write signal, and return a double-write processing result confirmation message to the source data source client.

[0184] The multi-data source data synchronization system provided in this application employs the multi-data source data synchronization method described in the above embodiments to ensure data synchronization consistency and integrity in high-concurrency scenarios, and to reduce or avoid performance impact on business systems. Compared with the prior art, the beneficial effects of the multi-data source data synchronization system provided in this application are the same as those of the multi-data source data synchronization method described in the above embodiments, and other technical features of the multi-data source data synchronization system are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0185] In addition, to achieve the above objectives, this application also proposes a multi-source data synchronization device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the multi-source data synchronization method described above.

[0186] The following is for reference. Figure 10 The diagram illustrates a structural schematic suitable for implementing a multi-source data synchronization device in the embodiments of this application. The multi-source data synchronization device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 10 The multi-data source data synchronization device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0187] like Figure 10As shown, the multi-source data synchronization device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in the read-only memory 1002 or a program loaded from the storage device 1003 into the random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the multi-source data synchronization device. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the multi-source data synchronization device to communicate wirelessly or wiredly with other devices to exchange data. Although the figure shows a multi-source data synchronization device with various systems, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0188] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0189] The multi-data source data synchronization device provided in this application employs the multi-data source data synchronization method described in the above embodiments to ensure data synchronization consistency and integrity in high-concurrency scenarios, and to reduce or avoid performance impact on business systems. Compared with the prior art, the beneficial effects of the multi-data source data synchronization device provided in this application are the same as those of the multi-data source data synchronization method described in the above embodiments, and other technical features of this multi-data source data synchronization device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0190] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0191] In addition, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the multi-data source data synchronization method described above.

[0192] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0193] The aforementioned computer-readable storage medium may be included in a multi-data source data synchronization device; or it may exist independently and not be assembled into a multi-data source data synchronization device.

[0194] The aforementioned computer-readable storage medium carries one or more programs. When these programs are executed by a multi-data source data synchronization device, the multi-data source data synchronization device: responds to a write request from the source data source client; determines whether a dual write operation is needed based on the dual write switch status and dual write conditions; if needed, generates a dual write signal; records a dual write guarantee record containing a unique sequence number and initial state in the guarantee data table of the source data source; sends a dual write signal for the target data source client to perform dual write processing; receives a dual write processing result confirmation message returned by the target data source client, the confirmation message containing the unique sequence number and processing status; and updates the status of the corresponding record in the guarantee data table according to the confirmation message. This solution addresses the performance issues arising from ensuring consistency during data synchronization between multiple data sources by adding a guarantee data table during data synchronization; simultaneously, it ensures data synchronization consistency and integrity between data sources in high-concurrency scenarios.

[0195] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0196] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0197] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0198] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., computer programs) for executing the above-described multi-data source data synchronization method. This ensures data synchronization consistency and integrity in high-concurrency scenarios and reduces or avoids performance impact on business systems. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the multi-data source data synchronization method provided in the above embodiments, and will not be repeated here.

[0199] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the multi-data source data synchronization method described above.

[0200] The computer program product provided in this application guarantees data synchronization consistency and integrity in high-concurrency scenarios, and reduces or avoids performance impact on business systems. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the multi-data source data synchronization method provided in the above embodiments, and will not be repeated here.

[0201] The one or more technical solutions proposed in this application have at least the following technical effects: On the source data source client side, dual-write triggering and tracking are achieved through steps such as dual-write switch judgment, dual-write signal generation, ensuring data table recording, signal sending, and status confirmation; on the target data source client side, a complete closed loop of data synchronization is achieved through steps such as receiving dual-write signals, processing according to operation type (using a deletion record table and a delayed processing record table to solve out-of-order issues), background periodic task compensation, guaranteed dual-write retry, and hourly verification and error correction. This solution solves the performance problem caused by ensuring consistency during data synchronization between multiple data sources by adding a guarantee data table during data synchronization; at the same time, it ensures the consistency and integrity of data synchronization between data sources in high-concurrency scenarios. In addition, this solution solves the problem of difficulty in guaranteeing consistency during data synchronization between multiple data sources by adding a delayed processing record table and a deletion record table during data synchronization; this solution adds a verification table during data synchronization for hourly verification, solving the problem of not being able to automatically fill in and correct data inconsistencies in a timely manner during data synchronization between multiple data sources.

[0202] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A method for synchronizing data from multiple data sources, characterized in that, Applied to the source data source client, the method includes: In response to a write request to the source data source, determine whether to perform a dual write based on the dual write switch status and dual write conditions. If necessary, generate a dual write signal. Record a double-write protection record containing a unique sequence number and initial state in the protection data table of the source data source; Send a double-write signal to the target data source client to perform double-write processing; Receive a confirmation message of the double-write processing result returned by the target data source client, the confirmation message containing the unique sequence number and processing status; Based on the confirmation message, update the status of the corresponding record in the protection data table.

2. The method as described in claim 1, characterized in that, In response to a write request to the source data source, the system determines whether a dual write operation is needed based on the dual write switch status and dual write conditions. If needed, a dual write signal is generated, including: In response to a write request to the source data source, read the dual-write switch configuration parameters and determine whether the dual-write switch is enabled. If the dual-write switch is enabled, the dual-write object to be synchronized is obtained through the dual-write aspect; The dual-write objects are filtered according to preset dual-write conditions. If the dual-write objects meet the dual-write conditions, a dual-write signal is generated. The dual-write signal includes operation type, data primary key, data content, timestamp or version number.

3. The method as described in claim 2, characterized in that, Sending the dual-write signal to the target data source client for dual-write processing includes: According to the sending timing configured in the dual-write aspect, the dual-write signal is sent to the message middleware or directly to the target data source client so that the target data source client can perform dual-write processing.

4. The method as described in claim 3, characterized in that, The sending timing includes sending immediately or delaying until after the source data source transaction is committed. The step of sending the dual-write signal to the message middleware or directly to the target data source client according to the sending timing configured in the dual-write aspect, so that the target data source client can perform dual-write processing, includes: Based on the transmission timing configured in the dual-write section, determine whether to immediately send the dual-write signal; If the configured sending timing is to send immediately, the dual-write signal will be pushed to the message middleware or sent to the target data source client. If the configured sending timing is to delay until the source data source transaction is committed, the double-write signal is temporarily stored in the local queue and a transaction synchronization callback is registered. After the source data source transaction is successfully committed, the double-write signal is retrieved from the local queue and sent. If the source data source transaction is rolled back, the double write signal in the local queue is discarded.

5. The method as described in claim 4, characterized in that, Before sending the dual-write signal to the target data source client for dual-write processing, the method further includes: The dual-write aspect can be configured through annotations or configuration files. The configuration items include: the expression for the dual-write switch, the SpEL extraction expression for the dual-write object, the Boolean expression for dual-write condition filtering, and the enumeration value for the timing of sending.

6. The method as described in claim 1, characterized in that, The protection data table is located in the source data source, and the fields of the protection data table include: serial number, primary key, double-write content summary, creation timestamp, status, failure reason, and number of retries.

7. The method as described in claim 6, characterized in that, The step of updating the status of the corresponding record in the protection data table according to the confirmation message includes: If the confirmation message indicates successful processing, the status of the corresponding record in the security data table will be updated to "processed". If the confirmation message indicates a processing failure, the number of retries is incremented, and an alarm is triggered when the number of retries exceeds a preset threshold.

8. A method for synchronizing data from multiple data sources, characterized in that, Applied to the target data source client, the method includes: Receive a dual-write signal from the source data source client. The dual-write signal is generated by the source data source client in response to a write request to the source data source, based on the dual-write switch status and dual-write conditions. Perform dual-write processing based on the dual-write signal; A confirmation message for the dual-write processing result is returned to the source data source client. The confirmation message includes the unique sequence number and processing status. The source data source client updates the status of the corresponding record in the guarantee data table of the source data source according to the confirmation message. Before sending the dual-write signal, the source data source client records a dual-write guarantee record in the guarantee data table that includes a unique sequence number and an initial status.

9. The method as described in claim 8, characterized in that, The dual-write processing based on the dual-write signal includes: The operation type of the dual write signal is analyzed, and the operation type includes write operation, update operation, or delete operation; Write, update, or delete operations are executed according to the operation type, and dual-write synchronization is performed using a delete record table and a delay processing record table. The delete record table is used to store the primary key of the data that has been successfully deleted and the deletion timestamp. The delay processing record table is used to temporarily store update or delete operations that cannot be executed because the data to be operated on does not exist in the target data source.

10. The method as described in claim 9, characterized in that, The dual-write signal includes operation type, data primary key, data content, timestamp or version number. The process of executing write, update, or delete operations according to the operation type, and using a delete record table and a delay processing record table for dual-write synchronization, includes: If the operation type is a write operation, then query whether there is a deleted record with the same primary key in the local deleted record table of the target data source; if not, then perform the write operation; if it exists, then compare the write timestamp in the double write signal with the deletion timestamp in the deleted record. If the write timestamp is later than the deletion timestamp, then perform the write operation; otherwise, discard the write signal. If the operation type is an update operation, then query whether there is a data record in the target data source that has the same primary key as the data; if not, then store this update operation in the local delayed processing record table; if it exists, then compare the version number or timestamp in the double write signal with the version number or timestamp of the existing record in the target data source, wherein, when the version number or timestamp in the double write signal is updated, optimistic locking is used to perform the update, otherwise the update is abandoned; If the operation type is a deletion operation, then query whether there is a data record in the target data source that has the same data primary key; if not, then store this deletion operation in the delayed processing record table; if it exists, then compare the timestamp in the double write signal with the write timestamp of the record in the target data source, and perform deletion when the deletion timestamp is later than the write timestamp, and insert a record containing the data primary key and the deletion timestamp into the deletion record table after successful deletion.

11. The method as described in claim 10, characterized in that, The dual-write processing based on the dual-write signal further includes at least one of the following: Initiate the first background cycle task, scan the delayed processing record table, and compensate for any unprocessed update or deletion operations. Initiate the second background cycle task to scan the protection data table in the source data source and perform compensatory double writes for double write protection records that have timed out and have not been confirmed. The received double-write signal is recorded in the verification table, and the third timed task is started to perform consistency verification and automatic error correction between the source data source and the target data source at the top of the hour.

12. The method as described in claim 11, characterized in that, The first background cycle task is initiated, which scans the delayed processing record table and performs compensation processing on any unprocessed update or deletion operations, including: Start the first background cycle task to scan the delayed processing record table for unprocessed update or delete operations; If it is an update operation, the latest data is retrieved from the source data source based on the data primary key; if the data does not exist in the source data source, the delayed record is deleted and the process ends; if the data exists in the source data source, the target data source is queried; if the data does not exist in the target data source, a write operation is performed; if the data exists in the target data source, an update is performed after version comparison; the delayed record is deleted after processing is complete. If it is a deletion operation, the deletion record table is queried. If there is a record with the same primary key and the deletion time is not earlier than the time of the current deletion operation, the processing ends. Otherwise, the target data source is queried. If the data to be deleted does not exist in the target data source, the process ends. If the data to be deleted exists, the deletion time is compared. If the current deletion time is later than the write time in the target data source, the deletion is performed. The delayed record is deleted after the processing is completed.

13. The method as described in claim 11, characterized in that, The initiation of the second background cycle task involves scanning the protection data table in the source data source and performing compensatory double writes on the double write protection records that have timed out and not been confirmed. Start the second background cycle task to periodically scan the protection data table in the source data source and find double-write protection records whose status is pending and whose creation time exceeds the preset threshold. For each record found, reconstruct the double-write signal based on the double-write content recorded therein; The local client's double-write processing logic is invoked to re-execute the double-write operation for compensation. If the compensation process is successful, a confirmation message is sent to the source data source client so that the source data source client updates the status of the corresponding protection record to "processed". If multiple compensation attempts fail, an alarm is triggered.

14. The method as described in claim 11, characterized in that, The process of recording the received dual-write signal to the verification table and starting the third timed task to perform consistency verification and automatic error correction between the source data source and the target data source at the top of each hour includes: For each received double-write signal, record a verification message in the local verification table. The verification message includes at least the data primary key, operation timestamp, and data content hash value. The third scheduled task is initiated to scan all verification records in the verification table at each preset hour, and to perform the following verification operations on each verification record: Query the data content and timestamp of the same primary key in the source data source and the target data source respectively, and compare whether the two are consistent; If they match, delete the verification record. If the data content is inconsistent or the timestamp difference exceeds the allowable range, the error correction operation will be performed based on the data from the source data source.

15. The method as described in claim 14, characterized in that, The error correction operation includes: If the source data source exists but the target data source does not exist, then write; If the source data source does not exist but the target data source does exist, then delete it. If both the source data source and the target data source exist but their data content is different, then update the target data source to the content of the source data source; After the error correction is completed, delete the verification record and record the error correction log.

16. A multi-source data synchronization system, characterized in that, The system includes: a source data source client and a target data source client, wherein: The source data source client is used to respond to write requests to the source data source, determine whether dual write is required based on the dual write switch status and dual write conditions, and if so, generate a dual write signal; record a dual write guarantee record containing a unique sequence number and initial state in the guarantee data table of the source data source; send a dual write signal for the target data source client to perform dual write processing; receive a dual write processing result confirmation message returned by the target data source client, the confirmation message containing the unique sequence number and processing status; and update the status of the corresponding record in the guarantee data table according to the confirmation message. The target data source client is configured to receive a double-write signal from the source data source client, perform double-write processing based on the double-write signal, and return a double-write processing result confirmation message to the source data source client.

17. A multi-source data synchronization device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the multi-data source data synchronization method as described in any one of claims 1 to 15.

18. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the multi-data source data synchronization method as described in any one of claims 1 to 15.

19. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the multi-data source data synchronization method as described in any one of claims 1 to 15.

Images