Data synchronization method, device and equipment and computer readable medium

Abstract

The invention relates to a data synchronization method, device and equipment and a computer readable medium. The method comprises the steps that under the condition that the synchronization process of a target database to a source database is abnormal, a trigger node of the abnormality is determined; obtaining a target request chain from the time of triggering the node to the time of enabling the source database to reach the latest state, the request chain being used for representing a service request recorded according to the request time; and executing the target request chain on the target database to re-synchronize the data to the target database according to the same service logic as the source database. According to the method, the corresponding complete service chain starting from the abnormal trigger node is found, and the complete service chain is executed in the target database, so that the target database synchronizes the abnormal point and the subsequent service data again according to the same service logic as the source database; the technical problem that the problems of logic abnormity and data inconsistency caused by the failure of the synchronization rule cannot be solved through breakpoint resume is solved.

Description

Technical Field

[0001] The present application relates to the field of Internet technology, and in particular to a data synchronization method, apparatus, device, and computer-readable medium. Background Art

[0002] In the information age, data synchronization is a key technology for achieving data consistency and business continuity across different databases. Currently, incremental synchronization is widely used as a highly efficient data synchronization method. Its core principle is to synchronize only the data that has changed in the source database. By using change data capture technology, it significantly reduces data transmission volume and improves synchronization efficiency. This technology has important application value in scenarios such as distributed systems and multi-database clusters.

[0003] However, existing incremental synchronization technology faces significant challenges in practical applications, especially when the source database structure is dynamically adjusted, such as when new fields are added or table structures are changed, and when network interruptions occur, the synchronization rules may become invalid. The logical anomalies and data inconsistencies caused by the failure of synchronization rules cannot be resolved by resuming the data from the breakpoint, which ultimately leads to the inability to synchronize some changed data correctly, and will seriously affect reverse synchronization and business in the future.

[0004] Currently, no effective solution has been proposed to the problem that logical anomalies and data inconsistencies caused by the failure of synchronization rules cannot be solved by resuming the data. Summary of the Invention

[0005] The present application provides a data synchronization method, apparatus, device and computer-readable medium to solve the technical problem that logical anomalies and data inconsistencies caused by failure of synchronization rules cannot be solved by resuming transmission from a breakpoint.

[0006] According to one aspect of an embodiment of the present application, the present application provides a data synchronization method, including: when an exception occurs in the synchronization process of a target database with a source database, determining the trigger node of the exception; obtaining a target request chain starting from the trigger node to make the source database reach the latest state, wherein the request chain is used to represent business requests recorded according to the request time; executing the target request chain on the target database to re-synchronize the data to the target database according to the same business logic as the source database.

[0007] Optionally, determining the triggering node of this exception includes any one of the following: determining the recording time of this exception in the exception log, and determining the recording time as the triggering node; determining the transaction identifier of this exception in the exception log, and determining the generation time of the transaction identifier as the triggering node; determining the triggering statement of this exception in the exception log, and determining the triggering statement as the triggering node.

[0008] Optionally, obtaining a target request chain starting from a trigger node to bringing the source database to the latest state includes: obtaining a request tracking table, wherein the request tracking table is used to record each business request in the source database according to the request time and track the completion status of each business request; determining the target request corresponding to the trigger node in the request tracking table; taking all business requests from the target request to the last request in the request tracking table as a first request chain; deleting the business request marked as incomplete from the first request chain to obtain a second request chain; and determining the second request chain as the target request chain.

[0009] Optionally, after obtaining the request tracking table, the method further includes: obtaining a first completion status of each business request recorded in the source database, and determining a second completion status of each business request in the request tracking table; when the first completion status and the second completion status of any business request are inconsistent, determining the target transaction to which the business request belongs; resubmitting the target transaction to the source database so that the source database re-updates the completion status of the target transaction; and recording the latest completion status of the target transaction by the source database in the request tracking table.

[0010] Optionally, after obtaining the second request chain, the method further includes: when any target business request in the second request chain has a predecessor business request with a dependent relationship, inserting the predecessor business request before the position of the target business request to obtain a third request chain; and determining the third request chain as the target request chain.

[0011] Optionally, after executing the target request chain on the target database to re-synchronize the data to the target database according to the same business logic as the source database, the method further includes: calculating a first check value of the data in the source database and a second check value of the data in the target database according to a preset period; if the first check value and the second check value in any period are inconsistent, issuing an alarm message to prompt the target object to check whether the request tracking table is damaged, and / or prompting the target object to check whether the business logic and data synchronization logic are abnormal.

[0012] Optionally, the method further includes determining that an abnormality has occurred in the synchronization process between the target database and the source database in accordance with at least one of the following methods: determining that an abnormality has occurred in the synchronization process between the target database and the source database in the event of a network interruption during the synchronization process; determining that an abnormality has occurred in the synchronization process between the target database and the source database in the event of a detected inconsistency in the synchronized data; determining that an abnormality has occurred in the synchronization process between the target database and the source database in the event of a detected inconsistency in the logical chain of the synchronized data; and determining that an abnormality has occurred in the synchronization process between the target database and the source database in the event of a detected change in the structure of the source database.

[0013] According to another aspect of an embodiment of the present application, the present application provides a data synchronization device, including: an exception capture module, used to determine the trigger node of the exception when an exception occurs in the synchronization process of the target database with the source database; a business chain acquisition module, used to obtain the target request chain starting from the trigger node to make the source database reach the latest state, wherein the request chain is used to represent the business request recorded according to the request time; a remediation module, used to execute the target request chain on the target database to re-synchronize the data to the target database according to the same business logic as the source database.

[0014] According to another aspect of an embodiment of the present application, the present application provides an electronic device, including a memory, a processor, a communication interface and a communication bus, wherein the memory stores a computer program that can be run on the processor, the memory and the processor communicate through the communication bus and the communication interface, and the steps of the above method are implemented when the processor executes the computer program.

[0015] According to another aspect of an embodiment of the present application, the present application further provides a computer-readable medium having a non-volatile program code executable by a processor, where the program code enables the processor to execute the above method.

[0016] The above technical solution provided by the embodiment of the present application has the following advantages compared with the related art:

[0017] The present application provides a data synchronization method, including: when an exception occurs in the synchronization process between the target database and the source database, determining the trigger node of the exception; obtaining a target request chain starting from the trigger node to make the source database reach the latest state, wherein the request chain is used to represent the business request recorded according to the request time; executing the target request chain on the target database to re-synchronize the data to the target database according to the same business logic as the source database. The present application finds the complete business chain corresponding to the trigger node of the exception when an exception occurs in the data synchronization process, and executes the complete business chain in the target database, thereby re-synchronizing the target database with the exception point and subsequent business data according to the same business logic as the source database. Even if occasional exceptions cause logical exceptions and data inconsistencies in the data synchronized by the target database, complete and correct synchronization can be achieved by re-overwriting the complete business chain, solving the technical problem that logical exceptions and data inconsistencies caused by the failure of synchronization rules cannot be solved by breakpoint resumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0020] Figure 1 A schematic diagram of a hardware environment for an optional data synchronization method provided according to an embodiment of the present application;

[0021] Figure 2 A flowchart of an optional data synchronization method provided according to an embodiment of the present application is provided;

[0022] Figure 3 A block diagram of an optional data synchronization device provided according to an embodiment of the present application;

[0023] Figure 4 A schematic diagram of an optional electronic device structure provided in an embodiment of the present application. DETAILED DESCRIPTION

[0024] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0025] In the subsequent description, the suffixes such as "module", "component" or "unit" used to represent elements are only used to facilitate the description of this application and have no specific meaning. Therefore, "module" and "component" can be used interchangeably.

[0026] Existing incremental synchronization technology faces significant challenges in practical applications. This is especially true when the source database structure undergoes dynamic adjustments, such as adding new fields or changing table structures, and a network outage occurs. This can cause synchronization rules to fail. For example, if a new "discount" field is added to the source database, but the target database does not receive the corresponding DDL statement due to network issues, the structure is not updated. When the incremental synchronization module attempts to write data containing the "discount" field to the target database, it fails because the target database does not have the "discount" field. This non-existence of the target database constitutes a synchronization rule failure. Furthermore, synchronization rule failures can occur when data is synchronized but the content is incorrect, such as missing or tampered fields; when the synchronization location is correct but the data logic is incomplete, such as uncommitted transactions or broken dependencies; and when data is successfully written to the target database but business logic conflicts occur, such as duplicate primary keys or foreign key constraint failures. Logical anomalies and data inconsistencies caused by synchronization rule failures cannot be resolved through breakpoint resuming, ultimately resulting in the inability to synchronize some changed data correctly, which can severely impact subsequent reverse synchronization and business operations.

[0027] In order to solve the problems mentioned in the background technology, according to one aspect of the embodiments of the present application, an embodiment of a data synchronization method is provided.

[0028] Optionally, in the embodiment of the present application, the above data synchronization method can be applied to Figure 1 In the hardware environment composed of the terminal 101 and the server 103 shown in FIG. Figure 1 As shown, the server 103 is connected to the terminal 101 via a network and can be used to provide services for the terminal or a client installed on the terminal. A database 105 can be set on the server or independently of the server to provide data storage services for the server 103. The above-mentioned network includes but is not limited to: a wide area network, a metropolitan area network or a local area network, and the terminal 101 includes but is not limited to a PC, a mobile phone, a tablet computer, etc.

[0029] A data synchronization method in the embodiment of the present application can be executed by the server 103, such as Figure 2 As shown, the method may include the following steps:

[0030] Step S202: When an exception occurs during synchronization between the target database and the source database, a triggering node of the exception is determined;

[0031] Step S204: Obtain a target request chain from the triggering node to the source database reaching the latest state, wherein the request chain is used to represent the business requests recorded according to the request time;

[0032] Step S206 : executing the target request chain on the target database to resynchronize the data to the target database according to the same business logic as the source database.

[0033] In the embodiment of the present application, the source database refers to a database that is directly connected to the business system, and changes in the business system are directly reflected in the source database. The target database refers to a database used to back up the source database. By monitoring the data changes in the source database in real time, the target database can synchronize the data updated by the source database to the target database based on incremental synchronization technology, thereby realizing the backup of the source database. The target database can be used to synchronize the data on the target database to the source database through reverse synchronization when the source database fails, thereby avoiding data loss caused by the failure of the source database. It can also be used to access the business system when the source database cannot provide services and provide business services instead of the source database.

[0034] In this embodiment of the application, a request chain refers to a logical sequence of business requests in chronological order, where each request corresponds to a data change operation on the source database, such as an SQL statement. Business logic refers to the rules by which the source database processes business requests, such as the logic of "verify inventory before executing order insertion."

[0035] In an embodiment of the present application, when an exception occurs in the synchronization process between the target database and the source database (such as a network interruption or data inconsistency), the system locates the starting point of the exception through the exception log. The trigger node includes but is not limited to key information such as the timestamp, transaction ID, error statement, and sequence number in the exception log. For example, if the exception log records "network interruption at 14:30 on June 5, 2025 caused synchronization interruption", the trigger node is this time point; if the exception log records "XXX statement execution error caused synchronization exception", the trigger node is this statement. Starting from the trigger node, the system extracts all subsequent business requests in the source database to form a continuous request chain. The request chain is a sequence of business operations recorded in chronological order, such as "user A inserts an order → user B updates the order status → user C deletes the order". The target request chain is executed one by one on the target database according to the business logic of the source database. For example, if the source database executes "insert product data → modify price → delete invalid records" after the trigger node, the target database re-executes these three operations in the same order, thereby re-synchronizing the target database with the exception point and subsequent business data according to the same business logic as the source database.

[0036] Through steps S202 to S206, the present application finds the complete business chain corresponding to the triggering node of the exception when an exception occurs during the data synchronization process, and executes the complete business chain in the target database, thereby resynchronizing the target database with the exception point and subsequent business data according to the same business logic as the source database. Even if occasional exceptions cause logical exceptions and data inconsistencies in the data synchronized by the target database, complete and correct synchronization can be achieved by re-overwriting the complete business chain, solving the technical problem that logical exceptions and data inconsistencies caused by failure of synchronization rules cannot be solved by resuming transmission from breakpoints.

[0037] In an embodiment of the present application, the target request chain can also be obtained through the message queue backtracking mechanism. Specifically, each business request of the source database is synchronously sent to the message queue when it is generated, and the message carries the request time, content and transaction identifier; after the exception is triggered, all messages after the node that have not been confirmed to be received by the target database are pulled from the message queue according to the triggering node timestamp or transaction identifier as the target request chain. In this way, when the request tracking table is damaged due to a database failure, the persistent record of the message queue can be used as a backup data source to ensure the reliability of the request chain acquisition.

[0038] In an optional embodiment, determining the triggering node of this exception includes any of the following:

[0039] Determine the recording time of this exception in the exception log and determine the recording time as the trigger node;

[0040] Determine the transaction ID of this exception in the exception log, and determine the generation time of the transaction ID as the trigger node;

[0041] The trigger statement of this exception is determined in the exception log, and the trigger statement is determined as the trigger node.

[0042] In the embodiments of the present application, the transaction identifier refers to the unique identifier generated by the source database for each business transaction, which is used to track the execution status of the transaction. The trigger statement refers to the specific database operation statement that causes the synchronization exception, such as INSERT, UPDATE, DELETE statements, etc.

[0043] In this embodiment of the present application, the trigger node can be determined based on the time the synchronization anomaly was recorded. This means extracting the timestamp of the anomaly from the anomaly log. For example, if the log record "2025-06-05 14:35:00 Data verification failed" is included, "14:35:00" will be used as the trigger node. This method of determining the trigger node based on the time the synchronization anomaly was recorded is suitable for time-sensitive anomalies such as network outages, as the outage point can be quickly located using the timestamp.

[0044] In an embodiment of the present application, the trigger node can be determined based on the transaction identifier of the transaction being processed when the synchronization exception occurs, that is, if the exception log contains a transaction ID (such as "TransactionID:TX202506050123"), then the generation time (such as "14:32:15") is queried in the source database based on the transaction ID, and the time is determined as the trigger node. The transaction ID can also be directly determined as the trigger node, and the exception remediation needs to find the business request related to the transaction ID. The target transaction can also be determined based on the transaction ID, and the submission time of the target transaction is determined as the trigger node. The method of determining the trigger node based on the transaction identifier of the transaction being processed when the synchronization exception occurs is applicable to transaction-level exceptions, such as when the transaction is interrupted halfway through execution, and the transaction ID is associated with the specific operation time to ensure that the remediation starts from the transaction.

[0045] In this embodiment, the trigger node can be determined based on the trigger statement. This means directly extracting the error statement from the exception log, such as "UPDATE orders SET status = 'invalid' WHERE id = 123 failed to execute," and using that SQL statement as the trigger node. This approach is suitable for statement-level exceptions, such as syntax errors and insufficient permissions, allowing for direct remediation of the error statement and avoiding repetitive execution of irrelevant requests.

[0046] In an embodiment of the present application, the trigger node can also be determined based on the business logic relevance. Specifically, the error information in the exception log can be parsed to identify the business entity related to the exception; the latest operation record of the business entity with business logic relevance is queried in the source database, and the execution time or transaction ID of the operation is used as the trigger node. For example, if "User 123" and "Order Status Update" are recorded in the exception log, the latest record of the order operation by "User 123" is queried, and the generation time of the latest record or the transaction ID of the corresponding transaction is used as the trigger node. When the exception log does not clearly record the timestamp or transaction ID, the trigger node can be quickly located through the business entity association, which is suitable for scenarios with complex business logic and long request chains.

[0047] This application uses a multi-dimensional trigger node positioning method to adapt to different types of abnormal scenarios and improve the flexibility and accuracy of abnormal positioning.

[0048] This application adds a separate request tracking table to fully record the business logic executed by the source database. Therefore, even if a synchronization anomaly occurs that cannot be resolved by resuming the download, the anomaly can be resolved through business chain coverage. The following describes the request tracking table and the method for resolving synchronization anomalies that cannot be resolved by resuming the download.

[0049] In an optional embodiment, obtaining a target request chain starting from a triggering node and bringing the source database to the latest state includes:

[0050] Step 1: Obtain a request tracking table, where the request tracking table is used to record each business request in the source database according to the request time and track the completion status of each business request;

[0051] Step 2: Determine the target request corresponding to the triggering node in the request tracking table;

[0052] Step 3: All business requests from the target request to the last request in the request tracking table are taken as the first request chain;

[0053] Step 4: Deleting the business request marked as unfinished from the first request chain to obtain a second request chain;

[0054] Step 5: Determine the second request chain as the target request chain.

[0055] In the embodiments of the present application, the request tracking table, request_tracking, refers to a metadata table in the source database used to record business requests. It contains fields such as the request time, content, and status, as well as the start and completion of the request execution, and is used to track the execution of the request. Whenever the source database performs a business operation, a record of the business operation is first inserted into the request tracking table. At this time, the status of the business operation is incomplete. When the source database reports that the business operation is completed, the status of the business operation in the request tracking table is updated to complete.

[0056] In the embodiment of the present application, since the request tracking table fully records the business logic executed by the source database, after determining the trigger node of the synchronization anomaly, the target request corresponding to the trigger node can be found in the request tracking table. For example, if the trigger node is the timestamp "14:35:00", the business request being executed at the moment of "14:35:00" is found in the request tracking table as the target request corresponding to the trigger node. Starting from the target request, all records up to the last request in the table are extracted to form a first request chain. The business request with an unfinished status is deleted from the first request chain to obtain a second request chain containing only business requests with a completed status, and the second request chain is used as the target request chain that is finally executed. When a business request is completed, the corresponding business operation is not successfully executed, and the unfinished business request will not change the business status, and will not change the business data in the source database. Therefore, it is only necessary to put the completed business requests that have an impact on the business data in the source database into the target database for execution to ensure that data synchronization is performed according to the same business logic as the source database.

[0057] This application adds a separate request tracking table to fully record the business logic executed by the source database. This ensures that even if occasional exceptions cause logical anomalies and data inconsistencies in the data synchronized with the target database, complete and correct synchronization can be achieved by re-overwriting the entire business chain. This solves the technical problem that logical anomalies and data inconsistencies caused by synchronization rule failure cannot be resolved through breakpoint resumption. Furthermore, by filtering the status of the request tracking table, it ensures that only successfully completed requests are re-executed, avoiding invalid operations and data conflicts, and improving remediation efficiency and accuracy.

[0058] In this application, conflicts may also occur between the source database and the request tracking table. For example, in a scenario where a user pays an order, when the source database confirms receipt of the payment information and needs to update the order status from "unpaid" to "paid", a record of the update statement update operation is inserted into the request tracking table and marked as "start". At this time, if the network interruption causes the source database order status to fail to update, that is, it is still "unpaid", then the update statement in the request tracking table will be marked as "incomplete". The source database will trigger remedial measures for such payment transaction anomalies, such as manual intervention or asynchronous retry, so that the source database updates the status of the order to "paid", but the request tracking table still shows that the update statement update operation is "incomplete", resulting in a conflict.

[0059] In an embodiment of the present application, in order to avoid similar conflicts that may cause inconsistencies in the business logic of the source database and the request tracking table, and further cause inconsistencies in the business logic of the target database and the source database, resulting in data synchronization anomalies, the present application provides a conflict coverage method, which is explained below.

[0060] In an optional embodiment, after obtaining the request tracking table, the method further includes:

[0061] Step 1: Obtain the first completion status of each business request recorded in the source database, and determine the second completion status of each business request in the request tracking table;

[0062] Step 2: When the first completion status and the second completion status of any business request are inconsistent, determine the target transaction to which the business request belongs;

[0063] Step 3: Resubmit the target transaction to the source database so that the source database can re-update the completion status of the target transaction;

[0064] Step 4: Record the latest completion status of the target transaction by the source database into the request tracking table.

[0065] In an embodiment of the present application, after obtaining the request tracking table, it is possible to check whether there is a business logic conflict between the source database and the request tracking table. If there is a conflict, regardless of the current status of the business request corresponding to the conflict in the source database, the transaction to which the business request belongs is re-executed, and then the request tracking table tracks the latest status to ensure that the business logic of the source database and the request tracking table are completely consistent. Specifically, the completion status of each business request is obtained from the source database and the request tracking table respectively. For example, if the status of the "order insertion request" in the source database is "completed", and the status of the request in the request tracking table is "uncompleted", then the status is determined to be inconsistent. According to the inconsistent request, the transaction to which it belongs is queried. For example, the order transaction with transaction ID TX20250605002 contains two requests, "order insertion" and "inventory deduction". The target transaction is resubmitted to the source database, triggering the source database to re-execute all requests in the transaction and update the completion status of the transaction. At this time, the request tracking table tracks the latest status of the target transaction to ensure that the two are consistent.

[0066] This application solves the inconsistency problem between the business logic of the source database and the request tracking table by performing a consistency comparison on the business logic of the source database and the request tracking table, overwriting the existing conflicts, ensuring that the remediation process is based on the accurate and consistent business logic of the source database and the request tracking table, and avoiding synchronization anomalies caused by conflicts in the business logic of the source database and the request tracking table.

[0067] In this application, there are dependencies between some business requests. For example, the request to write data containing a discount field depends on the request to add a new discount field. If the source database adds a new field discount due to accidental reasons, the target database does not receive the relevant DDL statement and the structure is not updated. Then, when the incremental synchronization module tries to write data containing a discount field to the target database, it will fail because the target database does not have the discount field. Therefore, in order to solve the synchronization anomaly caused by the lack of dependency, the preceding business request that the target business request depends on can be inserted into the target request chain based on the above embodiment, so that each business request in the target request chain has a complete dependency relationship, avoiding synchronization anomalies caused by the lack of dependency relationships. This is explained below.

[0068] In an optional embodiment, after obtaining the second request chain, the method further includes:

[0069] Step 1: If any target service request in the second request chain has a predecessor service request with a dependency relationship, insert the predecessor service request before the target service request to obtain a third request chain;

[0070] Step 2: Determine the third request chain as the target request chain.

[0071] In an embodiment of the present application, by traversing each target business request in the second request chain, it is checked whether there is a preceding dependent request. For example, the request to write data containing the discount field depends on the request to add a new field discount field, and the request to update the order status depends on the request to insert the order. If the request chain only contains the target business request but not the preceding business request of the target business request, there is a missing dependency relationship, and it is necessary to insert the corresponding preceding business request before the target business request to complete the dependency relationship. If the preceding business request is located after the target business request, there is a logical conflict, and the position of the preceding business request needs to be changed to before the target business request. Finally, the adjusted request chain is used as the final target request chain to ensure that the dependency relationship of each business request in the target request chain is complete and conforms to the business logic.

[0072] This application completes dependencies to avoid synchronization failures caused by logical dependencies.

[0073] In an optional embodiment, after executing the target request chain on the target database to resynchronize the data to the target database according to the same business logic as the source database, the method further includes:

[0074] Step 1: Calculate a first checksum of the data in the source database and a second checksum of the data in the target database according to a preset period;

[0075] Step 2: When the first check value and the second check value are inconsistent in any period, an alarm message is issued to prompt the target object to check whether the request tracking table is damaged and / or to prompt the target object to check whether the business logic and data synchronization logic are abnormal.

[0076] In the embodiment of the present application, data integrity checks can be performed regularly to compare the data in the source database and the target database to ensure the consistency and integrity of the data. Specifically, hash algorithms such as MD5, SHA-256, etc. are used to calculate hash check values ​​for the full data of the source database and the target database or the data of a certain incremental synchronization, and the two check values ​​are compared. If they are inconsistent, it means that there is a synchronization anomaly, and the above embodiment can be used for remediation based on business chain coverage. If the two check values ​​are still calculated to be inconsistent after remediation, it means that the system may encounter one of the following situations: the request tracking table itself is damaged, such as data loss or error; extreme anomalies not covered by the remediation logic, such as the target database is written successfully but no feedback is given; conflicts between the business logic and the data synchronization logic, such as concurrent operations resulting in dirty data. At this time, it is judged that the system cannot be automatically repaired, and the alarm process needs to be triggered to prompt the operation and maintenance personnel to check whether the request tracking table is damaged, or to check whether the business logic and data synchronization logic are abnormal.

[0077] This application promptly discovers potential data inconsistencies through regular verification, quickly locates the source of the problem through an alarm mechanism, and implements full-cycle monitoring of data synchronization, which can improve the reliability and stability of incremental data synchronization and reduce business risks.

[0078] In an optional embodiment, the method further includes determining that an abnormality occurs in the synchronization process between the target database and the source database in at least one of the following ways:

[0079] In the event of a network interruption during synchronization, it is determined that the synchronization process of the target database with the source database is abnormal;

[0080] When inconsistency in the synchronized data is detected, it is determined that an abnormality has occurred in the synchronization process of the target database with the source database;

[0081] When an inconsistency in the logical chain of synchronized data is detected, it is determined that an abnormality has occurred in the synchronization process of the target database with the source database;

[0082] When a change in the structure of the source database is detected, it is determined that an abnormality occurs in the synchronization process of the target database with the source database.

[0083] In the embodiment of the present application, incremental synchronization provides a breakpoint resume function, which can solve the synchronization interruption recovery at the physical level, such as network interruption, but the synchronization interruption at the logical level cannot be solved by breakpoint resume. The data synchronization method proposed in the present application can cover the logical anomalies and data consistency problems that cannot be solved by breakpoint resume.

[0084] In the embodiments of the present application, synchronization anomalies can be determined as physical-level synchronization interruptions, such as network interruptions, as well as logical-level synchronization interruptions, such as inconsistent synchronized data, which may be caused by the loss or tampering of some fields after data synchronization; correct synchronization positions but incomplete data logic, which may be caused by uncommitted transactions and broken dependencies; changes in the source database structure, such as the addition of new data tables and fields; and successful writes to the target database but business logic conflicts, which may be caused by duplicate primary keys and failed foreign key constraints. For abnormal situations such as physical-level synchronization interruptions and logical-level synchronization interruptions, synchronization anomalies can be resolved through remediation based on business chain coverage, thereby improving the reliability and stability of incremental data synchronization and reducing business risks.

[0085] The present application provides a data synchronization method, including: when an exception occurs in the synchronization process between the target database and the source database, determining the trigger node of the exception; obtaining a target request chain starting from the trigger node to make the source database reach the latest state, wherein the request chain is used to represent the business request recorded according to the request time; executing the target request chain on the target database to re-synchronize the data to the target database according to the same business logic as the source database. The present application finds the complete business chain corresponding to the trigger node of the exception when an exception occurs in the data synchronization process, and executes the complete business chain in the target database, thereby re-synchronizing the target database with the exception point and subsequent business data according to the same business logic as the source database. Even if occasional exceptions cause logical exceptions and data inconsistencies in the data synchronized by the target database, complete and correct synchronization can be achieved by re-overwriting the complete business chain, solving the technical problem that logical exceptions and data inconsistencies caused by the failure of synchronization rules cannot be solved by breakpoint resumption.

[0086] According to another aspect of the embodiment of the present application, Figure 3 As shown, a data synchronization device is provided, comprising:

[0087] The exception capture module 301 is used to determine the triggering node of the exception when an exception occurs during the synchronization process between the target database and the source database;

[0088] The service chain acquisition module 303 is used to acquire a target request chain starting from the trigger node to the target state of the source database, wherein the request chain is used to represent the service requests recorded according to the request time;

[0089] The remediation module 305 is configured to execute a target request chain on the target database to resynchronize data to the target database according to the same business logic as that of the source database.

[0090] It should be noted that the exception capture module 301 in this embodiment can be used to execute step S202 in the embodiment of the present application, the business chain acquisition module 303 in this embodiment can be used to execute step S204 in the embodiment of the present application, and the remediation module 305 in this embodiment can be used to execute step S206 in the embodiment of the present application.

[0091] It should be noted that the examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the contents disclosed in the above embodiments. Figure 1 In the hardware environment shown, it can be implemented by software or by hardware.

[0092] Optionally, the exception capture module is specifically used to: determine the recording time of this exception in the exception log, and determine the recording time as the trigger node; determine the transaction identifier of this exception in the exception log, and determine the generation time of the transaction identifier as the trigger node; determine the trigger statement of this exception in the exception log, and determine the trigger statement as the trigger node.

[0093] Optionally, the business chain acquisition module is specifically used to: obtain a request tracking table, wherein the request tracking table is used to record each business request in the source database according to the request time and track the completion status of each business request; determine the target request corresponding to the trigger node in the request tracking table; take all business requests from the target request to the last request in the request tracking table as the first request chain; delete the business requests marked as incomplete from the first request chain to obtain a second request chain; and determine the second request chain as the target request chain.

[0094] Optionally, the data synchronization device also includes a conflict coverage module, which is specifically used to: obtain the first completion status of each business request recorded in the source database, and determine the second completion status of each business request in the request tracking table; when the first completion status and the second completion status of any business request are inconsistent, determine the target transaction to which the business request belongs; resubmit the target transaction to the source database so that the source database re-updates the completion status of the target transaction; and record the latest completion status of the target transaction by the source database in the request tracking table.

[0095] Optionally, the business chain acquisition module is also used to: when any target business request in the second request chain has a predecessor business request with a dependent relationship, insert the predecessor business request before the position of the target business request to obtain a third request chain; and determine the third request chain as the target request chain.

[0096] Optionally, the data synchronization device also includes a data verification module, which is specifically used to: calculate a first verification value of the data in the source database and a second verification value of the data in the target database according to a preset period; when the first verification value and the second verification value in any period are inconsistent, an alarm message is issued to prompt the target object to check whether the request tracking table is damaged, and / or prompt the target object to check whether the business logic and data synchronization logic are abnormal.

[0097] Optionally, the exception capture module is further used to: determine that an exception has occurred in the synchronization process between the target database and the source database when a network interruption occurs during the synchronization process; determine that an exception has occurred in the synchronization process between the target database and the source database when an inconsistency in the synchronized data is detected; determine that an exception has occurred in the synchronization process between the target database and the source database when an inconsistency in the logical chain of the synchronized data is detected; and determine that an exception has occurred in the synchronization process between the target database and the source database when a change in the structure of the source database is detected.

[0098] According to another aspect of the embodiment of the present application, the present application provides an electronic device, such as Figure 4 As shown, it includes a memory 401, a processor 403, a communication interface 405 and a communication bus 407. The memory 401 stores a computer program that can be run on the processor 403. The memory 401 and the processor 403 communicate through the communication interface 405 and the communication bus 407. When the processor 403 executes the computer program, the steps of the above method are implemented.

[0099] The memory and processor in the electronic device communicate via a communication bus and a communication interface. The communication bus may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The communication bus may be divided into an address bus, a data bus, a control bus, and the like.

[0100] The memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage. Alternatively, the memory may be at least one storage device located away from the processor.

[0101] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, and discrete hardware components.

[0102] According to another aspect of the embodiments of the present application, a computer program product or computer program is provided, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps of any of the above embodiments.

[0103] Optionally, in an embodiment of the present application, the computer-readable medium is configured to store program codes for the processor to execute the following steps:

[0104] If an exception occurs during the synchronization process between the target database and the source database, determine the triggering node of the exception;

[0105] Obtain a target request chain from the trigger node to the source database reaching the latest state, where the request chain represents the business requests recorded according to the request time;

[0106] Execute the target request chain on the target database to resynchronize the data to the target database according to the same business logic as the source database.

[0107] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments, and this embodiment will not be described in detail here.

[0108] When implementing the embodiments of the present application, reference may be made to the above embodiments, which have corresponding technical effects.

[0109] It is understood that the embodiments described herein may be implemented using hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit may be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described herein, or a combination thereof.

[0110] For software implementation, the technology described herein can be implemented by a unit that performs the functions described herein. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0111] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0112] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0113] In the embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the modules is merely a logical function division. In actual implementation, there may be other division methods, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0114] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0115] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0116] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of the present application are essentially or partly contributed to the prior art or part of the technical solutions can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard drive, a ROM, a RAM, a magnetic disk, or an optical disk. It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such a process, method, article or device. Without further constraints, an element defined by the phrase "comprises a..." does not preclude the existence of additional identical elements in the process, method, article or apparatus that includes the element.

[0117] The foregoing is merely a list of specific embodiments of the present application, intended to enable those skilled in the art to understand or implement the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the broadest scope consistent with the principles and novel features of the present application.

Claims

1. A data synchronization method, characterized in that: include: If an exception occurs during the synchronization process between the target database and the source database, determine the triggering node of the exception; Obtaining a target request chain starting from the trigger node and leading to the source database reaching the latest state, wherein the request chain is used to represent business requests recorded according to request time; The target request chain is executed on the target database to resynchronize data to the target database according to the same business logic as that of the source database.

2. The method according to claim 1, characterized in that The triggering node for determining this anomaly includes any of the following: Determine the recording time of this exception in the exception log, and determine the recording time as the triggering node; Determine the transaction identifier of this exception in the exception log, and determine the generation time of the transaction identifier as the triggering node; A trigger statement of this exception is determined in the exception log, and the trigger statement is determined as the trigger node.

3. The method according to claim 1, characterized in that The acquiring of a target request chain from the triggering node to making the source database reach the latest state comprises: Obtaining a request tracking table, wherein the request tracking table is used to record each of the service requests in the source database according to the request time and track the completion status of each of the service requests; determining, in the request tracking table, a target request corresponding to the triggering node; taking all the business requests from the target request to the last request in the request tracking table as a first request chain; Deleting the business request marked as unfinished from the first request chain to obtain a second request chain; The second request chain is determined as the target request chain.

4. The method according to claim 3, characterized in that After obtaining the request tracking table, the method further includes: Obtaining a first completion status of each of the business requests recorded in the source database, and determining a second completion status of each of the business requests in the request tracking table; In a case where the first completion status and the second completion status of any of the business requests are inconsistent, determining a target transaction to which the business request belongs; Resubmitting the target transaction to the source database so that the source database re-updates the completion status of the target transaction; The latest completion status of the target transaction by the source database is recorded in the request tracking table.

5. The method according to claim 3, characterized in that After obtaining the second request chain, the method further includes: If any target service request in the second request chain has a predecessor service request with a dependency relationship, insert the predecessor service request before the position of the target service request to obtain a third request chain; The third request chain is determined as the target request chain.

6. The method according to claim 3, characterized in that After executing the target request chain on the target database to resynchronize data to the target database according to the same business logic as the source database, the method further includes: Calculating a first checksum of the data in the source database and a second checksum of the data in the target database according to a preset period; When the first check value and the second check value are inconsistent in any period, an alarm message is issued to prompt the target object to check whether the request tracking table is damaged, and / or to prompt the target object to check whether the business logic and data synchronization logic are abnormal.

7. The method according to any one of claims 1 to 6, characterized in that: The method further includes determining that an abnormality occurs during synchronization of the target database with the source database in at least one of the following ways: In the event of a network interruption during the synchronization process, determining that an abnormality occurs in the synchronization process of the target database with the source database; When inconsistency is detected in the synchronized data, determining that an abnormality occurs in the synchronization process of the target database with the source database; In the case where the logical chain of the synchronized data is detected to be inconsistent, determining that an abnormality occurs in the synchronization process of the target database with the source database; When it is detected that the structure of the source database has changed, it is determined that an abnormality occurs in the synchronization process of the target database with the source database.

8. A data synchronization device, characterized in that: include: The exception capture module is used to determine the triggering node of the exception when an exception occurs during the synchronization process between the target database and the source database; a service chain acquisition module, configured to acquire a target request chain starting from the trigger node and leading to the source database reaching the latest state, wherein the request chain is used to represent service requests recorded according to request time; A remediation module is configured to execute the target request chain on the target database to resynchronize data to the target database according to the same business logic as that of the source database.

9. An electronic device comprising a memory, a processor, a communication interface, and a communication bus, wherein the memory stores a computer program that can be run on the processor, and the memory and the processor communicate via the communication bus and the communication interface, characterized in that: When the processor executes the computer program, the data synchronization method according to any one of claims 1 to 7 is implemented.

10. A computer-readable medium having a non-volatile program code executable by a processor, characterized in that The program code enables the processor to execute the data synchronization method according to any one of claims 1 to 7.

Images