Data synchronization method, apparatus, device, and computer readable medium

Abstract

The application relates to a data synchronization method, device and equipment and a computer readable medium. The method comprises the following steps: in the case that an exception occurs in a synchronization process of a target database to a source database, determining a trigger node of the exception; acquiring a target request chain from the trigger node to a state in which the source database is up-to-date, wherein the request chain is used for recording a business request according to a request time; and executing the target request chain on the target database to re-synchronize data to the target database according to the same business logic as the source database. The application finds a complete business chain corresponding to the trigger node of the exception, and executes the complete business chain on the target database, so that the target database re-synchronizes the exception point and subsequent business data according to the same business logic as the source database, and solves the technical problem that logical exceptions and data inconsistency problems caused by invalid synchronization rules cannot be solved by breakpoint continuation.

Description

Technical Field

[0001] This application relates to the field of Internet technology, and in particular to a data synchronization method, apparatus, device, and computer-readable medium. Background Technology

[0002] In the information age, data synchronization is a key technology for achieving data consistency and business continuity between different databases. Currently, incremental synchronization is widely used as an efficient data synchronization method. Its core principle is to synchronize only the changed data portions in the source database. By using change data capture technology, it significantly reduces the amount of data transmission and improves synchronization efficiency, making it valuable in scenarios such as distributed systems and multi-database clusters.

[0003] However, existing incremental synchronization technology faces significant challenges in practical applications. In particular, when the source database structure is dynamically adjusted, such as adding fields or changing table structures, and the network is interrupted at the same time, the synchronization rules may fail. The logical anomalies and data inconsistencies caused by the failure of synchronization rules cannot be resolved by resuming interrupted transmissions. Ultimately, some changed data cannot be synchronized correctly, which will seriously affect reverse synchronization and business operations in the future.

[0004] There is currently no effective solution to the problem that logical anomalies and data inconsistencies caused by synchronization rule failures cannot be resolved by resuming interrupted downloads. Summary of the Invention

[0005] This application provides a data synchronization method, apparatus, device, and computer-readable medium to solve the technical problem that logical anomalies and data inconsistencies caused by synchronization rule failures cannot be resolved by resuming interrupted transmissions.

[0006] According to one aspect of the embodiments of this application, this application provides a data synchronization method, including: determining the triggering node of the current abnormality when an abnormality occurs in the synchronization process of the target database with the source database; obtaining a target request chain from the triggering node to the point where the source database reaches the latest state, wherein the request chain is used to represent business requests recorded according to the request time; and executing the target request chain on the target database to resynchronize the data to the target database according to the same business logic as the source database.

[0007] Optionally, determining the triggering node for this exception includes any of the following: determining the recording time of this exception in the exception log and using the recording time as the triggering node; determining the transaction identifier of this exception in the exception log and using the generation time of the transaction identifier as the triggering node; or determining the triggering statement of this exception in the exception log and using the triggering statement as the triggering node.

[0008] Optionally, obtaining the target request chain from the trigger node to the point where the source database reaches the latest state includes: obtaining a request tracking table, wherein the request tracking table is used to record each business request in the source database according to the request time and track the completion status of each business request; determining the target request corresponding to the trigger node in the request tracking table; taking all business requests from the target request to the last request in the request tracking table as a first request chain; deleting business requests marked as incomplete from the first request chain to obtain a second request chain; and determining the second request chain as the target request chain.

[0009] Optionally, after obtaining the request tracking table, the method further includes: obtaining the first completion status of each business request recorded in the source database, and determining the second completion status of each business request in the request tracking table; if the first completion status and the second completion status of any business request are inconsistent, determining the target transaction to which the business request belongs; resubmitting the target transaction to the source database so that the source database updates the completion status of the target transaction; and recording the latest completion status of the target transaction in the source database in the request tracking table.

[0010] Optionally, after obtaining the second request chain, the method further includes: if any target service request in the second request chain has a dependent preceding service request, inserting the preceding service request before the position of the target service request to obtain a third request chain; and determining the third request chain as the target request chain.

[0011] Optionally, after executing a target request chain on the target database to resynchronize the data to the target database according to the same business logic as the source database, the method further includes: calculating a first checksum of the data in the source database and a second checksum of the data in the target database according to a preset period; if the first checksum and the second checksum are inconsistent in any period, issuing an alarm message to prompt the target object to check whether the request tracking table is corrupted, and / or to prompt the target object to check whether the business logic and data synchronization logic are abnormal.

[0012] Optionally, the method further includes determining that the synchronization process of the target database to the source database is abnormal in at least one of the following ways: determining that the synchronization process of the target database to the source database is abnormal in the event of a network interruption during the synchronization process; determining that the synchronization process of the target database to the source database is abnormal in the event of detected inconsistencies in the synchronized data; determining that the synchronization process of the target database to the source database is abnormal in the event of detected inconsistencies in the logical chains of the synchronized data; and determining that the synchronization process of the target database to the source database is abnormal in the event of detected changes in the structure of the source database.

[0013] According to another aspect of the embodiments of this application, this application provides a data synchronization apparatus, including: an anomaly capture module, used to determine the triggering node of the anomaly when an anomaly occurs in the synchronization process between the target database and the source database; a business chain acquisition module, used to acquire the target request chain from the triggering node to the point where the source database reaches the latest state, wherein the request chain represents a business request recorded according to the request time; and a remediation module, used to execute the target request chain on the target database to resynchronize the data to the target database according to the same business logic as the source database.

[0014] According to another aspect of the embodiments of this application, this application provides an electronic device, including a memory, a processor, a communication interface and a communication bus. The memory stores a computer program that can run on the processor. The memory and the processor communicate with each other through the communication bus and the communication interface. When the processor executes the computer program, it implements the steps of the above method.

[0015] According to another aspect of the embodiments of this application, this application also provides a computer-readable medium having processor-executable non-volatile program code that causes the processor to perform the above-described method.

[0016] Compared with related technologies, the technical solutions provided in this application have the following advantages:

[0017] This application provides a data synchronization method, comprising: determining the triggering node of the anomaly when an anomaly occurs during the synchronization process between the target database and the source database; obtaining a target request chain from the triggering node to the point where the source database reaches the latest state, wherein the request chain represents business requests recorded according to the request time; and executing the target request chain on the target database to resynchronize the data to the target database according to the same business logic as the source database. This application, by finding the complete business chain corresponding to the triggering node of the anomaly when an anomaly occurs during data synchronization and executing the complete business chain on the target database, resynchronizes the target database according to the same business logic as the source database for the anomaly point and subsequent business data. Even if an occasional anomaly causes logical anomalies and data inconsistencies in the synchronized data of the target database, complete and correct synchronization can be achieved by overwriting the complete business chain, thus solving the technical problem that logical anomalies and data inconsistencies caused by synchronization rule failures cannot be resolved by resuming interrupted data transmission. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0020] Figure 1 This is a schematic diagram of the hardware environment for an optional data synchronization method provided according to an embodiment of this application;

[0021] Figure 2 This is a schematic diagram of an optional data synchronization method provided according to an embodiment of this application;

[0022] Figure 3 This is a block diagram of an optional data synchronization device according to an embodiment of this application;

[0023] Figure 4 This is a schematic diagram of an optional electronic device structure provided in an embodiment of this application. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0025] In the following description, the use of suffixes such as "module," "part," or "unit" to denote elements is solely for the purpose of illustration and has no specific meaning in itself. Therefore, "module" and "part" may be used interchangeably.

[0026] Existing incremental synchronization technologies face significant challenges in practical applications, especially when the source database structure is dynamically adjusted (e.g., adding fields, changing table structures) while a network interruption occurs. This can lead to synchronization rule failures. For example, if the source database adds a field named "discount," but the target database doesn't receive the relevant DDL statement due to network issues, its structure remains unchanged. When the incremental synchronization module attempts to write data containing the "discount" field to the target database, it will fail because the target database does not have the "discount" field. The absence of the "discount" field in the target database constitutes a case of synchronization rule failure. Additionally, data may be synchronized but contain errors (e.g., some fields are missing or tampered with); the synchronization location may be correct but the data logic is incomplete (e.g., transactions are not committed, dependencies are broken); or data may be successfully written to the target database but cause business logic conflicts (e.g., duplicate primary keys, failed foreign key constraints). These are all examples of synchronization rule failures. The logical anomalies and data inconsistencies caused by synchronization rule failures cannot be resolved through breakpoint resumption, ultimately leading to some changed data not being correctly synchronized, which can severely impact reverse synchronization and business operations.

[0027] To address the problems mentioned in the background art, according to one aspect of the embodiments of this application, an embodiment of a data synchronization method is provided.

[0028] Optionally, in the embodiments of this application, the above-described data synchronization method can be applied to, for example... Figure 1 The hardware environment shown consists of terminal 101 and server 103. Figure 1 As shown, server 103 is connected to terminal 101 via a network and can be used to provide services to the terminal or clients installed on the terminal. Database 105 can be set up on the server or independently of the server to provide data storage services for server 103. The network mentioned above includes, but is not limited to, wide area network, metropolitan area network or local area network. Terminal 101 includes, but is not limited to, PC, mobile phone, tablet computer, etc.

[0029] One data synchronization method in this application embodiment can be executed by server 103, such as... Figure 2 As shown, the method may include the following steps:

[0030] Step S202: If an anomaly occurs during the synchronization process between the target database and the source database, determine the triggering node of this anomaly.

[0031] Step S204: Obtain the target request chain from the trigger node to the point where the source database reaches the latest state, wherein the request chain is used to represent business requests recorded according to the request time;

[0032] Step S206: Execute the target request chain on the target database to resynchronize the data to the target database according to the same business logic as the source database.

[0033] In this embodiment, the source database refers to the database directly connected to the business system, and changes to the business system are directly reflected in the source database. The target database refers to the database used to back up the source database. By monitoring data changes in the source database in real time, the target database can synchronize each update of the source database to the target database based on incremental synchronization technology, thereby achieving backup of the source database. The target database can be used to synchronize data from the target database to the source database through reverse synchronization when the source database fails, thereby avoiding data loss due to source database failure. It can also be used to connect to the business system and provide business services in place of the source database when the source database is unable to provide services.

[0034] In this embodiment, a request chain refers to a logical sequence of business requests arranged chronologically, where each request corresponds to a data modification operation in the source database, such as an SQL statement. Business logic refers to the rules governing how the source database processes business requests, such as the logic of "verifying inventory before inserting an order."

[0035] In this embodiment, when an anomaly occurs during the synchronization process between the target database and the source database (such as network interruption or data inconsistency), the system locates the starting point of the anomaly through the anomaly log. This triggering node includes, but is not limited to, key information such as timestamps, transaction IDs, error statements, and sequence numbers in the anomaly log. For example, if the anomaly log records "Network interruption caused synchronization interruption on June 5, 2025, 14:30," then the triggering node is that time point; if the anomaly log records "Error in executing XXX statement caused synchronization anomaly," then the triggering node is that statement. Starting from the triggering node, the system extracts all subsequent business requests from the source database, forming a continuous request chain. The request chain is a sequence of business operations recorded in chronological order, such as "User A inserts an order → User B updates the order status → User C deletes the order." The target request chain is executed line by line in the target database according to the business logic of the source database. For example, if the source database executes "Insert product data → Modify price → Delete invalid record" after the triggering node, the target database re-executes these three operations in the same order, thereby resynchronizing the target database with the same business logic as the source database for the anomaly point and subsequent business data.

[0036] Through steps S202 to S206, this application finds the complete business chain corresponding to the trigger node of the exception when an exception occurs during the data synchronization process, and executes the complete business chain in the target database. This allows the target database to resynchronize the exception point and subsequent business data according to the same business logic as the source database. Even if an occasional exception causes logical anomalies and data inconsistencies in the synchronized data of the target database, complete and correct synchronization can be achieved by re-overwriting the complete business chain. This solves the technical problem that logical anomalies and data inconsistencies caused by the failure of synchronization rules cannot be solved by resuming interrupted transmission.

[0037] In this embodiment, the target request chain can also be obtained through a message queue backtracking mechanism. Specifically, each business request from the source database is synchronously sent to the message queue upon generation, carrying the request time, content, and transaction identifier. After an anomaly is triggered, all messages that have not been acknowledged by the target database after the trigger node are retrieved from the message queue based on the trigger node's timestamp or transaction identifier, forming the target request chain. In this way, when the request tracking table is corrupted due to database failure, the persistent records in the message queue can serve as a backup data source, ensuring the reliability of request chain acquisition.

[0038] In an optional embodiment, the triggering node for this anomaly includes any of the following:

[0039] Determine the recording time of this exception in the exception log and identify the recording time as the trigger node;

[0040] Determine the transaction identifier of this exception in the exception log, and determine the generation time of the transaction identifier as the trigger node;

[0041] Identify the triggering statement for this exception in the exception log and designate the triggering statement as the triggering node.

[0042] In this embodiment, the transaction identifier refers to a unique identifier generated by the source database for each business transaction, used to track the execution status of the transaction. The triggering statement refers to the specific database operation statement that causes a synchronization exception, such as INSERT, UPDATE, or DELETE statements.

[0043] In this embodiment, the trigger node can be determined based on the recording time of the synchronization anomaly. Specifically, the timestamp of the anomaly occurrence can be extracted from the anomaly log. For example, if the log record states "2025-06-05 14:35:00 Data verification failed," then "14:35:00" can be used as the trigger node. This method of determining the trigger node based on the recording time of the synchronization anomaly is suitable for time-sensitive anomalies such as network interruptions, allowing for rapid location of the interruption point using the timestamp.

[0044] In this embodiment, the triggering node can be determined based on the transaction identifier of the transaction being processed when the synchronization exception occurs. Specifically, if the exception log contains a transaction ID (e.g., "TransactionID:TX202506050123"), the generation time (e.g., "14:32:15") is queried in the source database based on this transaction ID, and that time is used to determine the triggering node. Alternatively, the transaction ID can be directly used as the triggering node; in this case, exception recovery requires finding the business request associated with the transaction ID. The target transaction can also be determined based on the transaction ID, and its commit time can be used as the triggering node. This method of determining the triggering node based on the transaction identifier of the transaction being processed when the synchronization exception occurs is suitable for transaction-level exceptions, such as when a transaction is interrupted halfway through execution. By associating the transaction ID with the specific operation time, it ensures that recovery begins from that transaction.

[0045] In this embodiment, the trigger node can be determined based on the triggering statement, that is, by directly extracting the erroneous statement from the exception log, such as "UPDATE orders SET status='invalid'WHERE id=123 execution failed", and then using this SQL statement as the trigger node. This method of determining the trigger node based on the triggering statement is suitable for statement-level exceptions, such as syntax errors or insufficient permissions, allowing for direct remediation of the erroneous statement and avoiding the repeated execution of irrelevant requests.

[0046] In this embodiment, the trigger node can also be determined based on business logic correlation. Specifically, error information in the exception log can be parsed to identify the business entity related to the exception; the latest operation record of the business entity with business logic correlation can be queried in the source database, and the execution time or transaction ID of the operation can be used as the trigger node. For example, if the exception log records "User 123" and "Order Status Update", then the latest record of "User 123"'s order operation can be queried, and the generation time of the latest record or the transaction ID of the corresponding transaction can be used as the trigger node. When the exception log does not explicitly record a timestamp or transaction ID, the trigger node can be quickly located through business entity correlation, which is suitable for scenarios with complex business logic and long request chains.

[0047] This application adopts a multi-dimensional trigger node location method to adapt to different types of abnormal scenarios, thereby improving the flexibility and accuracy of abnormal location.

[0048] This application uses a separate request tracing table to fully record the business logic executed by the source database. Therefore, even if a synchronization exception cannot be resolved by resuming interrupted downloads, it can be resolved through business chain overriding. The following describes the request tracing table and the method for resolving synchronization exceptions that cannot be resolved by resuming interrupted downloads based on business chain overriding.

[0049] In an optional embodiment, obtaining the target request chain from the triggering node to bringing the source database to the latest state includes:

[0050] Step 1: Obtain the request tracking table. The request tracking table is used to record each business request in the source database according to the request time and track the completion status of each business request.

[0051] Step 2: Determine the target request corresponding to the triggering node in the request tracing table;

[0052] Step 3: All business requests from the target request to the last request in the request tracking table are considered as the first request chain;

[0053] Step 4: Remove the business requests marked as incomplete from the first request chain to obtain the second request chain;

[0054] Step 5: Determine the second request chain as the target request chain.

[0055] In this embodiment, the request tracking table `request_tracking` refers to a metadata table in the source database used to record business requests. It includes fields such as request time, content, and status, as well as the start and end dates of request execution, used to track the execution status of requests. Whenever the source database performs a business operation, a record for that operation is first inserted into the request tracking table. At this time, the status of the business operation is "incomplete." When the source database reports that the business operation is complete, the status of the business operation is then updated to "complete" in the request tracking table.

[0056] In this embodiment, since the request tracking table fully records the business logic executed by the source database, after determining the trigger node of the synchronization anomaly, the target request corresponding to the trigger node can be found in the request tracking table. For example, if the trigger node is the timestamp "14:35:00", then the business request being executed at "14:35:00" in the request tracking table is found as the target request corresponding to the trigger node. Starting from the target request, all records up to the last request in the table are extracted to form a first request chain. Business requests with an incomplete status are deleted from the first request chain, resulting in a second request chain containing only business requests with a completed status. The second request chain is used as the final target request chain for execution. When a business request is incomplete, the corresponding business operation is not successfully executed. The incomplete business request will not change the business status, and therefore will not change the business data in the source database. Therefore, it is only necessary to place the completed business requests that affect the business data in the source database into the target database for execution to ensure that data synchronization is performed according to the same business logic as the source database.

[0057] This application comprehensively records the business logic executed by the source database by adding a separate request tracking table. This ensures that even if occasional exceptions cause logical anomalies and data inconsistencies in the target database, complete and correct synchronization can be achieved by overriding the entire business chain. This solves the technical problem that logical anomalies and data inconsistencies caused by failed synchronization rules cannot be resolved through breakpoint resumption. Furthermore, the application uses status filtering in the request tracking table to ensure that only successfully completed requests are re-executed, avoiding invalid operations and data conflicts, and improving the efficiency and accuracy of remediation.

[0058] In this application, conflicts can also occur between the source database and the request tracking table. For example, in a user payment order scenario, when the source database confirms receipt of payment information and needs to update the order status from "unpaid" to "paid," it inserts a record of the update operation in the request tracking table and marks it as "started." If a network interruption causes the order status update in the source database to fail, i.e., it remains "unpaid," then the update statement in the request tracking table will be marked as "incomplete." The source database will trigger remedial measures for such payment transaction anomalies, such as manual intervention or asynchronous retries, updating the order status to "paid." However, the request tracking table still shows the update operation as "incomplete," leading to a conflict.

[0059] In this embodiment of the application, in order to avoid the inconsistency of business logic between the source database and the request tracking table due to similar conflicts, which in turn leads to the inconsistency of business logic between the target database and the source database and causes data synchronization anomalies, this application provides a conflict coverage method, which will be described below.

[0060] In an optional embodiment, after obtaining the request tracking table, the method further includes:

[0061] Step 1: Obtain the first completion status of each business request recorded in the source database, and determine the second completion status of each business request in the request tracking table;

[0062] Step 2: If the first completion status and the second completion status of any business request are inconsistent, determine the target transaction to which the business request belongs;

[0063] Step 3: Recommit the target transaction to the source database so that the source database updates the completion status of the target transaction.

[0064] Step 4: Record the latest completion status of the target transaction in the source database into the request tracking table.

[0065] In this embodiment, after obtaining the request tracking table, it is possible to check whether there is a business logic conflict between the source database and the request tracking table. If a conflict exists, regardless of the current status of the business request corresponding to the conflict in the source database, the transaction to which the business request belongs is re-executed. Then, the request tracking table tracks the latest status to ensure that the business logic of the source database and the request tracking table is completely consistent. Specifically, the completion status of each business request is obtained from both the source database and the request tracking table. For example, if the status of "order insertion request" in the source database is "completed," while the status of the same request in the request tracking table is "not completed," then the status is determined to be inconsistent. Based on the inconsistent request, the transaction to which it belongs is queried. For example, if the order placement transaction with transaction ID TX20250605002 contains two requests, "order insertion" and "inventory deduction," the target transaction is resubmitted to the source database, triggering the source database to re-execute all requests in the transaction and update the completion status of the transaction. At this time, the request tracking table tracks the latest status of the target transaction, ensuring that the two are consistent.

[0066] This application resolves inconsistencies between the business logic of the request tracking table and the source database by performing a consistency comparison of the business logic of the source database and the request tracking table, covering any existing conflicts, and ensuring that the remediation process is based on accurate and consistent business logic of the source database and the request tracking table, thus avoiding synchronization anomalies caused by conflicts between the business logic of the source database and the request tracking table.

[0067] In this application, some business requests have dependencies. For example, a request to write data containing a discount field depends on a request to add a new discount field. If, due to unforeseen circumstances, the source database adds the discount field but the target database does not receive the relevant DDL statement and its structure is not updated, then when the incremental synchronization module attempts to write data containing the discount field to the target database, it will fail because the target database does not have the discount field. Therefore, to resolve synchronization anomalies caused by missing dependencies, based on the above embodiment, the preceding business requests that the target business request depends on can be inserted into the target request chain. This ensures that each business request in the target request chain has complete dependencies, avoiding synchronization anomalies caused by missing dependencies. This will be explained below.

[0068] In an optional embodiment, after obtaining the second request chain, the method further includes:

[0069] Step 1: If any target business request in the second request chain has a dependent preceding business request, insert the preceding business request before the target business request to obtain the third request chain.

[0070] Step 2: Identify the third request chain as the target request chain.

[0071] In this embodiment, each target business request in the second request chain is traversed to check for any preceding dependent requests. For example, a request to write data containing a discount field depends on a request to add a discount field, and a request to update an order status depends on a request to insert an order. If the request chain only contains the target business request but not its preceding business requests, a dependency relationship is missing, and the corresponding preceding business request needs to be inserted before the target business request to complete the dependency relationship. If the preceding business request is located after the target business request, a logical conflict exists, and the position of the preceding business request needs to be moved before the target business request. Finally, the adjusted request chain is used as the final target request chain to ensure that the dependencies of each business request in the target request chain are complete and conform to the business logic.

[0072] This application uses dependency completion to avoid synchronization failures caused by logical dependencies.

[0073] In an optional embodiment, after executing a target request chain on the target database to resynchronize data to the target database according to the same business logic as the source database, the method further includes:

[0074] Step 1: Calculate the first checksum of the data in the source database and the second checksum of the data in the target database according to a preset period;

[0075] Step 2: If the first check value and the second check value are inconsistent in any period, an alarm message is issued to prompt the target object to check whether the request tracking table is corrupted, and / or to prompt the target object to check whether the business logic and data synchronization logic are abnormal.

[0076] In this embodiment, data integrity checks can be performed periodically by comparing data in the source and target databases to ensure data consistency and integrity. Specifically, hash algorithms such as MD5 and SHA-256 are used to calculate hash checksums for the full data or data from a specific incremental synchronization in both the source and target databases. The two checksums are compared; if they are inconsistent, it indicates a synchronization anomaly. Remediation based on business chain coverage, as described in the above embodiment, can be implemented. If the two checksums still do not match after remediation, the system may be experiencing one of the following: the request tracking table itself is corrupted, such as due to data loss or errors; an extreme anomaly not covered by the remediation logic, such as successful writing to the target database without feedback; or a conflict between business logic and data synchronization logic, such as concurrent operations leading to dirty data. In this case, if the system cannot automatically repair, an alarm process needs to be triggered to prompt maintenance personnel to check whether the request tracking table is corrupted or whether the business logic and data synchronization logic are abnormal.

[0077] This application enables timely detection of potential data inconsistencies through regular verification, and quickly locates the source of the problem by combining an alarm mechanism, achieving full-cycle monitoring of data synchronization. This can improve the reliability and stability of incremental data synchronization and reduce business risks.

[0078] In an optional embodiment, the method further includes determining that the synchronization process between the target database and the source database has encountered an anomaly in at least one of the following ways:

[0079] In the event of a network interruption during synchronization, it is determined that the synchronization process between the target database and the source database has encountered an anomaly.

[0080] If inconsistencies in the synchronized data are detected, it is determined that there is an anomaly in the synchronization process between the target database and the source database.

[0081] If an inconsistency is detected in the logical chain of synchronized data, it is determined that the synchronization process between the target database and the source database has encountered an anomaly.

[0082] If a change in the structure of the source database is detected, it is determined that an anomaly has occurred in the synchronization process between the target database and the source database.

[0083] In this application embodiment, incremental synchronization provides a breakpoint resume function, which can solve the problem of physical-level synchronization interruption recovery, such as network interruption. However, logical-level synchronization interruption cannot be solved by breakpoint resume. The data synchronization method proposed in this application can cover the logical anomalies and data consistency problems that breakpoint resume cannot solve.

[0084] In this embodiment, the determination of synchronization anomalies can be physical-level synchronization interruptions, such as network interruptions, or logical-level synchronization interruptions. Examples include inconsistent synchronized data (potentially due to missing or tampered fields after synchronization), incomplete data logic despite correct synchronization location (potentially due to uncommitted transactions or broken dependencies), changes in the source database structure (such as adding tables or fields), and successful writes to the target database but business logic conflicts (potentially due to duplicate primary keys or failed foreign key constraints). Both physical and logical-level synchronization interruptions can be addressed through remediation based on business chain coverage, improving the reliability and stability of incremental data synchronization and reducing business risks.

[0085] This application provides a data synchronization method, comprising: determining the triggering node of the anomaly when an anomaly occurs during the synchronization process between the target database and the source database; obtaining a target request chain from the triggering node to the point where the source database reaches the latest state, wherein the request chain represents business requests recorded according to the request time; and executing the target request chain on the target database to resynchronize the data to the target database according to the same business logic as the source database. This application, by finding the complete business chain corresponding to the triggering node of the anomaly when an anomaly occurs during data synchronization and executing the complete business chain on the target database, resynchronizes the target database according to the same business logic as the source database for the anomaly point and subsequent business data. Even if an occasional anomaly causes logical anomalies and data inconsistencies in the synchronized data of the target database, complete and correct synchronization can be achieved by overwriting the complete business chain, thus solving the technical problem that logical anomalies and data inconsistencies caused by synchronization rule failures cannot be resolved by resuming interrupted data transmission.

[0086] According to another aspect of the embodiments of this application, such as Figure 3 As shown, a data synchronization device is provided, comprising:

[0087] The exception capture module 301 is used to determine the triggering node of the exception when an exception occurs during the synchronization process between the target database and the source database.

[0088] The business chain acquisition module 303 is used to acquire the target request chain from the trigger node to the point where the source database reaches the latest state. The request chain represents the business requests recorded according to the request time.

[0089] The remediation module 305 is used to execute the target request chain on the target database to resynchronize the data to the target database according to the same business logic as the source database.

[0090] It should be noted that the exception capture module 301 in this embodiment can be used to execute step S202 in this application embodiment, the business chain acquisition module 303 in this embodiment can be used to execute step S204 in this application embodiment, and the remediation module 305 in this embodiment can be used to execute step S206 in this application embodiment.

[0091] It should be noted that the examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the content disclosed in the above embodiments. It should also be noted that the above modules, as part of a device, can operate in environments such as... Figure 1 The hardware environment shown can be implemented either through software or through hardware.

[0092] Optionally, the exception capture module is specifically used to: determine the recording time of the current exception in the exception log, and determine the recording time as the trigger node; determine the transaction identifier of the current exception in the exception log, and determine the generation time of the transaction identifier as the trigger node; determine the triggering statement of the current exception in the exception log, and determine the triggering statement as the trigger node.

[0093] Optionally, the business chain acquisition module is specifically used to: acquire a request tracking table, wherein the request tracking table is used to record each business request in the source database according to the request time and track the completion status of each business request; determine the target request corresponding to the triggering node in the request tracking table; take all business requests from the target request to the last request in the request tracking table as a first request chain; delete the business requests marked as incomplete from the first request chain to obtain a second request chain; and determine the second request chain as the target request chain.

[0094] Optionally, the data synchronization device further includes a conflict coverage module, specifically used for: obtaining the first completion status of each business request recorded in the source database, and determining the second completion status of each business request in the request tracking table; determining the target transaction to which the business request belongs if the first completion status and the second completion status of any business request are inconsistent; resubmitting the target transaction to the source database so that the source database updates the completion status of the target transaction; and recording the latest completion status of the target transaction in the source database in the request tracking table.

[0095] Optionally, the business chain acquisition module is further configured to: if any target business request in the second request chain has a dependent preceding business request, insert the preceding business request before the position of the target business request to obtain a third request chain; and determine the third request chain as the target request chain.

[0096] Optionally, the data synchronization device further includes a data verification module, specifically used for: calculating a first verification value of data in the source database and a second verification value of data in the target database according to a preset period; issuing an alarm message if the first verification value and the second verification value are inconsistent in any period, so as to prompt the target object to check whether the request tracking table is corrupted, and / or to prompt the target object to check whether the business logic and data synchronization logic are abnormal.

[0097] Optionally, the anomaly capture module is further configured to: determine that the synchronization process between the target database and the source database is abnormal in the event of a network interruption during synchronization; determine that the synchronization process between the target database and the source database is abnormal in the event of inconsistency in synchronized data; determine that the synchronization process between the target database and the source database is abnormal in the event of inconsistency in the logical chain of synchronized data; and determine that the synchronization process between the target database and the source database is abnormal in the event of a change in the structure of the source database.

[0098] According to another aspect of the embodiments of this application, this application provides an electronic device, such as... Figure 4 As shown, the system includes a memory 401, a processor 403, a communication interface 405, and a communication bus 407. The memory 401 stores a computer program that can run on the processor 403. The memory 401 and the processor 403 communicate through the communication interface 405 and the communication bus 407. When the processor 403 executes the computer program, it implements the steps of the above method.

[0099] The memory and processor in the aforementioned electronic devices communicate with each other via a communication bus and a communication interface. The communication bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into an address bus, a data bus, a control bus, etc.

[0100] The memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0101] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0102] According to another aspect of the embodiments of this application, a computer program product or computer program is also provided, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps of any of the above embodiments.

[0103] Optionally, in embodiments of this application, the computer-readable medium is configured to store program code for the processor to perform the following steps:

[0104] If an anomaly occurs during the synchronization process between the target database and the source database, determine the triggering node of this anomaly;

[0105] Obtain the target request chain from the trigger node to the point where the source database reaches the latest state. The request chain represents the business requests recorded according to the request time.

[0106] Execute the target request chain on the target database to resynchronize the data to the target database according to the same business logic as the source database.

[0107] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments, and will not be repeated here.

[0108] In specific implementation, the embodiments of this application can be referred to the above embodiments and have corresponding technical effects.

[0109] It is understood that the embodiments described herein can be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described herein, or combinations thereof.

[0110] For software implementation, the techniques described herein can be implemented by units that perform the functions described herein. The software code can be stored in memory and executed by a processor. The memory can be implemented in the processor or external to the processor.

[0111] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0112] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0113] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0114] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0115] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0116] If the aforementioned function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks. It should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. In the absence of further restrictions, an element defined by the phrase "comprising a..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0117] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A data synchronization method, characterized in that, include: If an anomaly occurs during the synchronization process between the target database and the source database, determine the triggering node of this anomaly; Obtain the target request chain from the trigger node to the point where the source database reaches the latest state, wherein the request chain represents business requests recorded according to the request time; The target request chain is executed on the target database to resynchronize the data to the target database according to the same business logic as the source database; Determining the triggering node for this anomaly includes determining the triggering node based on business logic relevance, specifically including: parsing the error information in the anomaly log and identifying the business entity related to the anomaly; querying the latest operation record of the business entity with business logic relevance in the source database, and using the execution time or transaction identifier of the operation as the triggering node; The step of obtaining the target request chain from the trigger node to the point where the source database reaches the latest state includes: obtaining a request tracking table, wherein the request tracking table is used to record each of the business requests in the source database according to the request time and track the completion status of each of the business requests; determining the target request corresponding to the trigger node in the request tracking table; taking all the business requests from the target request to the last request in the request tracking table as a first request chain; deleting the business requests marked as incomplete from the first request chain to obtain a second request chain; and determining the second request chain as the target request chain. After obtaining the request tracking table, the method further includes: obtaining a first completion status of each business request recorded in the source database, and determining a second completion status of each business request in the request tracking table; if the first completion status and the second completion status of any business request are inconsistent, determining the target transaction to which the business request belongs; resubmitting the target transaction to the source database so that the source database updates the completion status of the target transaction; and recording the latest completion status of the target transaction in the source database in the request tracking table.

2. The method according to claim 1, characterized in that, The triggering node for this anomaly includes any of the following: Determine the recording time of this exception in the exception log, and identify the recording time as the trigger node; The transaction identifier of this exception is determined in the exception log, and the generation time of the transaction identifier is determined as the trigger node; The triggering statement for this exception is determined in the exception log, and the triggering statement is identified as the triggering node.

3. The method according to claim 1, characterized in that, After obtaining the second request chain, the method further includes: If any target service request in the second request chain has a dependent preceding service request, the preceding service request is inserted before the position of the target service request to obtain the third request chain. The third request chain is identified as the target request chain.

4. The method according to claim 1, characterized in that, After executing the target request chain on the target database to resynchronize data to the target database according to the same business logic as the source database, the method further includes: Calculate the first verification value of the data in the source database and the second verification value of the data in the target database according to a preset period; If the first verification value and the second verification value are inconsistent in any period, an alarm message is issued to prompt the target object to check whether the request tracking table is corrupted, and / or to prompt the target object to check whether the business logic and data synchronization logic are abnormal.

5. The method according to any one of claims 1 to 4, characterized in that, The method further includes determining, by at least one of the following methods, that the synchronization process between the target database and the source database has encountered an anomaly: In the event of a network interruption during synchronization, it is determined that the synchronization process between the target database and the source database has encountered an anomaly. If inconsistency is detected in the synchronized data, it is determined that the synchronization process between the target database and the source database has encountered an anomaly. If an inconsistency is detected in the logical chain of synchronized data, it is determined that the synchronization process between the target database and the source database has encountered an anomaly. If a change in the structure of the source database is detected, it is determined that the synchronization process between the target database and the source database has malfunctioned.

6. A data synchronization device for implementing the data synchronization method as described in any one of claims 1 to 5, characterized in that, include: The exception capture module is used to determine the triggering node when an exception occurs during the synchronization process between the target database and the source database. The business chain acquisition module is used to acquire the target request chain from the trigger node to the point where the source database reaches the latest state, wherein the request chain is used to represent business requests recorded according to the request time; The remediation module is used to execute the target request chain on the target database to resynchronize the data to the target database according to the same business logic as the source database.

7. An electronic device comprising a memory, a processor, a communication interface, and a communication bus, wherein the memory stores a computer program executable on the processor, and the memory and the processor communicate via the communication bus and the communication interface, characterized in that... When the processor executes the computer program, it implements the data synchronization method according to any one of claims 1 to 5.

8. A computer-readable medium having processor-executable non-volatile program code, characterized in that, The program code causes the processor to execute the data synchronization method according to any one of claims 1 to 5.

Images