Data consistency verification and compensation method and device, storage medium and electronic equipment

Abstract

This application discloses a data consistency verification and compensation method, apparatus, storage medium, and electronic device. The method includes: detecting failed transactions and transactions containing insignificant events during data synchronization; recording failed transactions and transactions containing insignificant events to form a set of transactions to be verified; parsing the transactions in the set of transactions to be verified to obtain multiple independent events; comparing the verification values ​​of the corresponding data rows of the independent events at the source and target ends; selecting events to be compensated based on the comparison results to form a set of events to be compensated; determining the execution dependencies related to the events in the set of events to be compensated; controlling the execution order according to the execution dependencies; and performing corresponding data compensation operations on the events in the set of events to be compensated in the target database. By detecting transactions with insignificant events during synchronization and performing targeted verification, unnecessary data comparisons are reduced while ensuring data consistency, thereby improving the processing efficiency and resource utilization of the compensation process.

Description

Technical Field

[0001] This application relates to the field of database technology, and in particular to a data consistency verification and compensation method, apparatus, storage medium and electronic device. Background Technology

[0002] Driven by the demands for data security, sharing, and high reliability in production environments, database synchronization technologies are becoming increasingly widespread. However, during database synchronization, regardless of whether the discrepancy occurs due to human error, system issues, or unforeseen circumstances, it often leads to a series of chain reactions and application problems. This is especially true between heterogeneous databases, where the probability of errors is particularly high due to differences in the data types and type restrictions they support.

[0003] Existing technical solutions typically employ a method of comparing data shards from the source and target ends within a specified period to achieve data consistency verification and compensation. For example, in patent CN117762672A, a verification time zone is obtained based on a preset start time and time interval. The data within the verification time zone is divided into several shards, and the source and target data within each shard are read into the built-in memory. A row-by-row comparison is performed based on the primary key and timestamp fields of each table. The target end data is repaired based on the comparison results and the timestamps. Another example is patent CN116610657A, which shards the data from the source and target ends, obtains the maximum timestamp from each shard, and determines consistency by comparing the timestamp and hash value. When the consistency verification fails, the source end data shard is pulled to the target end shadow table for repair.

[0004] However, these existing technical solutions have two common problems: First, they all require loading the entire data shard into memory for comparison or processing. For data shards containing only a few inconsistent records, this approach results in a significant waste of storage space, especially when processing data tables containing large object data, where resource consumption is particularly prominent. Second, existing solutions do not consider the correlation between data shards. Data shards with correlations need to be executed sequentially, otherwise data inconsistency may occur. However, if all shards are simply executed sequentially without considering correlations, the overall processing efficiency will be reduced.

[0005] Therefore, an improved data consistency verification and compensation method is needed to reduce unnecessary resource consumption and improve processing efficiency while ensuring data consistency, especially to achieve efficient and accurate data repair in heterogeneous database synchronization scenarios. Summary of the Invention

[0006] To address the aforementioned issues, this application provides a data consistency verification and compensation method, apparatus, storage medium, and electronic device. By accurately identifying and recording failed and invalid transactions during the synchronization process for targeted verification, and intelligently controlling the compensation execution order based on the dependencies between events, the scope of unnecessary data comparisons is reduced while ensuring data consistency, and the processing efficiency and resource utilization of the compensation process are improved.

[0007] In a first aspect, embodiments of this application provide a data consistency verification and compensation method, the method comprising:

[0008] Detect failed transactions and transactions containing insignificant events during the data synchronization process, record failed transactions and transactions with insignificant events to form a set of transactions to be verified; among them, insignificant events refer to data operation events that return zero affected rows after execution in the target database;

[0009] The transactions in the set of transactions to be verified are parsed to obtain multiple independent events. The verification values ​​of the corresponding data rows of the independent events are compared between the source and target ends. Based on the comparison results, the events to be compensated are selected to form a set of events to be compensated.

[0010] Determine the execution dependencies related to the events in the set of events to be compensated, control the execution order based on the execution dependencies, and perform the corresponding data compensation operations on the target database for the events in the set of events to be compensated.

[0011] In one possible implementation, the checksum value is the MD5 value;

[0012] Compare the checksums of the corresponding data rows for independent events at the source and target ends, including:

[0013] Query the data rows corresponding to the source and target ends of independent events;

[0014] Compare the MD5 value of the corresponding data row at the source end of the independent event with the MD5 value of the corresponding data row at the target end.

[0015] In one possible implementation, events to be compensated are selected based on the comparison results to form a set of events to be compensated, including:

[0016] When the source MD5 value matches the target MD5 value, the corresponding independent event is ignored;

[0017] When the source MD5 value is empty and the target MD5 value is not empty, the corresponding independent event is added to the set of events to be compensated and identified as the event that needs to be deleted.

[0018] When the source MD5 value is not empty and the target MD5 value is empty, the corresponding independent event is added to the set of events to be compensated and identified as the event for which the insertion operation needs to be performed.

[0019] When both the source and target MD5 values ​​are not empty and the source and target MD5 values ​​are inconsistent, the corresponding independent events are added to the set of events to be compensated and identified as events that need to be updated.

[0020] In one possible implementation, determining the execution dependencies related to events in the set of events to be compensated includes:

[0021] Based on the table constraint information recorded in the events to be compensated, analyze the foreign key dependencies between the data tables involved in different events;

[0022] Based on foreign key dependencies, corresponding tagging information is generated for each event. The tagging information includes the unique identifier of the current event and the set of prerequisite event identifiers that must be completed for the event to be executed.

[0023] Determine the execution dependencies related to the event based on the tagging information.

[0024] In one possible implementation, controlling the execution order based on execution dependencies includes:

[0025] Query the set of preceding event identifiers corresponding to each target event in the set of events to be compensated; wherein, the marker information is stored in a global cache;

[0026] If the set of preceding event identifiers is empty, the compensation operation for the target event is executed immediately or in parallel.

[0027] If the set of preceding event identifiers is not empty, the target event is placed in a waiting queue until all preceding events of the target event have completed their compensation operations before the compensation operation of the target event is executed.

[0028] In one possible implementation, before performing the corresponding data compensation operation on the events in the set of events to be compensated, after controlling the execution order according to the execution dependency, the method further includes:

[0029] Query the MD5 value of the data row corresponding to the event to be compensated from both the source and target ends again and compare them;

[0030] If the comparison results are still inconsistent, the latest data is retrieved from the source and compensation operations are performed on the target side based on the latest data.

[0031] In one possible implementation, after performing the corresponding data compensation operation on the target database for the events in the set of events to be compensated, the method further includes:

[0032] Remove the tag information corresponding to events that have been compensated from the global cache;

[0033] Check other tag information in the global cache and remove the event identifiers that have been compensated from the set of preceding event identifiers of other tag information.

[0034] Secondly, embodiments of this application provide a data consistency verification and compensation device, which includes: a recording module, a recording module, and an execution module, wherein:

[0035] The recording module is used to detect failed transactions and transactions containing insignificant events during the data synchronization process, and to record failed transactions and transactions with insignificant events to form a set of transactions to be verified; among them, insignificant events refer to data operation events that return zero affected rows after execution in the target database;

[0036] The recording module is used to parse the transactions in the transaction set to be verified, obtain multiple independent events, compare the verification values ​​of the corresponding data rows of the independent events at the source and target ends, and filter out the events to be compensated based on the comparison results to form a set of events to be compensated.

[0037] The execution module is used to determine the execution dependencies related to events in the set of events to be compensated, control the execution order based on the execution dependencies, and perform corresponding data compensation operations on the target database for the events in the set of events to be compensated.

[0038] Thirdly, embodiments of this application provide a computer storage medium storing multiple instructions adapted for loading by a processor and executing the steps of the above-described method.

[0039] Fourthly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being adapted to be loaded by the processor and to execute the steps of the above-described method.

[0040] The beneficial effects of the technical solutions provided by some embodiments of this application include at least the following: by accurately capturing abnormal transactions during the synchronization process as verification objects, the scope of data comparison is effectively narrowed, and unnecessary verification workload is significantly reduced; by adopting the mechanism of replacing full data loading with MD5 checksum comparison, memory consumption and network transmission overhead are reduced; by establishing an execution order control model based on data dependencies, the maximum parallel processing of compensation operations is achieved while ensuring data consistency; at the same time, the secondary verification mechanism before compensation further ensures the accuracy and reliability of data repair, and ultimately achieves an optimized balance between resource consumption and processing efficiency as a whole. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 A system architecture diagram of a data consistency verification and compensation system provided in this application embodiment;

[0043] Figure 2 A flowchart illustrating a data consistency verification and compensation method provided in an embodiment of this application;

[0044] Figure 3 A logical block diagram for data consistency verification provided in this application embodiment;

[0045] Figure 4 A data compensation logic block diagram provided for an embodiment of this application;

[0046] Figure 5 A structural block diagram of a data consistency verification and compensation device provided in an embodiment of this application;

[0047] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0048] To make the features and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0049] In the following description, when referring to the accompanying drawings, the same numbers in different drawings denote the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0050] In the description of this application, it should be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances. Furthermore, in the description of this application, unless otherwise stated, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship.

[0051] As mentioned earlier, with the deepening of enterprise informatization, data synchronization technology between databases has become a key support for ensuring business continuity, achieving data sharing, and building a highly available architecture. Especially in heterogeneous database environments, the inherent differences in data types, storage mechanisms, and constraint implementation among different database management systems significantly increase the risk of data inconsistencies between the source and target databases during synchronization. Such inconsistencies can be caused by various factors such as network interruptions, system failures, resource contention, or heterogeneous type conversion errors, and once they occur, they will directly affect the correct operation of business systems. Currently, the most commonly used data consistency assurance solutions in the industry are based on a periodic data sharding comparison approach. The basic idea of ​​this approach is to divide the source and target data into multiple shards according to a preset time interval, load the entire shard data into memory for a full comparison to detect differences, and then perform data compensation operations.

[0052] However, this traditional approach has significant limitations in practical applications: First, it suffers from low resource utilization efficiency. Existing solutions require loading the entire data shard into memory for comparison, while in reality, only a small number of records within a shard may be inconsistent. This "full loading, partial use" approach results in a huge waste of storage resources, especially when processing data containing large text or large binary objects, where memory consumption is particularly prominent, severely restricting the system's ability to process massive amounts of data. Second, it limits parallel processing capabilities. Different data tables in a database often have complex business logic relationships, especially dependencies established through foreign key constraints. Existing solutions fail to fully consider these inherent dependencies when processing and compensating for data shards. Ignoring dependencies and blindly processing related data shards in parallel may lead to new data inconsistencies due to improper execution order; however, using fully sequential execution to ensure safety would severely sacrifice processing efficiency, failing to fully leverage the advantages of a distributed computing environment. Furthermore, the accuracy of compensation needs improvement. Traditional solutions typically treat the entire data shard as the processing unit, failing to accurately locate specific inconsistent records, resulting in an excessively large scope of compensation operations. This increases the system burden and may also affect the availability of normal data.

[0053] In view of this, this application provides a data consistency verification and compensation method, apparatus, computer storage medium, and electronic device. It aims to accurately capture failed and invalid transactions during the synchronization process as verification objects, identify differences by comparing verification values ​​instead of loading the entire data, and intelligently control the compensation execution order based on data dependencies. This significantly reduces memory usage and computational resource consumption while ensuring data consistency, and improves data compensation efficiency through parallel processing, effectively solving the problems of resource waste and low processing efficiency in traditional solutions.

[0054] Please see Figure 1 , Figure 1 An exemplary system architecture diagram of a data consistency verification and compensation method provided in this application embodiment.

[0055] like Figure 1 As shown, the system architecture may include a terminal 101, a network 102, and a server 103. The network 102 serves as the medium for providing a communication link between the terminal 101 and the server 103. The network 102 may include various types of wired or wireless communication links, such as wired communication links including fiber optic cables, twisted-pair cables, or coaxial cables, and wireless communication links including Bluetooth communication links, Wireless-Fidelity (Wi-Fi) communication links, or microwave communication links, etc.

[0056] Terminal 101 can interact with server 103 via network 102 to receive messages from or send messages to server 103. Alternatively, terminal 101 can interact with server 103 via network 102 to receive messages or data sent to server 103 by other users. Terminal 101 can be hardware or software. When terminal 101 is hardware, it can be various electronic devices, including but not limited to smartwatches, smartphones, tablets, laptops, and desktop computers. When terminal 101 is software, it can be installed in the aforementioned electronic devices and can be implemented as multiple software programs or software modules (e.g., to provide distributed services) or as a single software program or software module; no specific limitation is made here.

[0057] In this embodiment, terminal 101 can detect failed transactions and transactions containing insignificant events during the data synchronization process, record the failed transactions and transactions containing insignificant events, and form a set of transactions to be verified. Among them, insignificant events refer to data operation events that return zero affected rows after execution in the target database. The transactions in the set of transactions to be verified are parsed to obtain multiple independent events. The verification values ​​of the corresponding data rows of the independent events in the source and target ends are compared. Based on the comparison results, events to be compensated are selected to form a set of events to be compensated. The execution dependencies related to the events in the set of events to be compensated are determined. The execution order is controlled according to the execution dependencies. The events in the set of events to be compensated are subjected to corresponding data compensation operations in the target database.

[0058] Server 103 can be a business server providing various services. It should be noted that server 103 can be either hardware or software. When server 103 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When server 103 is software, it can be implemented as multiple software programs or software modules (e.g., used to provide distributed services), or as a single software program or software module; no specific limitations are made here.

[0059] Alternatively, the system architecture may not include server 103. In other words, server 103 may be an optional device in the embodiments of this specification. That is, the method provided in the embodiments of this specification can be applied to a system structure that only includes terminal 101. The embodiments of this application do not limit this.

[0060] It should be understood that Figure 1 The number of terminals, networks, and servers shown is only illustrative; the number can be any number of terminals, networks, and servers depending on the implementation requirements.

[0061] Please see Figure 2 , Figure 2This is a flowchart illustrating a data consistency verification and compensation method provided in an embodiment of this application. The execution entity in this embodiment can be an electronic device performing data consistency verification and compensation, a processor within the electronic device performing the data consistency verification and compensation method, or a data consistency verification and compensation service within the electronic device performing the data consistency verification and compensation method. For ease of description, the following uses a processor within an electronic device as an example to illustrate the specific execution process of the data consistency verification and compensation method.

[0062] like Figure 2 As shown, data consistency verification and compensation methods can include at least the following:

[0063] S201. Detect failed transactions and transactions containing insignificant events during the data synchronization process, record the failed transactions and transactions containing insignificant events, and form a set of transactions to be verified.

[0064] Specifically, during data synchronization, to accurately locate the range of data to be verified and avoid the resource waste caused by full comparison, the data synchronization process can be monitored in real time. When a transaction fails due to execution anomaly, or a transaction contains an insignificant event that returns zero affected rows after execution on the target end, the complete information of these transactions is immediately captured. Specifically, these two types of transactions are identified by parsing database logs or listening to the execution feedback of the synchronization component, and the metadata of these transactions (including transaction identifiers, involved table objects, specific DML operation statements, and related column information, etc.) is persistently stored on disk or a dedicated database, thus forming a clear and concise set of transactions to be verified.

[0065] S202. Parse the transactions in the transaction set to be verified to obtain multiple independent events. Compare the verification values ​​of the corresponding data rows of the independent events at the source and target ends. Based on the comparison results, select the events to be compensated to form a set of events to be compensated.

[0066] Specifically, each transaction in the set can be analyzed one by one, decomposing its database operations into independent events. Each event fully records the specific operation information on a single row of data. Then, an efficient data verification strategy is employed: by querying and comparing the verification values ​​of the corresponding data row for each event on both the source and target ends, this method avoids directly transmitting and storing complete row data, reducing network and memory overhead. Based on the verification value comparison results, the specific events requiring compensation are accurately identified: when there is a difference in data state between the source and target ends, the corresponding event is marked as an event to be compensated.

[0067] In one possible implementation, the checksum is the MD5 hash. For each data row corresponding to an individual event, MD5 query requests are sent to both the source and target databases. A specially optimized query retrieves only the MD5 hash of that data row. After obtaining the MD5 hashes from both ends, the system performs a precise comparison. This method of transmitting only the 32-bit MD5 hash, compared to transmitting the entire data row, especially when handling data rows containing large text or large binary objects, reduces network traffic and memory usage while maintaining the same checksum accuracy as a full data comparison. Please refer to [link to relevant documentation]. Figure 3 , Figure 3 A logical block diagram for data consistency verification provided in this application embodiment is shown below. Figure 3 As shown, the set of transactions to be verified formed by S201 is parsed to obtain multiple independent events. Then, each independent event is traversed, and the MD5 value of the corresponding data row is queried in the relevant tables of the source and target sides according to the primary key value of the event record and compared to form a set of events to be compensated. At the same time, the added independent events can continue to be traversed to perform automatic verification.

[0068] Furthermore, different handling methods are applied to independent events based on different MD5 value comparison scenarios. After the MD5 value comparison is completed, each independent event is precisely categorized based on the comparison results. When the MD5 values ​​of the source and target ends are completely identical, it indicates that the data row corresponding to the event is consistent at both ends, and this event will be ignored without further processing. When the source MD5 value is empty while the target MD5 value is not empty, it means that the data exists at the target end but has been deleted at the source end; this event can be marked as a deletion operation and added to the set of events to be compensated. When the source MD5 value is not empty while the target MD5 value is empty, it indicates that the data exists only at the source end; this event can be marked as an insertion operation and added to the set of events to be compensated. When the MD5 values ​​at both ends exist but are inconsistent, it indicates that there is a deviation in the data content during synchronization; this event can be marked as an update operation and added to the set of events to be compensated. Through this intelligent judgment mechanism based on MD5 value state combinations, all data operation types requiring compensation can be accurately identified, providing clear operational guidance for subsequent compensation execution.

[0069] S203. Determine the execution dependencies related to the events in the set of events to be compensated, control the execution order according to the execution dependencies, and perform corresponding data compensation operations on the target database for the events in the set of events to be compensated.

[0070] Specifically, after obtaining the set of events to be compensated, the execution dependencies between these events are analyzed. By analyzing each event in the set, a dependency graph is constructed. Based on this analysis, the set of prerequisite events that must be completed for each event to execute is recorded. Then, the execution flow is controlled according to the established dependencies: events without prerequisite dependencies are executed in parallel, while events with dependencies are executed sequentially only after all their prerequisite events have been compensated. Finally, the corresponding data insertion, update, or deletion compensation operations are executed in an orderly manner in the target database according to this execution order, ensuring maximum processing efficiency while maintaining data logical consistency.

[0071] In one possible implementation, after determining the set of events to be compensated, the execution dependencies between these events are analyzed. First, by parsing the table constraint information recorded in the events to be compensated, the data dependencies established through foreign keys between the data tables involved in different events are identified. Based on the identified foreign key dependencies, corresponding tagging information is generated for each event, containing a unique identifier for the current event and a set of identifiers for all prerequisite events that must be completed before the current event can be executed. Based on the generated tagging information, a network of execution dependencies between events can be clearly established, providing a decision-making basis for the execution order of subsequent compensation operations. Furthermore, a secondary verification mechanism is implemented before executing the compensation operation. For the data row corresponding to the current event to be compensated, its MD5 checksum is again obtained from both the source and target databases for comparison. If inconsistencies are still found between the two databases, the complete and latest data for that data row is immediately obtained from the source database and used as a benchmark to perform precise data compensation operations on the target database. This mechanism effectively avoids inaccurate compensation due to data changes during the period from the initial verification to actual compensation, ensuring the timeliness and accuracy of data repair. When processing the set of events to be compensated, the execution conditions are determined by querying the set of preceding event identifiers corresponding to each event in the global cache. Based on the query results, if the preceding event set is empty, the compensation operation for that event is initiated immediately or executed in parallel; if there are incomplete preceding events, the event is temporarily stored in a waiting queue and executed only after all its preceding events have been compensated. This mechanism ensures that related events are executed in the correct order through dynamic dependency detection, while fully releasing the parallel processing capabilities of unrelated events.

[0072] It should be noted that after the compensation operation for an event is completed, its corresponding tagging information is simultaneously cleaned up: first, the complete tagging record corresponding to the event is removed from the global cache; then, all other tagging information in the cache is traversed, and the completed event identifier is removed from its preceding event set. This maintenance mechanism ensures that dependencies are updated in real time, releases the logical dependency resources occupied by completed events, and activates execution conditions for subsequent events in the waiting queue. Please refer to [link to relevant documentation]. Figure 4 , Figure 4 A data compensation logic block diagram provided for an embodiment of this application, such as Figure 4 As shown, after acquiring an event, its preceding list is queried. If the preceding list is empty, compensation can be performed in parallel. If the preceding list is not empty, the preceding event is placed in the preceding queue for execution. To prevent changes in the source and target data before compensation, for events eligible for data compensation, the MD5 value corresponding to the event is first queried on both the source and target sides, and the data is compared. If they are inconsistent, the source data takes precedence, the latest data from the source is retrieved, and compensation is performed on the target side. After data compensation is complete, the preceding event corresponding to the event is removed from the global cache, and the relevant events are notified to update their preceding lists. If they are consistent, the corresponding event is deleted. After compensation is completed, the next event is automatically acquired, and the above steps are executed.

[0073] This application provides a data consistency verification and compensation method. By accurately identifying and recording failed transactions and transactions containing insignificant events during data synchronization, the scope of data verification is narrowed, avoiding the resource waste caused by full data comparison in traditional solutions, improving verification efficiency, and reducing unnecessary system overhead. A lightweight comparison mechanism based on MD5 checksums is adopted, which reduces network transmission load and memory consumption by querying and comparing only the hash values ​​of data rows rather than the complete data content. This is particularly effective when processing tables containing large amounts of data such as large text or binary objects, saving resources while maintaining the same verification accuracy as full data comparison. By establishing an intelligent scheduling mechanism based on data dependencies, analyzing inter-table constraints, and constructing an event execution dependency graph, sequential execution of dependent events and parallel processing of unrelated events are achieved. This ensures data logical consistency while improving compensation efficiency, solving the technical challenge of balancing parallelism and data consistency in traditional solutions. Furthermore, a secondary verification mechanism was introduced before compensation. This mechanism re-verifies the data status before executing the compensation operation, ensuring that the repair is based on the latest data from the source. This effectively avoids inaccurate compensation caused by changes in data status, improving the reliability of data repair and the robustness of the system. By establishing a complete tagging information maintenance mechanism, the dependencies in the global cache are updated in real time after compensation, ensuring continuous synchronization of the system status. This provides an accurate basis for subsequent compensation operations, forming a complete closed loop for data consistency assurance.

[0074] Please see Figure 5 , Figure 5 This is a structural block diagram of a data consistency verification and compensation device provided in an embodiment of this application. Figure 5 As shown: The data consistency verification and compensation device includes: a recording module 510, a comparison module 520, and an execution module 530.

[0075] The recording module 510 is used to detect failed transactions and transactions containing insignificant events during the data synchronization process, record failed transactions and transactions with insignificant events to form a set of transactions to be verified; among them, insignificant events refer to data operation events that return zero affected rows after execution in the target database;

[0076] The comparison module 520 is used to parse the transactions in the transaction set to be verified, obtain multiple independent events, compare the verification values ​​of the corresponding data rows of the independent events at the source end and the target end, and filter out the events to be compensated based on the comparison results to form a set of events to be compensated.

[0077] The execution module 530 is used to determine the execution dependencies related to events in the set of events to be compensated, control the execution order based on the execution dependencies, and perform corresponding data compensation operations on the target database for the events in the set of events to be compensated.

[0078] In some possible embodiments, the checksum is the MD5 hash; the comparison module 520 includes:

[0079] The query unit is used to query the data rows corresponding to the source and target ends of an independent event.

[0080] The comparison unit is used to compare the MD5 value of the corresponding data row at the source end of an independent event with the MD5 value of the corresponding data row at the target end.

[0081] In some possible embodiments, the comparison module 520 includes:

[0082] The ignore unit is used to ignore the corresponding independent event when the source MD5 value and the target MD5 value are the same;

[0083] The first determining unit is used to add the corresponding independent event to the set of events to be compensated and determine it as the event that needs to be deleted when the source MD5 value is empty and the target MD5 value is not empty.

[0084] The second determining unit is used to add the corresponding independent event to the set of events to be compensated and determine it as the event that needs to be inserted when the source MD5 value is not empty and the target MD5 value is empty.

[0085] The third determining unit is used to add the corresponding independent event to the set of events to be compensated and determine it as the event that needs to be updated when the source MD5 value and the target MD5 value are not empty and the source MD5 value and the target MD5 value are inconsistent.

[0086] In some possible embodiments, the execution module 530 includes:

[0087] The analysis unit is used to analyze the foreign key dependencies between data tables involved in different events based on the table constraint information recorded in the events to be compensated.

[0088] The tagging unit is used to generate corresponding tagging information for each event based on foreign key dependencies. The tagging information includes the unique identifier of the current event and the set of prerequisite event identifiers that must be completed before the event can be executed.

[0089] The fourth determining unit is used to determine the execution dependencies related to the event based on the tagging information.

[0090] In some possible embodiments, the execution module 530 includes:

[0091] The query unit is used to query the set of preceding event identifiers corresponding to each target event in the set of events to be compensated; the tagging information is stored in a global cache.

[0092] The first execution unit is used to immediately or in parallel execute the event compensation operation if the set of preceding event identifiers is empty.

[0093] The second execution unit, if the set of preceding event identifiers is not empty, puts the event into the waiting queue and executes it only after all preceding events have completed the compensation operation.

[0094] In some possible embodiments, the execution module 530 includes:

[0095] The comparison unit is used to query the MD5 value of the data row corresponding to the event to be compensated from the source and target ends again and compare them.

[0096] The compensation unit is used to retrieve the latest data from the source end and perform compensation operations on the target end based on the latest data when the comparison results are still inconsistent.

[0097] In some possible embodiments, the data consistency verification and compensation device 500 further includes:

[0098] The deletion module is used to remove the tag information corresponding to events that have been compensated from the global cache;

[0099] The removal module is used to check other tag information in the global cache and remove the event identifiers that have been compensated from the set of preceding event identifiers of other tag information.

[0100] It should be noted that the data consistency verification and compensation device provided in the above embodiments is only illustrated by the division of the above functional modules when executing the data consistency verification and compensation method. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the data consistency verification and compensation device and the data consistency verification and compensation method embodiments provided in the above embodiments belong to the same concept, and the implementation process is detailed in the method embodiments, which will not be repeated here.

[0101] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0102] Please see Figure 6 , Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 6 As shown, the electronic device 600 may include: at least one processor 601, at least one network interface 604, a user interface 603, a memory 605, and at least one communication bus 602.

[0103] The communication bus 602 is used to enable communication between these components.

[0104] The user interface 603 may include a display screen and a camera. Optional user interfaces 603 may include standard wired interfaces and wireless interfaces.

[0105] The network interface 604 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface).

[0106] The processor 601 may include one or more processing cores. The processor 601 connects to various parts within the electronic device 600 using various interfaces and lines, and performs various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 605, and by calling data stored in the memory 605. Optionally, the processor 601 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 601 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content required for display; and the modem handles wireless communication. It is understood that the modem may also not be integrated into the processor 601 and may be implemented as a separate chip.

[0107] The memory 605 may include random access memory (RAM) or read-only memory. Optionally, the memory 605 may include a non-transitory computer-readable storage medium. The memory 605 may be used to store instructions, programs, code, code sets, or instruction sets. The memory 605 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above-described method embodiments, etc.; the data storage area may store data involved in the above-described method embodiments, etc. Optionally, the memory 605 may also be at least one storage device located remotely from the aforementioned processor 601. Figure 6 As shown, the memory 605, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and a data consistency verification and compensation application.

[0108] exist Figure 6In the electronic device 600 shown, the user interface 603 is mainly used to provide an input interface for the user and obtain the user input data; while the processor 601 can be used to call the data consistency verification and compensation application stored in the memory 605, and specifically perform the following operations: detect failed transactions and transactions containing insignificant events during the data synchronization process, record failed transactions and transactions with insignificant events to form a set of transactions to be verified; among them, insignificant events refer to data operation events that return zero affected rows after execution in the target database; parse the transactions in the set of transactions to be verified to obtain multiple independent events, compare the verification values ​​of the corresponding data rows of the independent events in the source and target ends, filter out the events to be compensated based on the comparison results, and form a set of events to be compensated; determine the execution dependencies related to the events in the set of events to be compensated, control the execution order according to the execution dependencies, and perform corresponding data compensation operations on the events in the set of events to be compensated in the target database.

[0109] In some possible embodiments, the checksum is the MD5 hash; when processor 601 performs the checksum comparison of the corresponding data rows of independent events at the source and target ends, it specifically performs the following:

[0110] Query the data rows corresponding to the source and target of the independent event; compare the MD5 value of the data row corresponding to the source of the independent event with the MD5 value of the data row corresponding to the target.

[0111] In some possible embodiments, when processor 601 performs the task of filtering out events to be compensated based on the comparison results to form a set of events to be compensated, it specifically performs the following:

[0112] When the source MD5 value matches the target MD5 value, the corresponding independent event is ignored. When the source MD5 value is empty and the target MD5 value is not empty, the corresponding independent event is added to the event set to be compensated and identified as an event requiring deletion. When the source MD5 value is not empty and the target MD5 value is empty, the corresponding independent event is added to the event set to be compensated and identified as an event requiring insertion. When both the source and target MD5 values ​​are not empty and the source and target MD5 values ​​are inconsistent, the corresponding independent event is added to the event set to be compensated and identified as an event requiring update.

[0113] In some possible embodiments, processor 601 performs the determination of execution dependencies related to events in the set of events to be compensated, specifically for performing:

[0114] Based on the table constraint information recorded in the events to be compensated, analyze the foreign key dependencies between the data tables involved in different events; generate corresponding tag information for each event according to the foreign key dependencies, the tag information includes the unique identifier of the current event and the set of predecessor event identifiers that must be completed for the event to be executed; determine the execution dependencies related to the events based on the tag information.

[0115] In some possible embodiments, when processor 601 executes the execution order based on execution dependencies, it is specifically used to execute:

[0116] Query the set of preceding event identifiers corresponding to each target event in the set of events to be compensated; the marking information is stored in the global cache; if the set of preceding event identifiers is empty, the compensation operation of the event is executed immediately or in parallel; if the set of preceding event identifiers is not empty, the event is placed in the waiting queue until all preceding events of the event have completed the compensation operation before execution.

[0117] In some possible embodiments, before the processor 601 performs the corresponding data compensation operation on the events in the set of events to be compensated, it controls the execution order according to the execution dependency and then performs the following:

[0118] The MD5 values ​​of the data rows corresponding to the events to be compensated are queried again from the source and target ends and compared. If the comparison results are still inconsistent, the latest data is retrieved from the source end and the compensation operation is performed on the target end based on the latest data.

[0119] In some possible embodiments, after the processor 601 performs the corresponding data compensation operation on the target database for the events in the set of events to be compensated, it is also used to perform:

[0120] Remove the tag information corresponding to the events that have been compensated from the global cache; check other tag information in the global cache and remove the event identifiers that have been compensated from the set of preceding event identifiers of other tag information.

[0121] This application also provides a computer-readable storage medium storing instructions that, when executed on a computer or processor, cause the computer or processor to perform the above-described instructions. Figure 2 One or more steps in the illustrated embodiment. If the constituent modules of the above-described data consistency verification and compensation device are implemented as software functional units and sold or used as independent products, they can be stored in the computer-readable storage medium.

[0122] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., digital versatile discs (DVDs)), or semiconductor media (e.g., solid state disks (SSDs)).

[0123] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The aforementioned storage medium includes various media capable of storing program code, such as read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks. Unless otherwise specified, the technical features of this embodiment and its implementation schemes can be combined arbitrarily.

[0124] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. It will be apparent to those skilled in the art that various modifications can be made to these embodiments, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for data consistency check and compensation, characterized in that, The method comprises: detecting a failed transaction and a transaction containing a non-impact event in a data synchronization process, recording the failed transaction and the non-impact event, and forming a to-be-verified transaction set; wherein the non-impact event refers to a data operation event that returns zero impact rows after execution on a target end database; analyzing transactions in the to-be-verified transaction set to obtain a plurality of independent events, comparing the independent events with corresponding data rows at a source end and a target end, and screening out to-be-compensated events according to a comparison result to form a to-be-compensated event set; based on table constraint information recorded in the to-be-compensated events, analyzing foreign key dependency relationships between data tables involved in different events to determine execution dependency relationships of events in the to-be-compensated event set, controlling an execution order according to the execution dependency relationships, and executing corresponding data compensation operations on the events in the to-be-compensated event set in the target end database.

2. The method of claim 1, wherein, The check value is an MD5 value; The comparison of the check values of the independent events with corresponding data rows at the source end and the target end comprises: querying data rows corresponding to the independent events at the source end and the target end; comparing MD5 values of the data rows corresponding to the independent events at the source end with MD5 values of the data rows corresponding to the independent events at the target end.

3. The method of claim 2, wherein, The screening of the to-be-compensated events according to the comparison result to form the to-be-compensated event set comprises: when the source end MD5 value is consistent with the target end MD5 value, ignoring the corresponding independent event; when the source end MD5 value is empty and the target end MD5 value is not empty, adding the corresponding independent event to the to-be-compensated event set and determining the corresponding independent event as an event requiring a deletion operation; when the source end MD5 value is not empty and the target end MD5 value is empty, adding the corresponding independent event to the to-be-compensated event set and determining the corresponding independent event as an event requiring an insertion operation; when the source end MD5 value and the target end MD5 value are both not empty and the source end MD5 value is inconsistent with the target end MD5 value, adding the corresponding independent event to the to-be-compensated event set and determining the corresponding independent event as an event requiring an update operation.

4. The method of claim 1, wherein, The determination of the execution dependency relationships of the events in the to-be-compensated event set comprises: generating corresponding marking information for each event according to the foreign key dependency relationships, wherein the marking information contains a unique identifier of a current event and a set of pre-event identifiers that must be completed for event execution; determining the execution dependency relationships of the events according to the marking information.

5. The method of claim 4, wherein, The control of the execution order according to the execution dependency relationships comprises: querying a set of pre-event identifiers corresponding to each target event in the to-be-compensated event set; wherein the marking information is stored in a global cache; if the set of pre-event identifiers queried is empty, immediately or in parallel executing a compensation operation of the target event; if the set of pre-event identifiers queried is not empty, putting the target event into a waiting queue until all pre-events of the target event complete the compensation operation and then executing the compensation operation of the target event.

6. The method of claim 1, wherein, The method further comprises: Again, the MD5 value of the data row corresponding to the event to be compensated is queried from the source end and the target end and compared; When the comparison result is still inconsistent, the latest data is pulled from the source end, and compensation operation is performed at the target end based on the latest data.

7. The method of claim 5, wherein, After the corresponding data compensation operation is performed on the events in the event set to be compensated at the target end database, the method further comprises: The marker information corresponding to the event that has completed compensation is deleted from the global cache; The other marker information in the global cache is checked, and the event identifier that has completed compensation is removed from the set of front event identifiers of the other marker information.

8. A data consistency check and compensation apparatus, characterized by comprising: The device comprises: A recording module is configured to detect failed transactions and transactions containing non-impact events in a data synchronization process, record the failed transactions and the transactions containing non-impact events, and form a set of transactions to be verified; wherein the non-impact event refers to a data operation event that returns zero impact rows after being executed at a target end database; A comparison module is configured to analyze transactions in the set of transactions to be verified, obtain a plurality of independent events, compare the check values of the independent events corresponding to data rows at a source end and a target end, filter out events to be compensated according to a comparison result, and form a set of events to be compensated; An execution module is configured to analyze the foreign key dependency relationship between data tables involved in different events based on table constraint information recorded in the events to be compensated, determine the execution dependency relationship of events related in the set of events to be compensated, control the execution order according to the execution dependency relationship, and execute corresponding data compensation operations on the events in the set of events to be compensated at a target end database.

9. A computer storage medium, characterized in that The computer storage medium stores a plurality of instructions, and the instructions are suitable for being loaded and executed by a processor to perform the steps of the method according to any one of claims 1-7.

10. An electronic device, comprising: The computer program is stored in the memory and can be run on the processor, and the processor executes the program to implement the steps of the method according to any one of claims 1-7.

Images