Method and apparatus for data compensation

Abstract

This invention discloses a method and apparatus for data compensation, relating to the field of computer technology. One specific implementation of the method includes: obtaining the time points for extracting source and target data; extracting key data information from the source and target ends based on the time points; performing data consistency verification between the source and target ends based on the key data information; determining the target data fragment to be compensated based on the verification results; extracting source data from each target data fragment based on the verification results; loading the extracted source data into a target shadow table; and compensating the target formal table based on the verification results and the target shadow table. This implementation can quickly, accurately, and efficiently repair data replication results without affecting existing data replication and while ensuring normal business processing of the target application as much as possible.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method and apparatus for data compensation. Background Technology

[0002] Data consistency replication technology is a logical replication technique based on data consistency verification and data compensation. Data consistency verification is a mechanism used to ensure the consistency of data between the source and target ends. It typically records the differences between the source and target data, and then uses a data compensation mechanism to compensate for these differences based on the verification results, thereby achieving the goal of data replication consistency. In this data consistency replication technology, the data object being replicated is generally called the source data, and the data being repaired is generally called the target data. Because the source and target ends of this data consistency replication technology are completely decoupled, it does not logically intrude on existing applications, and therefore can be well applied to application scenarios such as data read / write separation, multi-site active-active disaster recovery, and data governance.

[0003] In the process of realizing this invention, the inventors discovered at least the following problems in the prior art:

[0004] Because both the source and target ends of this data consistency replication technology are readable and writable data sources, discrepancies may arise due to data changes during the consistency replication process. Data that should be consistent between the source and target ends may become inconsistent. At this point, the target end may have already begun accepting business operations. Therefore, how to quickly compensate for data discrepancies without affecting business processing and data consistency replication at the target end is a major challenge for current data consistency replication technologies. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide a data compensation method and apparatus that can quickly, accurately and efficiently repair the data consistency replication system without affecting existing data replication and while ensuring normal business processing of the target application as much as possible, and can guarantee the accuracy of the data compensation results.

[0006] To achieve the above objectives, according to one aspect of the present invention, a data compensation method is provided.

[0007] A data compensation method includes:

[0008] Obtain the time points for extracting source and target data;

[0009] Based on the time points mentioned above, extract key data information from the source and target ends respectively, perform data consistency verification between the source and target ends based on the key data information, and determine the target end data fragments that need to be compensated based on the verification results.

[0010] For each target data segment, the source data is extracted based on the verification result, and the extracted source data is loaded into the target shadow table;

[0011] Based on the verification results and the target-side shadow table, data compensation is performed on the target-side official table.

[0012] Optionally, the method further includes:

[0013] After the data compensation is completed, the data consistency of the source and target official tables is checked again to verify whether the data compensation has passed.

[0014] If data compensation fails, the target end formal table is compensated again using the data compensation method according to the present invention until data compensation succeeds.

[0015] Optionally, the key data information includes the data shard to which the data belongs and the maximum timestamp of each data shard;

[0016] The data consistency verification between the source and target ends based on the aforementioned key data information includes:

[0017] For each source data shard included in the source end, if the maximum timestamp of the source data shard and the target data shard corresponding to the source data shard are consistent, then the data hash values ​​of the source data shard and the target data shard are compared. If the data hash values ​​are the same, the data consistency check passes.

[0018] If the maximum timestamp of the source data shard is greater than the maximum timestamp of the target data shard corresponding to the source data shard, then the data consistency check between the source data shard and the target data shard fails.

[0019] If the maximum timestamp of the source data shard is greater than the time point, then the time point for extracting the source data and target data is re-acquired, and the application corresponding to the source data is terminated from service acceptance.

[0020] Optionally, the time points for obtaining the source and target data include:

[0021] The maximum timestamp of the source data is used as the time point for extracting the source and target data.

[0022] The time points at which the source and target data are re-acquired include:

[0023] The maximum timestamp of the source data fragment is used as the time point for extracting the source and target data.

[0024] Optionally, data compensation for the target-side official table based on the verification result and the target-side shadow table includes:

[0025] Based on the verification result and the target end shadow table, the data to be compensated is determined from the target end shadow table. The data to be compensated corresponds to a business status, and the business status includes the final state of the business and non-final state of the business. The final state of the business is the final state of the business.

[0026] For data awaiting compensation that is in the final stage of the business process, update the target end formal table based on the data awaiting compensation.

[0027] For data to be compensated that is not in the final business state, the data compensation operation is performed on the target end formal table according to the business elements corresponding to the data to be compensated.

[0028] Optionally, the data compensation operation performed on the target end's formal table based on the business elements corresponding to the data to be compensated includes:

[0029] If the data to be compensated does not exist in the target end official table when compensating for the target end data, then the data to be compensated is inserted into the target end official table.

[0030] If, during the compensation of data in the target-side official table, the data to be compensated exists in the target-side official table, but the business status of the data to be compensated in the target-side official table is different from the business status of the data to be compensated in the target-side shadow table, then the target-side shadow table is updated according to the data to be compensated in the target-side shadow table, and the data to be compensated in the target-side shadow table is marked as invalid.

[0031] Optionally, the method further includes:

[0032] When performing data compensation on the target end's official table, stop the business processing of applications on both the source and target ends, and record the duration of the business processing suspension.

[0033] Optionally, the extraction of source data and the loading of the extracted source data into the target shadow table are performed concurrently.

[0034] According to another aspect of the present invention, a data compensation apparatus is provided.

[0035] A data compensation device, comprising:

[0036] The time point determination module is used to obtain the time points for extracting source and target data;

[0037] The consistency verification module is used to extract key data information from the source and target ends according to the time points, perform data consistency verification between the source and target ends according to the key data information, and determine the target end data fragments that need to be compensated based on the verification results.

[0038] The data loading module is used to shard the data of each target end, extract the source end data according to the verification result, and load the extracted source end data into the target end shadow table;

[0039] The data compensation module is used to perform data compensation on the target end formal table based on the verification result and the target end shadow table.

[0040] According to another aspect of the present invention, an electronic device for data compensation is provided.

[0041] An electronic device for data compensation includes: one or more processors; and a storage device for storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement the data compensation method provided in the embodiments of the present invention.

[0042] According to another aspect of the present invention, a computer-readable medium is provided.

[0043] A computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the data compensation method provided in embodiments of the present invention.

[0044] One embodiment of the above invention has the following advantages or beneficial effects: By obtaining the time points for extracting source and target data; extracting key information of the source and target data according to the time points; performing data consistency verification between the source and target based on the key information; determining the target data shards to be compensated based on the verification results; extracting source data for each target data shard based on the verification results; loading the extracted source data into the target shadow table; and performing data compensation operations on the target formal table based on the verification results and the target shadow table, the technical solution for data consistency processing can quickly, accurately, and efficiently repair the data consistency replication system without affecting existing data replication and while ensuring normal business processing of the target application as much as possible. By performing data consistency verification of the source and target formal tables after data compensation is completed to verify whether data compensation has been successfully completed, the accuracy of the data consistency processing results can be guaranteed.

[0045] The further effects of the aforementioned unconventional alternative methods will be explained below in conjunction with specific implementation methods. Attached Figure Description

[0046] The accompanying drawings are provided to better understand the invention and are not intended to unduly limit the scope of the invention. Wherein:

[0047] Figure 1 This is a schematic diagram illustrating the main steps of a data compensation method according to an embodiment of the present invention;

[0048] Figure 2 This is a schematic diagram of a data compensation processing procedure according to an embodiment of the present invention;

[0049] Figure 3 This is a schematic diagram of the main modules of a data compensation device according to an embodiment of the present invention;

[0050] Figure 4 This is an exemplary system architecture diagram in which embodiments of the present invention can be applied;

[0051] Figure 5 This is a schematic diagram of the structure of a computer system suitable for implementing terminal devices or servers of the present invention. Detailed Implementation

[0052] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of the present invention, including various details to aid understanding. These details should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0053] With rapid business growth, many enterprises face several challenges with single-region technology architectures, including: limited infrastructure restricting business scalability; and data center and city-level failures impacting service sustainability. To address these issues, enterprises choose to build multi-site active-active architectures, establishing multiple units (business centers) in the same city or different locations. These business units can be distributed across different regions, effectively resolving the infrastructure scalability limitations and service sustainability issues inherent in single-region deployments.

[0054] To overcome the data replication consistency problems existing in the prior art, this invention provides a data compensation method and apparatus to quickly, accurately, and efficiently repair data consistency replication results without affecting existing data replication or the normal business processing of the target application. In embodiments of this invention, data centers (or server rooms) may include multiple data centers, which can operate simultaneously, and each data center may be deployed in different regions.

[0055] Figure 1This is a schematic diagram illustrating the main steps of a data compensation method according to an embodiment of the present invention. Figure 1 As shown, the data compensation method of this embodiment mainly includes the following steps S101 to S104.

[0056] Step S101: Obtain the time points for extracting source and target data. In a data consistency replication system, a fault detection program is set up to promptly detect system faults, thereby determining the time points for extracting source and target data based on the time of fault detection. In one embodiment of the invention, when obtaining the time points for extracting source and target data, for example, the maximum timestamp of the source data can be used as the time point for extraction. That is, consistency verification and data compensation are performed based on all current source data. Generally, data compensation operations are performed periodically, such as once every hour, or when the fault detection program detects a fault.

[0057] Step S102: Extract key data information from the source and target ends based on the time points, perform data consistency verification between the source and target ends based on the key data information, and determine the target end data shards that need data compensation based on the verification results. When extracting key data information, data within the time period from the last successful data consistency verification point to the time point determined in step S101 can be extracted, and consistency verification can be performed on the data within this time period to determine the data shards that need data compensation. Generally, for ease of data management, business application data is sharded. Specifically, data sharding can be based on the person who generated the application data, the region, etc. For example, data can be divided into data shards corresponding to user A and user B; or data can be divided into data shards corresponding to region C and region D, etc.

[0058] According to one embodiment of the present invention, the key data information includes the data shard to which the data belongs and the maximum timestamp of each data shard. Additionally, the key data information may also include information such as the data identifier and the application system corresponding to the data, which can be obtained according to application needs. When performing data consistency verification between the source and target ends based on the key data information, the following steps can be followed:

[0059] For each source data shard included in the source end, if the maximum timestamp of the source data shard and the target data shard corresponding to the source data shard are consistent, then the data hash values ​​of the source data shard and the target data shard are compared. If the data hash values ​​are the same, the data consistency check passes.

[0060] If the maximum timestamp of the source data shard is greater than the maximum timestamp of the target data shard corresponding to the source data shard, then the data consistency check between the source data shard and the target data shard fails.

[0061] If the maximum timestamp of the source data shard is greater than the time point, then the time point for extracting the source data and target data is re-acquired, and the application corresponding to the source data is terminated from service acceptance.

[0062] In this embodiment, when the maximum timestamp of the source data shard and the target data shard corresponding to the source data shard are the same, it can be preliminarily determined that the source data shard and the target data shard are consistent. To obtain a more accurate consistency verification result, the data hash values ​​of the source data shard and the target data shard can also be compared. The data hash value is obtained by performing a hash algorithm on the data shards; if the data hash values ​​are the same, the data shards must be consistent. Additionally, the MD5 message digest of the data shard can be calculated using a hash algorithm, and the source data shard and the target data shard can be compared based on the MD5 message digest. When the maximum timestamp of the source data shard is greater than the maximum timestamp of the target data shard corresponding to the source data shard, it means that some data has been lost, and therefore the data consistency verification fails. If the maximum timestamp of the source data shard is greater than the aforementioned timestamp for extracting source and target data, it indicates a problem with obtaining this timestamp. Since the source was still processing business logic after this timestamp was obtained, and the source data has still changed, the data consistency processing result will be inaccurate. Therefore, it is necessary to re-obtain the timestamps for extracting source and target data, and then re-perform data acquisition and consistency verification. It should be noted that the application corresponding to the source data should have its business operations terminated during this process to prevent similar situations from recurring.

[0063] After consistency verification, the data shard information requiring data compensation is saved to the consistency verification result table. Specifically, a data consistency verification result table is first established on the target side. This result table contains the data shard result set corresponding to each data object (e.g., a person or a region). After performing consistency verification on the data shards, the target-side data shards requiring data compensation can be determined and saved to this result table.

[0064] According to an embodiment of the present invention, when determining the time point for extracting source - end data and target - end data, the time point for extracting source - end data and target - end data is determined according to the time when the fault is detected. In an embodiment of the present invention, after the fault is detected, the key information of the source - end and target - end data will be extracted for consistency verification. At this time, the maximum timestamp of the source - end data can be determined as the time point for extracting source - end data and target - end data to determine the data compensation time point as accurately as possible.

[0065] In an embodiment of the present invention, assume that the time point for extracting source - end data and target - end data is t1. For a certain data shard A at the source - end and the corresponding data shard A' at the target - end (i.e., copied from data shard A), when the maximum timestamp of the source - end data shard A, "t2.source - end", is consistent with the maximum timestamp of the corresponding target - end data shard A', "t2.target - end", generally, it is considered that the data of the two is consistent. For the sake of insurance, the hash values of data shard A and data shard A' will be compared to further ensure the consistency of the data in these two data shards. When t2.source - end and t2.target - end are inconsistent, the data in data shard A and data shard A' must be inconsistent. At this time, data compensation for the target - end data shard A' is required. Among them, the inconsistency between t2.source - end and t2.target - end includes two cases: t2.source - end > t2.target - end. At this time, data compensation needs to be executed, and the time point of t2.source - end is sent to the target - end data loading thread, and the data before the time point of t2.source - end is extracted from the source - end and sent to the target - end, and the target - end shadow table loads the data. t2.source - end < t2.target - end, and this situation does not exist. In addition, if t2.source - end > t1, after the fault occurs, the source - end application is still providing business services, so the data is still being updated. At this time, step S101 needs to be executed again to re - determine the time point for extracting source - end data and target - end data, and the business acceptance of the application corresponding to the source - end data is terminated.

[0066] The present invention can ensure that there is no data change in the source - end data shard based on the time point t1 by stopping business acceptance, user locking, write locking, etc. corresponding to the source - end data shard. In an embodiment of the present invention, when re - determining the time point for extracting source - end data and target - end data, the maximum timestamp of the data included in the certain source - end data shard can be determined as the time point for extracting source - end data and target - end data. In an embodiment of the present invention, the extraction of source - end data and the loading of the extracted source - end data into the target - end shadow table are processed concurrently. In this way, the data compensation time can be saved as much as possible and the data compensation efficiency can be improved.

[0067] Step S103: For each target data shard, extract source data according to the verification result and load the extracted source data into the target shadow table. For each data shard requiring data compensation, extract source data and load it into the target shadow table according to the consistency verification result of the data shard. This avoids terminating service acceptance for the target application during data extraction and loading, thereby ensuring service availability and minimizing downtime. According to an embodiment of the present invention, if the data consistency verification result between source data shard A and the corresponding target data shard A' fails, it indicates that target data shard A' needs data compensation. In this case, the data of source data shard A can be extracted and loaded into the target shadow table. During the process of extracting and loading the data of source data shard A into the target shadow table, the target formal table can normally accept service requests.

[0068] In embodiments of the present invention, the target-side shadow table can be, for example, a new data table automatically created based on the table structure of the target-side official table, with its name generated by adding a prefix or suffix to the name of the target-side official table. Since the shadow table has the same table structure as the official table, the data in the official table can be easily updated based on the shadow table.

[0069] Step S104: Based on the verification result and the target shadow table, perform data compensation on the target formal table.

[0070] According to an embodiment of the present invention, when performing data compensation on the target-side formal table based on the verification result and the target-side shadow table, the specific steps may include:

[0071] Based on the verification result and the target end shadow table, the data to be compensated is determined from the target end shadow table. The data to be compensated corresponds to a business status, and the business status includes the final state of the business and non-final state of the business. The final state of the business is the final state of the business.

[0072] For data awaiting compensation that is in the final stage of the business process, update the target end formal table based on the data awaiting compensation.

[0073] For data to be compensated that is not in the final business state, the data compensation operation is performed on the target end formal table according to the business elements corresponding to the data to be compensated.

[0074] Specifically, both the source and target data have a "business status" attribute to identify the corresponding business state. Business status includes final and non-final states, with the final state being the defined end state of the business. For example, for an order business, its business states include "order placed," "shipped," "delivering," "signed for," and "received." The final state of the order business is set as "received." Correspondingly, other business states such as "order placed," "shipped," "delivering," and "signed for" are non-final states. For each piece of data awaiting compensation in the final state, it can be directly compared with the data awaiting compensation in the target's official table. If the target's official table already contains the same data, no action is needed; otherwise, the data awaiting compensation in the final state can be directly inserted into the target's official table. For each piece of data awaiting compensation in a non-final state, the appropriate compensation method can be determined based on the corresponding business elements, such as the order number from the order system or the transaction number from the transaction operation.

[0075] According to an embodiment of the present invention, the data compensation operation performed on the target end formal table based on the business elements corresponding to the data to be compensated may specifically include:

[0076] If the data to be compensated does not exist in the target end official table when compensating for the target end data, then the data to be compensated is inserted into the target end official table.

[0077] If, during the compensation of data in the target-side official table, the data to be compensated exists in the target-side official table, but the business status of the data to be compensated in the target-side official table is different from the business status of the data to be compensated in the target-side shadow table, then the target-side shadow table is updated according to the data to be compensated in the target-side shadow table, and the data to be compensated in the target-side shadow table is marked as invalid.

[0078] In an embodiment of the present invention, for each data shard requiring data compensation, the maximum timestamp of the source data shard is used as the data compensation time point during data compensation, and the start time t3 of the data compensation is recorded. For a target data shard A' requiring data compensation, the maximum timestamp t2 of the data in its corresponding source data shard A is the data compensation time. The data to be compensated can be determined based on the verification result and the target shadow table, and data compensation is then performed. Specifically, during data compensation, for data in a final state prior to the data compensation time point t2 (e.g., an order has been received), an insert / update operation is performed from the target-side shadow table to the target-side official table. For data in a non-final state prior to the data compensation time point t2, the target-side official table data status is determined according to business elements (e.g., order number in the order system, transaction number in the transaction system, etc.): If the target-side official table does not contain this non-final data from the source end at the data compensation start time t3, an insert operation is performed from the target-side shadow table to the target-side official table; if the target-side official table contains data with the same business elements but different business states at t3 (e.g., order status "paid" and "in delivery" are different business states), the corresponding data is marked as discarded in the target-side shadow table before being inserted into the target-side official table. After execution, time point t4 is recorded, and the time interval t3-t4 is the target application's external service downtime. During the data compensation operation, the target application is taken out of service, and the downtime is recorded.

[0079] According to an embodiment of the present invention, when extracting source data and loading it into the target shadow table, the target official table can normally handle business during the time interval t2-t3, while the time interval t2-t3 is used to process the loading of target shadow table data. During the time interval t3-t4, the target application is shut down. In the prior art, when performing data consistency compensation processing, both the source and target applications are shut down upon detection of a fault, resulting in a downtime of t1-t4. However, in this invention, downtime only needs to be performed within the time interval t3-t4, thus significantly reducing downtime.

[0080] Based on steps S101 to S104 above, the following technical solution can be implemented to achieve data consistency processing: obtaining the time points for extracting source and target data; extracting key data information from the source and target based on these time points; performing data consistency verification on the source and target based on the key data information; determining the target data shards to be compensated based on the verification results; extracting source data from each target data shard based on the verification results; loading the extracted source data into the target shadow table; and performing data compensation operations on the target formal table based on the verification results and the target shadow table. This solution allows for rapid, accurate, and efficient repair of the data consistency replication system without affecting existing data replication and while ensuring normal business processing of the target application as much as possible.

[0081] According to one embodiment of the present invention, after the data compensation operation is completed, a data consistency check is performed again on the source and target official tables to verify whether the data compensation has been successfully completed. If the data compensation fails, data compensation is performed again on the target official table according to steps S101 to S104 described above until the data compensation is successful. By performing a data consistency check on the source and target official tables after the data compensation is completed to verify whether the data compensation has been successfully completed, the accuracy of the data consistency processing results can be guaranteed.

[0082] Compared with existing technologies, the present invention provides a rapid repair method for data consistency replication. This method first determines the data compensation time point t1 for the source and target ends; based on the determined time point t1, it extracts key information from the official tables of the source and target ends respectively, determines the maximum timestamp of the source and target official table data within the determined time point t1, and places the specified data shard consistency verification results into the target end data consistency verification result table; referring to the verification result table, it loads the source data into the target end shadow table; referring to the target end shadow table data and the target end official table data, it performs data compensation operations; after data compensation is completed, it performs a data consistency verification of the source and target end official tables to verify whether the data compensation has been successfully completed. This invention can quickly, accurately, and efficiently repair the data consistency replication system without affecting existing data replication and normal business processing of the target end application.

[0083] Figure 2 This is a schematic diagram of the data compensation process according to an embodiment of the present invention. Figure 2 As shown, the source end includes the source end official table, and the target end includes the target end shadow table, the target end official table, and the verification result table. The main process for data compensation is as follows:

[0084] S1: By monitoring the data in the source and target official tables, the time points for extracting source and target data can be determined.

[0085] S2: Perform consistency checks on the data in the source official table and the data in the target official table, and save the check results to the check result table;

[0086] S3: Based on the verification results, extract data from the source end and load the extracted source end data into the target end shadow table;

[0087] S4: Based on the verification results and the target side shadow table, perform data compensation processing on the target side official table.

[0088] Figure 3 This is a schematic diagram of the main modules of a data compensation device according to an embodiment of the present invention. Figure 3 As shown, the data compensation device 300 of this embodiment mainly includes a time point determination module 301, a consistency verification module 302, a data loading module 303, and a data compensation module 304.

[0089] The time point determination module 301 is used to obtain the time points for extracting source data and target data.

[0090] The consistency verification module 302 is used to extract key data information from the source end and the target end according to the time point, perform data consistency verification between the source end and the target end according to the key data information, and determine the target end data fragment to be compensated according to the verification result.

[0091] The data loading module 303 is used to shard the data of each target end, extract the source end data according to the verification result, and load the extracted source end data into the target end shadow table;

[0092] The data compensation module 304 is used to perform data compensation on the target end formal table based on the verification result and the target end shadow table.

[0093] According to one embodiment of the present invention, the consistency verification module 302 can also be used for:

[0094] After the data compensation is completed, the data consistency of the source and target official tables is checked again to verify whether the data compensation has passed.

[0095] If the data compensation fails, the target end formal table will be compensated again according to the aforementioned data compensation method until the data compensation is successful.

[0096] According to another embodiment of the present invention, the key data information includes the data shard to which the data belongs and the maximum timestamp of each data shard;

[0097] The consistency verification module 302 can also be used for:

[0098] For each source data shard included in the source end, if the maximum timestamp of the source data shard and the target data shard corresponding to the source data shard are consistent, then the data hash values ​​of the source data shard and the target data shard are compared. If the data hash values ​​are the same, the data consistency check passes.

[0099] If the maximum timestamp of the source data shard is greater than the maximum timestamp of the target data shard corresponding to the source data shard, then the data consistency check between the source data shard and the target data shard fails.

[0100] If the maximum timestamp of the source data shard is greater than the time point, then the time point for extracting the source data and target data is re-acquired, and the application corresponding to the source data is terminated from service acceptance.

[0101] According to another embodiment of the present invention, the time point determination module 301 can also be used for:

[0102] The maximum timestamp of the source data is used as the time point for extracting the source and target data.

[0103] The consistency verification module 302 can also be used for:

[0104] The maximum timestamp of the source data fragment is used as the time point for extracting the source and target data.

[0105] According to another embodiment of the present invention, the data compensation module 304 can also be used for:

[0106] Based on the verification result and the target end shadow table, the data to be compensated is determined from the target end shadow table. The data to be compensated corresponds to a business status, and the business status includes the final state of the business and non-final state of the business. The final state of the business is the final state of the business.

[0107] For data awaiting compensation that is in the final stage of the business process, update the target end formal table based on the data awaiting compensation.

[0108] For data to be compensated that is not in the final business state, the data compensation operation is performed on the target end formal table according to the business elements corresponding to the data to be compensated.

[0109] According to another embodiment of the present invention, the data compensation module 304 can also be used for:

[0110] If the data to be compensated does not exist in the target end official table when compensating for the target end data, then the data to be compensated is inserted into the target end official table.

[0111] If, during the compensation of data in the target-side official table, the data to be compensated exists in the target-side official table, but the business status of the data to be compensated in the target-side official table is different from the business status of the data to be compensated in the target-side shadow table, then the target-side shadow table is updated according to the data to be compensated in the target-side shadow table, and the data to be compensated in the target-side shadow table is marked as invalid.

[0112] According to another embodiment of the present invention, the data compensation module 304 can also be used for:

[0113] When performing data compensation on the target end's official table, stop the business processing of applications on both the source and target ends, and record the duration of the business processing suspension.

[0114] According to another embodiment of the present invention, the extraction of source data and the loading of the extracted source data into the target shadow table are performed concurrently.

[0115] According to the technical solution of this invention, data consistency processing is achieved by obtaining the time points for extracting source and target data; extracting key data information from the source and target sides based on these time points; performing data consistency verification between the source and target sides based on the key data information; determining the target data shards to be compensated based on the verification results; extracting source data from each target data shard based on the verification results; loading the extracted source data into the target shadow table; and performing data compensation operations on the target formal table based on the verification results and the target shadow table. This technical solution enables rapid, accurate, and efficient repair of the data consistency replication system without affecting existing data replication and while ensuring normal business processing of the target application as much as possible. By performing data consistency verification between the source and target formal tables after data compensation is completed to verify whether the data compensation has been successfully completed, the accuracy of the data consistency processing results can be guaranteed.

[0116] Figure 4 An exemplary system architecture 400 is shown, in which the data compensation method or data compensation apparatus of embodiments of the present invention can be applied.

[0117] like Figure 4 As shown, system architecture 400 may include terminal devices 401, 402, and 403, a network 404, and a server 405. Network 404 serves as the medium for providing communication links between terminal devices 401, 402, and 403 and server 405. Network 404 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0118] Users can use terminal devices 401, 402, and 403 to interact with server 405 via network 404 to receive or send messages, etc. Various client applications can be installed on terminal devices 401, 402, and 403, such as time monitoring applications, data comparison applications, database applications, data processing software, etc. (for example only).

[0119] Terminal devices 401, 402, and 403 can be various electronic devices with displays that support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0120] Server 405 can be a server providing various services, such as a backend management server supporting data compensation requests sent by users using terminal devices 401, 402, and 403 (for example only). The backend management server can acquire and extract the time points of source and target data from the received data compensation requests; extract key data information from the source and target sides based on the time points; perform data consistency verification between the source and target sides based on the key data information; and determine the target data fragment to be compensated based on the verification results. For each target data fragment, source data is extracted based on the verification results, and the extracted source data is loaded into the target shadow table. Based on the verification results and the target shadow table, data compensation and other processing are performed on the target formal table, and the processing results (e.g., data compensation results – for example only) are fed back to the terminal device.

[0121] It should be noted that the data compensation method provided in the embodiments of the present invention is generally executed by server 405, and correspondingly, the data compensation device is generally set in server 405.

[0122] It should be understood that Figure 4 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0123] The following is for reference. Figure 5 It shows a schematic diagram of the structure of a computer system 500 suitable for implementing terminal devices or servers of the present invention. Figure 5 The terminal device or server shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of the present invention.

[0124] like Figure 5As shown, the computer system 500 includes a central processing unit (CPU) 501, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 502 or programs loaded from storage section 508 into random access memory (RAM) 503. The RAM 503 also stores various programs and data required for the operation of the system 500. The CPU 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0125] The following components are connected to I / O interface 505: an input section 506 including a keyboard, mouse, etc.; an output section 507 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card, modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to I / O interface 505 as needed. A removable medium 511, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 510 as needed so that computer programs read from it can be installed into storage section 508 as needed.

[0126] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 509, and / or installed from removable medium 511. When the computer program is executed by central processing unit (CPU) 501, it performs the functions defined above in the system of this invention.

[0127] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0128] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0129] The units or modules described in the embodiments of the present invention can be implemented in software or hardware. The described units or modules can also be housed in a processor; for example, a processor can be described as including a time point determination module, a consistency verification module, a data loading module, and a data compensation module. The names of these units or modules do not necessarily limit the specific unit or module itself; for example, the time point determination module can also be described as "a module for obtaining the time points for extracting source and target data."

[0130] In another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs, which, when executed by the device, cause the device to include: acquiring time points for extracting source data and target data; extracting key data information from the source and target ends based on the time points; performing data consistency verification between the source and target ends based on the key data information; and determining the target data fragment to be compensated based on the verification result; extracting source data from each target data fragment based on the verification result; loading the extracted source data into a target shadow table; and performing data compensation on the target formal table based on the verification result and the target shadow table.

[0131] According to the technical solution of this invention, data consistency processing is achieved by obtaining the time points for extracting source and target data; extracting key data information from the source and target sides based on these time points; performing data consistency verification between the source and target sides based on the key data information; determining the target data shards to be compensated based on the verification results; extracting source data from each target data shard based on the verification results; loading the extracted source data into the target shadow table; and performing data compensation operations on the target formal table based on the verification results and the target shadow table. This technical solution enables rapid, accurate, and efficient repair of the data consistency replication system without affecting existing data replication and while ensuring normal business processing of the target application as much as possible. By performing data consistency verification between the source and target formal tables after data compensation is completed to verify whether the data compensation has been successfully completed, the accuracy of the data consistency processing results can be guaranteed.

[0132] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method of data compensation, characterized by, The method comprises: Step S101: obtaining a time point for extracting source end data and target end data; Step S102: extracting data key information of the source end and the target end respectively according to the time point, performing data consistency verification of the source end and the target end according to the data key information, and determining target end data fragments to be compensated according to a verification result; Step S103: for each target end data fragment, extracting source end data according to the verification result, and loading the extracted source end data into a target end shadow table; Step S104: performing data compensation on a target end formal table according to the verification result and the target end shadow table; Step S104 comprises: determining to-be-compensated data from the target end shadow table according to the verification result and the target end shadow table, the to-be-compensated data corresponding to a business state, and the business state comprising a business terminal state and a non-business terminal state, the business terminal state being a final state of a business; updating the target end formal table according to the to-be-compensated data for to-be-compensated data in the business terminal state; if, when performing data compensation on the target end formal table, the to-be-compensated data does not exist in the target end formal table, inserting the to-be-compensated data into the target end formal table for to-be-compensated data in the non-business terminal state; if, when performing data compensation on the target end formal table, the to-be-compensated data exists in the target end formal table but the business state of the to-be-compensated data in the target end formal table is different from the business state of the to-be-compensated data in the target end shadow table, updating the target end shadow table according to the to-be-compensated data in the target end shadow table, and marking the to-be-compensated data in the target end shadow table as invalid.

2. The method of claim 1, wherein, The method further comprises: After the data compensation is completed, data consistency verification of the source end and the target end formal table is performed again to verify whether the data compensation is passed; In the case that the data compensation is not passed, the target end formal table is compensated again according to steps S101 to S104 until the data compensation is passed.

3. The method of claim 1, wherein, The data key information comprises data fragments to which data belongs and a maximum timestamp of each data fragment; The data consistency verification of the source end and the target end according to the data key information comprises: For each source end data fragment included in the source end, if the maximum timestamp of the source end data fragment is consistent with that of a target end data fragment corresponding to the source end data fragment, data hash values of the source end data fragment and the target end data fragment are compared, and in the case that the data hash values are the same, the data consistency verification is passed; If the maximum timestamp of the source end data fragment is greater than that of the target end data fragment corresponding to the source end data fragment, the data consistency verification of the source end data fragment and the target end data fragment is not passed; If the maximum timestamp of the source end data fragment is greater than the time point, the time point for extracting the source end data and the target end data is obtained again, and a terminal business of an application corresponding to the source end data is terminated.

4. The method of claim 3, wherein, The time point for extracting the source end data and the target end data comprises: acquire a maximum timestamp of the source end data as a time point of extracting the source end data and the target end data; re-acquire the time point of extracting the source end data and the target end data comprises: acquire the maximum timestamp of the source end data slice as the time point of extracting the source end data and the target end data.

5. The method of claim 1, wherein, The method further comprises: stop the service acceptance of the application of the source end and the target end when compensating the data of the target end official table, and record the duration of stopping the service acceptance.

6. The method of claim 1, wherein, The source end data extraction and the loading of the extracted source end data to the target end shadow table are processed concurrently.

7. An apparatus for data compensation, the apparatus comprising: comprise: a time point determination module configured to acquire a time point of extracting source end data and target end data; a consistency check module configured to extract data key information of the source end and the target end respectively according to the time point, perform data consistency check of the source end and the target end according to the data key information, and determine target end data slices to be compensated according to a check result; a data loading module configured to, for each target end data slice, perform source end data extraction according to the check result, and load the extracted source end data to a target end shadow table; a data compensation module configured to compensate data of a target end official table according to the check result and the target end shadow table; The data compensation module is further configured to: determine to-be-compensated data from the target end shadow table according to the check result and the target end shadow table, the to-be-compensated data corresponding to a business state, and the business state comprising a business terminal state and a non-business terminal state, the business terminal state being a final state of the business; for the to-be-compensated data in the business terminal state, update the target end official table according to the to-be-compensated data; for the to-be-compensated data in the non-business terminal state, if the to-be-compensated data does not exist in the target end official table when compensating the data of the target end official table, insert the to-be-compensated data into the target end official table; if the to-be-compensated data exists in the target end official table when compensating the data of the target end official table, but the business state of the to-be-compensated data in the target end official table is different from the business state of the to-be-compensated data in the target end shadow table, update the target end shadow table according to the to-be-compensated data in the target end shadow table, and mark the to-be-compensated data in the target end shadow table as invalid.

8. An electronic device for data compensation, characterized by comprise: one or more processors; a storage device configured to store one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1-6.

9. A computer readable medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the method according to any one of claims 1-6. The program is executed by the processor to implement the method according to any one of claims 1-6.

Images