Data verification method and device in data synchronization, computer equipment, storage medium and program product
By applying the calibration table to slice and check processing functions, the problem of low data verification efficiency in data synchronization is solved, and fast and accurate data consistency verification is achieved.
Patent Information
- Application Number
- CN202411770294.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-04
- Publication Date
- 2025-02-28
AI Technical Summary
The prior art has low data verification efficiency during data synchronization, which leads to data inconsistency and may bring immeasurable losses and negative impacts.
By sharding the check table, data sharding information is obtained, and the verification processing function is run in the source database and the target database according to the storage type of the source database and the target database, and the target verification function is executed to obtain the verification result.
This method can flexibly adapt to various source databases and target databases for verification processing, speed up verification efficiency, ensure data consistency, and reduce potential losses.
Smart Images
Figure CN119537479A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of database technology, and in particular to a data verification method, device, computer equipment, computer-readable storage medium and computer program product in data synchronization. Background Art
[0002] Data is the digital life of every industry, and the database that carries data plays a core role in every field. Currently, there are many types of databases with different characteristics, and many different usage scenarios have brought about the need for database synchronization.
[0003] In actual applications, data synchronization programs or database systems may encounter various unforeseen problems, resulting in exceptions when synchronizing data to the peer database, which in turn leads to data inconsistency errors. Especially in scenarios where data accuracy is extremely high, such data inconsistency may bring immeasurable losses and negative impacts.
[0004] In order to ensure the consistency of synchronized data, it is necessary to verify the synchronized data of the two database systems and discover potential problems in a timely manner.
[0005] Currently, data verification mainly relies on the verification function of data synchronization tools, but there is a technical problem of low data verification efficiency. Summary of the invention
[0006] Based on this, it is necessary to provide a data verification method, device, computer equipment, computer-readable storage medium and computer program product in data synchronization to address the above technical problems.
[0007] In a first aspect, the present application provides a data verification method in data synchronization, comprising:
[0008] Slice the table to be checked to obtain data slicing information of the table to be checked;
[0009] If the source database and the target database of the table to be checked meet the check conditions in the database, the check processing function is respectively run in the source database and the target database according to the data sharding information; wherein the check processing function is used to splice the database data corresponding to the data sharding information according to the respective storage types of the source database and the target database to obtain the spliced data and then execute the target check function to obtain the check information; the check information of each of the source database and the target database is obtained to obtain the check result of the table to be checked;
[0010] If the source database and the target database of the table to be checked do not meet the verification conditions in the database, then according to the data sharding information and the storage type, the spliced data of the source database and the target database are respectively obtained, and the target verification function is executed according to the spliced data of the source database and the target database respectively to obtain the verification result of the table to be checked.
[0011] In one of the embodiments, before slicing the table to be checked and obtaining data slicing information of the table to be checked, the method further includes: determining a selected table; if the selected table has a unique identifier, determining the selected table as the table to be checked.
[0012] In one of the embodiments, after determining the selected table as the table to be checked if the selected table has a unique identifier, the method further includes: determining whether there is a delay in the synchronization task between the source database and the target database; if so, executing the step of sharding the table to be checked to obtain data sharding information of the table to be checked; if not, executing the step of determining whether there is a delay in the synchronization task between the source database and the target database again after a target time interval.
[0013] In one of the embodiments, sharding the table to be checked to obtain data sharding information of the table to be checked includes: sharding the table to be checked according to a unique identifier of the table to be checked to obtain data sharding information of the table to be checked; wherein the unique identifier is determined according to key information of the table to be checked; the key information includes a composite primary key, a composite unique key, a primary key of a string type, or a unique key of a string type; the data sharding information includes: a sharding key, an upper boundary, and a lower boundary of the sharding key.
[0014] In one of the embodiments, after sharding the table to be checked and obtaining data sharding information of the table to be checked, the method further includes: if the source database and the target database support in-database verification, obtaining the round-trip delay to the source database and the target database respectively; if the round-trip delay is greater than a delay threshold, obtaining the respective loads of the source database and the target database; if the respective loads of the source database and the target database are less than or equal to the load threshold, determining that the source database and the target database meet the database verification conditions.
[0015] In one of the embodiments, the obtaining of the spliced data of the source database and the target database respectively according to the data sharding information and the storage type includes: if the storage types of the source database and the target database are the same, splicing is performed in the source database and the target database respectively according to the data sharding information and the storage type to obtain the spliced data, and the spliced data is obtained from the source database and the target database respectively; if the storage types of the source database and the target database are different, querying the corresponding database data from the source database and the target database respectively according to the data sharding information, and splicing the respective database data according to the respective storage types of the source database and the target database to obtain the spliced data.
[0016] In a second aspect, the present application also provides a data verification device in data synchronization, including:
[0017] A fragmentation processing module, used for fragmenting the table to be verified to obtain data fragmentation information of the table to be verified;
[0018] A first verification processing module is used to run a verification processing function in the source database and the target database respectively according to the data sharding information if the source database and the target database of the table to be verified meet the verification conditions in the database; wherein the verification processing function is used to splice the database data corresponding to the data sharding information according to the respective storage types of the source database and the target database to obtain the spliced data and then execute the target verification function to obtain the verification information; obtain the respective verification information of the source database and the target database to obtain the verification result of the table to be verified;
[0019] The second verification processing module is used to obtain the spliced data of the source database and the target database respectively according to the data sharding information and the storage type if the source database and the target database of the table to be verified do not meet the verification conditions in the database, and execute the target verification function according to the spliced data of the source database and the target database respectively to obtain the verification result of the table to be verified.
[0020] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:
[0021] The table to be checked is sliced to obtain data slicing information of the table to be checked; if the source database and the target database of the table to be checked meet the verification conditions in the database, the verification processing function is respectively run in the source database and the target database according to the data slicing information; wherein the verification processing function is used to splice the database data corresponding to the data slicing information according to the respective storage types of the source database and the target database to obtain the spliced data and then execute the target verification function to obtain the verification information; the verification information of the source database and the target database is respectively obtained to obtain the verification result of the table to be checked; if the source database and the target database of the table to be checked do not meet the verification conditions in the database, the spliced data of the source database and the target database are respectively obtained according to the data slicing information and the storage type, and the target verification function is executed according to the spliced data of the source database and the target database to obtain the verification result of the table to be checked.
[0022] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:
[0023] The table to be checked is sliced to obtain data slicing information of the table to be checked; if the source database and the target database of the table to be checked meet the verification conditions in the database, the verification processing function is respectively run in the source database and the target database according to the data slicing information; wherein the verification processing function is used to splice the database data corresponding to the data slicing information according to the respective storage types of the source database and the target database to obtain the spliced data and then execute the target verification function to obtain the verification information; the verification information of the source database and the target database is respectively obtained to obtain the verification result of the table to be checked; if the source database and the target database of the table to be checked do not meet the verification conditions in the database, the spliced data of the source database and the target database are respectively obtained according to the data slicing information and the storage type, and the target verification function is executed according to the spliced data of the source database and the target database to obtain the verification result of the table to be checked.
[0024] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:
[0025] The table to be checked is sliced to obtain data slicing information of the table to be checked; if the source database and the target database of the table to be checked meet the verification conditions in the database, the verification processing function is respectively run in the source database and the target database according to the data slicing information; wherein the verification processing function is used to splice the database data corresponding to the data slicing information according to the respective storage types of the source database and the target database to obtain the spliced data and then execute the target verification function to obtain the verification information; the verification information of the source database and the target database is respectively obtained to obtain the verification result of the table to be checked; if the source database and the target database of the table to be checked do not meet the verification conditions in the database, the spliced data of the source database and the target database are respectively obtained according to the data slicing information and the storage type, and the target verification function is executed according to the spliced data of the source database and the target database to obtain the verification result of the table to be checked.
[0026] The data verification method, device, computer equipment, computer-readable storage medium and computer program product in the above-mentioned data synchronization, slice the table to be verified to obtain data slicing information of the table to be verified, if the source database and the target database of the table to be verified meet the verification conditions in the database, then run the verification processing function in the source database and the target database respectively according to the data slicing information, the verification processing function is used to splice the database data corresponding to the data slicing information according to the respective storage types of the source database and the target database to obtain the spliced data, and then execute the target verification function to obtain the verification information, obtain the respective verification information of the source database and the target database, and obtain the verification result of the table to be verified; if the source database and the target database of the table to be verified do not meet the verification conditions in the database, then obtain the spliced data of the source database and the target database respectively according to the data slicing information and the storage type, execute the target verification function according to the spliced data of the source database and the target database, and obtain the verification result of the table to be verified. After the table to be checked is sharded, this solution can take different verification processes according to whether the source database and target database of the table to be checked meet the verification conditions in the database. In the corresponding verification process, the respective storage types are combined to form spliced data according to the data sharding information and the target verification function is executed to obtain the verification information. Finally, the verification result can be obtained according to the respective verification information, so that it can flexibly adapt to various source databases and target databases for verification processing and speed up the verification efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.
[0028] FIG1 is a diagram showing an application environment of a data verification method in data synchronization according to an embodiment;
[0029] FIG2 is a schematic diagram of a flow chart of a data verification method in data synchronization according to an embodiment;
[0030] FIG3 is a schematic flow chart of a data verification method in data synchronization according to another embodiment;
[0031] FIG4 is a structural block diagram of a data verification device in data synchronization in one embodiment;
[0032] FIG. 5 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION
[0033] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0034] An introduction to the terms involved in the embodiments of this application;
[0035] Data synchronization: Migrate the structure and data of objects in one database system to another database system.
[0036] Source database, target database: During data synchronization, the end where data is migrated out is called the source database, and the end where data is migrated in is called the target database.
[0037] Data verification: Verify the database data of the source database and the target database to determine whether they are consistent.
[0038] MurMurHash3 algorithm: A non-cryptographic hash function with high speed, low collision rate and good statistical distribution characteristics.
[0039] The data verification method in data synchronization provided in the embodiment of the present application can be applied to the application environment shown in Figure 1. Among them, the terminal communicates with the server through the network. The data storage system can store the data that the server needs to process. The data storage system can be integrated on the server, or it can be placed on the cloud or other network servers. Among them, the terminal can be but is not limited to various personal computers, laptops, smart phones, tablets, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, projection devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The head-mounted device can be a virtual reality (VR) device, an augmented reality (AR) device, smart glasses, etc. The server can be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services.
[0040] In an exemplary embodiment, as shown in FIG. 2 , a data verification method in data synchronization is provided. The method may be executed by the server in FIG. 1 , and the method may include the following steps:
[0041] Step S201, sharding the table to be checked to obtain data sharding information of the table to be checked.
[0042] In this step, the table to be checked may be sharded to obtain data sharding information of the table to be checked, wherein the data sharding information may include a sharding key, an upper boundary and a lower boundary of the sharding key, etc., which may be used to indicate the shard to which the database data belongs.
[0043] Step S202: If the source database and the target database of the table to be verified meet the verification conditions in the database, the verification processing function is run in the source database and the target database respectively according to the data sharding information, and the verification information of the source database and the target database is obtained to obtain the verification result of the table to be verified.
[0044] The verification condition in the database refers to the condition used to determine whether it is appropriate to perform verification in the source database and the target database.
[0045] Wherein, the verification processing function can be run in the source database and the target database respectively. The verification processing function is used to splice the database data corresponding to the data sharding information to obtain spliced data according to the storage types of the source database and the target database, and then execute the target verification function on the spliced data to obtain verification information. Wherein, the storage type may include row storage, column storage, etc., splicing the database data corresponding to the data sharding information according to row storage can be splicing the database data corresponding to the data sharding information by row to obtain spliced data, and splicing the database data corresponding to the data sharding information according to column storage can be splicing the database data corresponding to the data sharding information by column to obtain spliced data. Wherein, the target verification function can adopt the MurMurHash3 function, and the MurMurHash3 function can adopt the MurMurHash3 algorithm to improve the verification efficiency. Wherein, after executing the target verification function on the respective spliced data in the source database and the target database respectively, the respective verification information can be obtained. Then, the verification information of the source database and the target database can be obtained, and the verification result of the table to be verified can be obtained according to the verification information of the source database and the target database, such as whether the verification of the table to be verified by the two databases is consistent.
[0046] Step S203, if the source database and the target database of the table to be checked do not meet the verification conditions in the database, then according to the data sharding information and the storage type, the spliced data of the source database and the target database are respectively obtained, and the target verification function is executed according to the spliced data of the source database and the target database respectively to obtain the verification result of the table to be checked.
[0047] Among them, if the source database and the target database of the table to be verified do not meet the verification conditions in the database, it means that it is not suitable to perform relevant data verification processing in the source database and the target database. In this step, if the source database and the target database of the table to be verified do not meet the verification conditions in the database, the spliced data of the source database and the target database can be obtained respectively according to the data sharding information and the storage type, and the target verification function (such as MurMurHash3 function) is executed outside the database according to the spliced data of the source database and the target database respectively, so that the verification information of the source database and the target database can be obtained respectively, and the verification result of the table to be verified can be obtained according to the verification information of the source database and the target database respectively.
[0048] The data verification method in the data synchronization of the present embodiment is to slice the table to be verified to obtain data slicing information of the table to be verified. If the source database and the target database of the table to be verified meet the verification conditions in the database, the verification processing function is respectively run in the source database and the target database according to the data slicing information. The verification processing function is used to splice the database data corresponding to the data slicing information according to the respective storage types of the source database and the target database to obtain the spliced data, and then execute the target verification function to obtain the verification information, obtain the respective verification information of the source database and the target database, and obtain the verification result of the table to be verified; if the source database and the target database of the table to be verified do not meet the verification conditions in the database, then according to the data slicing information and the storage type, the spliced data of the source database and the target database are respectively obtained, and the target verification function is executed according to the spliced data of the source database and the target database to obtain the verification result of the table to be verified. After the table to be checked is sharded, this solution can take different verification processes according to whether the source database and target database of the table to be checked meet the verification conditions in the database. In the corresponding verification process, the respective storage types are combined to form spliced data according to the data sharding information and the target verification function is executed to obtain the verification information. Finally, the verification result can be obtained according to the respective verification information, so that it can flexibly adapt to various source databases and target databases for verification processing and speed up the verification efficiency.
[0049] In an exemplary embodiment, before slicing the table to be checked and obtaining data slicing information of the table to be checked in step S201, the method of the present application may further include:
[0050] Determine the selected table; if the selected table has a unique identifier, the selected table is determined as the table to be verified.
[0051] In this embodiment, a data table in the database can be selected as the selected table, and whether the selected table has a unique identifier can be checked. If the selected table does not have a unique identifier, no verification process can be performed on it. If the selected table has a unique identifier, the selected table can be determined as the table to be verified. The unique identifier can be, for example, a primary key or a unique key.
[0052] In an exemplary embodiment, further, after determining the selected table as the table to be checked if the selected table has a unique identifier, the following may also be included:
[0053] Determine whether there is a delay in the synchronization task between the source database and the target database; if so, execute the step of sharding the table to be checked to obtain data sharding information of the table to be checked; if not, execute the step of determining whether there is a delay in the synchronization task between the source database and the target database after the target time interval.
[0054] In this embodiment, after determining the table to be checked, it is possible to first determine whether there is a delay in the synchronization task between the source database and the target database of the table to be checked, that is, whether all the incremental data generated by the source database has been synchronized to the target database. If so, the next step can be continued, that is, the step of slicing the table to be checked to obtain the data slicing information of the table to be checked (step S201); if not, the above-mentioned step of determining whether there is a delay in the synchronization task between the source database and the target database can be performed again after the target time interval, for example, the above-mentioned step of determining whether there is a delay in the synchronization task between the source database and the target database can be performed after randomly waiting for a suitable period of time (target time interval). In practical applications, if the condition of synchronization without delay is still not met after, for example, three times of repetition, an alarm of synchronization delay can be initiated, and step S201 can be executed. In this way, the accuracy of data verification can be improved as much as possible.
[0055] In an exemplary embodiment, slicing the table to be checked in step S201 to obtain data slicing information of the table to be checked may include:
[0056] The table to be checked is fragmented according to the unique identifier of the table to be checked to obtain data fragmentation information of the table to be checked.
[0057] In this embodiment, the table to be checked can be sharded according to the unique identifier of the table to be checked, so as to obtain data sharding information of the table to be checked. The unique identifier can be determined according to the key information of the table to be checked. The key information can include a composite primary key, a composite unique key, a primary key of a string type, or a unique key of a string type, etc. As an example, if it is a composite primary key or a composite unique key, the first key can be selected as the unique identifier to shard the table to be checked; if it is a primary key of a string type or a unique key of a string type, the unique identifier can be obtained by converting characters into ASCII codes or by using a built-in function of the database to shard the table to be checked. The data sharding information thus obtained may include a sharding key, an upper boundary and a lower boundary of the sharding key.
[0058] In an exemplary embodiment, after the table to be checked is fragmented in step S201 to obtain data fragmentation information of the table to be checked, the method of the present application may further include:
[0059] If the source database and the target database support in-database verification, the round-trip delay to the source database and the target database is obtained respectively; if the round-trip delay is greater than the delay threshold, the load of the source database and the target database is obtained respectively; if the load of the source database and the target database is less than or equal to the load threshold respectively, it is determined that the source database and the target database meet the in-database verification conditions.
[0060] In this embodiment, after the table to be checked is segmented and data segmentation information of the table to be checked is obtained, it is determined whether the source database and the target database meet the verification conditions in the database.
[0061] It can be determined whether the source database and the target database support in-database verification. If the source database or the target database does not support in-database verification, it can be determined that the source database and the target database do not meet the in-database verification conditions. If the source database and the target database support in-database verification, the round-trip delay from the server to the source database and the target database can be obtained. The round-trip delay can be tested by executing the select 1 statement. In order to avoid the influence of other factors, the statement for testing the round-trip delay can use a simple query statement and does not retrieve data from the table.
[0062] Wherein, when the round-trip delay is less than or equal to the delay threshold, it can be determined that the source database and the target database do not meet the database verification condition. If the round-trip delay is greater than the delay threshold, the load of the source database and the target database can be obtained, and the load can include the usage of CPU and memory.
[0063] Wherein, if the load is greater than the load threshold, it can be determined that the source database and the target database do not meet the database verification condition. If the load of the source database and the target database is less than or equal to the load threshold, it can be determined that the source database and the target database meet the database verification condition.
[0064] Therefore, verification can be performed in the source database and the target database when they support database verification and the round-trip delay is greater than the set delay threshold and the load is within the set load threshold.
[0065] In an exemplary embodiment, obtaining the spliced data of the source database and the target database respectively according to the data sharding information and the storage type in step S203 may include:
[0066] If the storage types of the source database and the target database are the same, the spliced data are obtained by splicing them in the source database and the target database according to the data sharding information and the storage type, and the spliced data are obtained from the source database and the target database respectively; if the storage types of the source database and the target database are different, the corresponding database data are queried from the source database and the target database respectively according to the data sharding information, and according to the respective storage types of the source database and the target database, the respective database data are spliced to obtain the spliced data.
[0067] In the case that the source database and the target database do not meet the verification conditions in the database, the data needs to be read out first and then the verification process is performed. In this embodiment, the processing is divided into two cases: the storage types of the source database and the target database are the same and different.
[0068] If the storage type of the source database and the target database is the same, for example, both are row-based storage or column-based storage, then the splicing data is obtained in the source database and the target database according to the data sharding information and storage type, and then the respective splicing data is obtained from the source database and the target database. For example, if both the source database and the target database are row-based storage, then the source database and the target database respectively splice the corresponding database data by row according to the data sharding information to obtain the splicing data and then return the splicing data; if both are column-based storage, then the source database and the target database respectively splice the corresponding database data by column according to the data sharding information to obtain the splicing data and then return the splicing data. After obtaining the returned splicing data, no deserialization processing is performed, and the target verification function can be executed on it to reduce the overall verification time.
[0069] If the storage types of the source database and the target database are different, for example, one is row-based storage and the other is column-based storage, then it is necessary to query the corresponding database data from the source database and the target database respectively according to the data sharding information, and then splice the respective database data according to the storage types of the source database and the target database to obtain the spliced data, and then execute the target verification function on it.
[0070] In one embodiment, as shown in FIG3 , a data verification method in data synchronization is also provided, and the method may include the following steps:
[0071] ① You can configure the tables and columns for data comparison, the size of the data block for comparison, the proportion of data spot checks, and whether to allow verification calculations in the source and target databases. If the above conditions allow, you can further configure thresholds such as round-trip delay and database load. If the set thresholds are not met, verification calculations can be performed in the source and target databases. Start the data comparison task.
[0072] ② Check whether the selected table has a unique identifier, such as a primary key or a unique key. Data verification supports tables with unique identifiers.
[0073] ③ Determine whether the data synchronization delay is zero, that is, whether the incremental data generated by the source database has been fully synchronized to the target database. If yes, proceed to the next step; if not, wait for a suitable period of time at random and continue this step ③. If the synchronization delay condition is still not met after three repetitions, an alarm of synchronization delay will be given, and then proceed to the next step.
[0074] ④ Select a table to be verified and then shard it. If it is a composite primary key or composite unique key, you can select the first key. If the primary key type or unique key type is a string, you can shard it by converting the characters into ASCII codes or using the built-in functions of the database.
[0075] ⑤ Obtain the field mapping rules of the source database and the target database. There are differences in type or precision in the synchronization of heterogeneous databases. Therefore, when querying, you can obtain the corresponding data according to the mapping rules. Determine whether the source database or the target database supports calculating the verification results in the database. If it does, you can proceed to step ⑥. If not, you can proceed to step ⑧.
[0076] ⑥ Obtain the round-trip delay from the server (with a verification program) to the source database and the target database. For example, if the source database is MySQL, you can execute the select 1 statement for testing. To avoid the influence of other factors, the test statement should be a simple query statement and will not retrieve data from the table. Then, you can obtain the load of the source database and the target database, including the usage of CPU and memory.
[0077] ⑦ If verification calculations are supported in the source database and the target database, and the round-trip delay is greater than the set delay threshold and the load is within the set load threshold, then the verification process can be performed on the source database and the target database. In order to avoid differences due to databases, a MurMurHash3 function for calculation can be created in the source database and the target database. You can also create a compute function that can be used to calculate the hash value of a shard. The compute parameters can include the shard key, the upper boundary and the lower boundary of the shard key. If the table is stored in row format, the data of this shard is spliced by row, and then the hash calculation is performed; if it is stored in column format, the shard data is spliced by column, and then the hash calculation is performed.
[0078] As an example, if the storage is row-based, such as the database is MySQL, the shard key is id, the upper boundary is 2000, and the lower boundary is 1, then the content of the function can be:
[0079] select coalesce(LOWER(CONV(BIT_XOR(cast(
[0080] MurMurHash3(CONCAT_WS('#', `id`, `birth_date`, convert(`name` usingutf8mb4))) as unsigned)), 10, 16)), 0) as result from `test`.`t1` force index(`PRIMARY`) where `id` >= 1 and `id` <= 2000
[0081] ⑧Concurrently perform data verification. If the verification is in the database, you can obtain the verification results of the data shards of the source database and the target database by calling the function in step ⑦ and compare them. If verification in the database is not supported, first read the data of the shard, and then calculate it in the verification program of the server. The verification program can also use the MurMurHash3 algorithm for calculation. Among them, if the source database and the target database are both row-stored tables, the data of this shard is spliced by row and then the result is returned; if the source database and the target database are both column-stored, the data of the shard is spliced by column and then the result is returned. After the result is returned, no deserialization processing is performed, and the binary array is directly calculated to reduce the overall verification time. If one of the source database and the target database belongs to row storage and the other belongs to column storage, it is necessary to query the data of the shard, splice it in the verification program, and then calculate it after splicing. Finally, compare the verification information of the source database and the target database to determine whether they are consistent.
[0082] ⑨When inconsistencies are found, the corresponding shard data of the source database and the target database can be queried, and then the data of the source database and the target database can be verified row by row, and the inconsistent columns can be recorded.
[0083] ⑩ Repeat step ⑧ until all shards have completed data verification.
[0084] Repeat step ④ until all tables have completed data verification.
[0085] The solution of this embodiment can adopt the MurMurHash3 algorithm for calculation according to the characteristics that the source database and the target database are highly consistent and do not need encryption. It has the advantages of short time and high performance. It can also select the optimal computing carrier according to the round-trip delay, database load and other conditions, greatly shortening the overall verification time. It can also adopt different query methods according to the data storage characteristics of the source database and the target database to speed up the verification rate. It also supports users to configure the ratio of data spot check comparison. In the case of large data volume, configuring a more appropriate spot check ratio can reduce the pressure on the source and target databases. It also supports heterogeneous database verification and can support Oracle, SQL Server, PostgreSQL, and MySql relational databases.
[0086] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.
[0087] Based on the same inventive concept, the embodiment of the present application also provides a data verification device in data synchronization for implementing the data verification method in data synchronization involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the embodiments of one or more data verification devices in data synchronization provided below can refer to the limitations of the data verification method in data synchronization above, and will not be repeated here.
[0088] In an exemplary embodiment, as shown in FIG4 , a data verification device in data synchronization is provided. The device 400 may include:
[0089] The fragmentation processing module 401 is used to fragment the table to be verified and obtain data fragmentation information of the table to be verified;
[0090] The first verification processing module 402 is used to run the verification processing function in the source database and the target database respectively according to the data sharding information if the source database and the target database of the table to be verified meet the verification conditions in the database; wherein the verification processing function is used to splice the database data corresponding to the data sharding information according to the storage types of the source database and the target database to obtain the spliced data and then execute the target verification function to obtain the verification information; obtain the verification information of the source database and the target database respectively, and obtain the verification result of the table to be verified;
[0091] The second verification processing module 403 is used to obtain the spliced data of the source database and the target database respectively according to the data sharding information and the storage type if the source database and the target database of the table to be verified do not meet the verification conditions in the database, and execute the target verification function according to the spliced data of the source database and the target database respectively to obtain the verification result of the table to be verified.
[0092] In one embodiment, the shard processing module 401 is further used to determine a selected table; if the selected table has a unique identifier, the selected table is determined as the table to be checked.
[0093] In one embodiment, the sharding processing module 401 is also used to determine whether there is a delay in the synchronization task between the source database and the target database; if so, the step of sharding the table to be checked to obtain data sharding information of the table to be checked is executed; if not, the step of determining whether there is a delay in the synchronization task between the source database and the target database is executed again after the target time interval.
[0094] In one embodiment, the sharding processing module 401 is used to shard the table to be checked according to the unique identifier of the table to be checked to obtain data sharding information of the table to be checked; wherein the unique identifier is determined according to the key information of the table to be checked; the key information includes a composite primary key, a composite unique key, a primary key of a string type or a unique key of a string type; the data sharding information includes: a sharding key, an upper boundary and a lower boundary of the sharding key.
[0095] In one embodiment, the device 400 may also include: a condition judgment module, which is used to obtain the round-trip delay to the source database and the target database respectively if the source database and the target database support in-database verification; if the round-trip delay is greater than the delay threshold, obtain the respective loads of the source database and the target database; if the respective loads of the source database and the target database are less than or equal to the load threshold, determine that the source database and the target database meet the in-database verification condition.
[0096] In one embodiment, the second verification processing module 403 is used to, if the storage types of the source database and the target database are the same, perform splicing in the source database and the target database according to the data sharding information and the storage type to obtain the spliced data, and obtain the spliced data from the source database and the target database respectively; if the storage types of the source database and the target database are different, query the corresponding database data from the source database and the target database respectively according to the data sharding information, and splice the respective database data according to the respective storage types of the source database and the target database to obtain the spliced data.
[0097] Each module in the data verification device in the above data synchronization can be implemented in whole or in part by software, hardware and a combination thereof. Each module can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to each module above.
[0098] In an exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be shown in FIG5. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data such as verification results. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a data verification method in data synchronization is implemented.
[0099] Those skilled in the art will understand that the structure shown in FIG. 5 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.
[0100] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.
[0101] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.
[0102] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.
[0103] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.
[0104] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.
[0105] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.
[0106] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.
Claims
1. A data verification method in data synchronization, characterized in that: The method comprises: Slice the table to be checked to obtain data slicing information of the table to be checked; If the source database and the target database of the table to be checked meet the check conditions in the database, the check processing function is respectively run in the source database and the target database according to the data sharding information; wherein the check processing function is used to splice the database data corresponding to the data sharding information according to the respective storage types of the source database and the target database to obtain the spliced data and then execute the target check function to obtain the check information; the check information of each of the source database and the target database is obtained to obtain the check result of the table to be checked; If the source database and the target database of the table to be checked do not meet the verification conditions in the database, then according to the data sharding information and the storage type, the spliced data of the source database and the target database are respectively obtained, and the target verification function is executed according to the spliced data of the source database and the target database respectively to obtain the verification result of the table to be checked.
2. The method according to claim 1, characterized in that Before slicing the table to be checked and obtaining data slicing information of the table to be checked, the method further includes: Determine the selected table; If the selected table has a unique identifier, the selected table is determined as the table to be checked.
3. The method according to claim 2, characterized in that After determining the selected table as the table to be checked if the selected table has a unique identifier, the method further includes: Determine whether there is a delay in the synchronization task between the source database and the target database; If yes, then execute the step of slicing the table to be checked to obtain data slicing information of the table to be checked; If not, the step of determining whether there is a delay in the synchronization task between the source database and the target database is performed again after the target time interval.
4. The method according to claim 1, characterized in that: The slicing of the table to be checked to obtain data slicing information of the table to be checked includes: The table to be checked is sharded according to the unique identifier of the table to be checked to obtain data sharding information of the table to be checked; wherein the unique identifier is determined according to the key information of the table to be checked; the key information includes a composite primary key, a composite unique key, a primary key of a string type or a unique key of a string type; the data sharding information includes: a sharding key, an upper boundary and a lower boundary of the sharding key.
5. The method according to claim 1, characterized in that After the table to be checked is fragmented to obtain data fragmentation information of the table to be checked, the method further includes: If the source database and the target database support in-database verification, obtaining round-trip delays to the source database and the target database respectively; If the round-trip delay is greater than the delay threshold, obtaining the load of the source database and the target database respectively; If the loads of the source database and the target database are respectively less than or equal to the load threshold, it is determined that the source database and the target database meet the database verification condition.
6. The method according to any one of claims 1 to 5, characterized in that The acquiring the spliced data of the source database and the target database respectively according to the data sharding information and the storage type includes: If the storage types of the source database and the target database are the same, performing splicing in the source database and the target database according to the data sharding information and the storage type to obtain the spliced data, respectively, and obtaining the spliced data from the source database and the target database; If the storage types of the source database and the target database are different, the corresponding database data are queried from the source database and the target database respectively according to the data sharding information, and the respective database data are spliced according to the respective storage types of the source database and the target database to obtain the spliced data.
7. A data verification device in data synchronization, characterized in that: The device comprises: A fragmentation processing module, used for fragmenting the table to be verified to obtain data fragmentation information of the table to be verified; A first verification processing module is used to run a verification processing function in the source database and the target database respectively according to the data sharding information if the source database and the target database of the table to be verified meet the verification conditions in the database; wherein the verification processing function is used to splice the database data corresponding to the data sharding information according to the respective storage types of the source database and the target database to obtain the spliced data and then execute the target verification function to obtain the verification information; obtain the respective verification information of the source database and the target database to obtain the verification result of the table to be verified; The second verification processing module is used to obtain the spliced data of the source database and the target database respectively according to the data sharding information and the storage type if the source database and the target database of the table to be verified do not meet the verification conditions in the database, and execute the target verification function according to the spliced data of the source database and the target database respectively to obtain the verification result of the table to be verified.
8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.
10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.