Data synchronization verification method, device, computing device and storage medium

Abstract

The present invention discloses a data synchronization verification method, device, computing equipment and storage medium. According to the technical solution provided by the present invention, in the process of synchronizing source data from a source database to a target database, a source object and a target object corresponding to the source data are generated, and a source hash mapping table is constructed according to the source object, and a cuckoo filter set is constructed according to the target object; wherein the fingerprint data of the target object is stored in the cuckoo filter set; the source objects in the source hash mapping table are traversed concurrently, and for each source object in the source hash mapping table, the fingerprint data of the source object is searched in the cuckoo filter set, and the data synchronization verification result is determined according to the search result. The present invention can quickly and accurately perform data migration and verification and greatly ensure data consistency by using fingerprint data to complete the verification of all data in data migration after data synchronization.

Description

Technical Field

[0001] The present invention relates to the fields of data processing and data storage, and in particular to a data synchronization verification method, device, computing equipment and computer storage medium. Background Art

[0002] With the development of the digital age, data, as a vital resource, is playing an increasingly important role in our lives. Various data processing technologies based on data are also constantly developing. Consequently, higher requirements are being placed on data migration technologies to meet the current implementation and application of various data-based technologies.

[0003] Existing technologies focus on automating the data migration process and improving data migration throughput, but pay little attention to ensuring data consistency and integrity. This makes it impossible to guarantee data consistency and integrity after migration is complete, and there is a lack of efficient data verification technology that guarantees 100% coverage of the entire data. Instead, they often simply manually randomly sample the original and target data for comparison, using this as the final indicator for data consistency verification. This method fails to achieve comprehensive data verification, resulting in low data sample coverage, excessive randomness in verification, and inability to implement subsequent data consistency remediation measures. Summary of the Invention

[0004] In view of the above problems, the present invention is proposed to provide a data synchronization verification method, a corresponding data synchronization verification device, a computing device and a computer storage medium that overcome the above problems or at least partially solve the above problems.

[0005] According to one aspect of the present invention, a data synchronization verification method is provided, the method comprising:

[0006] In the process of synchronizing source data from a source database to a target database, a source object and a target object corresponding to the source data are generated, a source hash map is constructed based on the source object, and a cuckoo filter set is constructed based on the target object; wherein the cuckoo filter set stores the fingerprint data of the target object;

[0007] Concurrently traverse the source objects in the source hash mapping table, search for fingerprint data of each source object in the source hash mapping table in the cuckoo filter set, and determine a data synchronization verification result according to the search result.

[0008] In the above solution, in the process of synchronizing source data from the source database to the target database, generating source objects and target objects corresponding to the source data, constructing a source hash map based on the source objects, and constructing a cuckoo filter set based on the target objects further include:

[0009] Read source data from a source database, perform data conversion on the source data, and generate source objects and target objects corresponding to the source data;

[0010] Add the source object to the source message queue, consume the source message queue to build a source hash map;

[0011] Add the target object to a target message queue, write the target object to a target database, and consume the target message queue to construct a cuckoo filter set.

[0012] In the above solution, consuming the source message queue to construct a source hash map further includes:

[0013] Consume the source object in the source message queue, serialize and store the source object, and set a hash algorithm;

[0014] The linked list and red-black tree models are used to process the serialized stored source objects and construct the source hash map.

[0015] In the above solution, consuming the target message queue to construct a cuckoo filter set further includes:

[0016] The target object in the target message queue is consumed, data with a preset number of bits in the hash value of the target object is used as fingerprint data of the target object, and the fingerprint data is written into the cuckoo filter set.

[0017] In the above solution, searching for fingerprint data of each source object in the source hash map in the cuckoo filter set further includes:

[0018] Calculating fingerprint data of the source object;

[0019] Hash the source object using a first hash algorithm to obtain a first bucket address;

[0020] Performing hash processing on the fingerprint data of the source object using a second hash algorithm to obtain a fingerprint hash value;

[0021] Performing an XOR operation on the first bucket address and the fingerprint hash value to obtain a second bucket address;

[0022] In the cuckoo filter set, it is searched whether the bucket corresponding to the first bucket address or the bucket corresponding to the second bucket address stores the fingerprint data of the source object.

[0023] In the above solution, determining the data synchronization verification result according to the search result further includes:

[0024] If the fingerprint data of the source object is found, determining that the data synchronization verification result of the source data corresponding to the source object is normal, and continuing to traverse other source objects in the source hash map;

[0025] If the fingerprint data of the source object is not found, it is determined that the data synchronization verification result of the source data corresponding to the source object is an abnormal result, and the source data corresponding to the source object is stored as abnormal data.

[0026] In the above solution, the method further includes:

[0027] Process abnormal data according to the preset abnormality handling plan.

[0028] According to another aspect of the present invention, a data synchronization verification device is provided, comprising: a synchronization module and a verification module; wherein,

[0029] The synchronization module is configured to generate a source object and a target object corresponding to the source data during synchronization of the source data from the source database to the target database, and to construct a source hash map based on the source object and a cuckoo filter set based on the target object; wherein the cuckoo filter set stores the fingerprint data of the target object;

[0030] The verification module is used to concurrently traverse the source objects in the source hash mapping table, search for fingerprint data of each source object in the source hash mapping table in the cuckoo filter set, and determine a data synchronization verification result according to the search result.

[0031] According to another aspect of the present invention, there is provided a computing device, comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus;

[0032] The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the above-mentioned data synchronization verification method.

[0033] According to another aspect of the present invention, a computer storage medium is provided, wherein the storage medium stores at least one executable instruction, and the executable instruction enables a processor to execute operations corresponding to the above-mentioned data synchronization verification method.

[0034] According to the technical solution provided by the present invention, during the process of synchronizing source data from a source database to a target database, source objects and target objects corresponding to the source data are generated, and a source hash map is constructed based on the source objects, and a cuckoo filter set is constructed based on the target objects; wherein the cuckoo filter set stores the fingerprint data of the target objects; the source objects in the source hash map are traversed concurrently, and for each source object in the source hash map, the fingerprint data of the source object is searched in the cuckoo filter set, and the data synchronization verification result is determined based on the search result. This solves the problem in the prior art of simply pursuing data migration throughput, failing to achieve comprehensive data verification, having too low data sample coverage, too high verification randomness, and failing to carry out subsequent data consistency remediation measures. The technical solution provided by the present invention quickly achieves synchronization verification of all data by constructing a source hash map during the data synchronization process and searching the cuckoo filter set based on the fingerprint data of all source data, greatly improving the accuracy of verification and ensuring the consistency and integrity before and after data migration.

[0035] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are specifically listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Various other advantages and benefits will become apparent to those skilled in the art upon reading the detailed description of the preferred embodiment below. The accompanying drawings are for illustration purposes only and are not to be considered as limiting the present invention. The same reference symbols are used throughout the drawings to represent the same components. In the drawings:

[0037] Figure 1 A schematic diagram of a process flow of a data synchronization verification method according to an embodiment of the present invention is shown;

[0038] Figure 2 A schematic diagram of a data synchronization method according to an embodiment of the present invention is shown;

[0039] Figure 3 A schematic diagram of the source message queue structure according to an embodiment of the present invention is shown;

[0040] Figure 4A A schematic diagram of a chain table according to an embodiment of the present invention is shown;

[0041] Figure 4B A schematic diagram of the red-black tree model structure according to an embodiment of the present invention is shown;

[0042] Figure 5 A schematic diagram of a data comparison method according to an embodiment of the present invention is shown;

[0043] Figure 6 A schematic diagram of a data comparison process according to another embodiment of the present invention is shown;

[0044] Figure 7 A structural block diagram of a data synchronization verification device according to an embodiment of the present invention is shown;

[0045] Figure 8 A schematic structural diagram of a computing device according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0046] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0047] Figure 1 FIG. 1 shows a flow chart of a data synchronization verification method according to an embodiment of the present invention. Figure 1 As shown, the method includes the following steps:

[0048] Step S101, in the process of synchronizing source data from a source database to a target database, generates a source object and a target object corresponding to the source data, constructs a source hash map based on the source object, and constructs a cuckoo filter set based on the target object; wherein the cuckoo filter set stores the fingerprint data of the target object.

[0049] Step S102 , concurrently traversing the source objects in the source hash mapping table, searching for fingerprint data of each source object in the source hash mapping table in the cuckoo filter set, and determining a data synchronization verification result according to the search result.

[0050] According to a data synchronization verification method provided by this embodiment, in the process of synchronizing source data from a source database to a target database, a source object and a target object corresponding to the source data are generated, and a source hash mapping table is constructed based on the source object, and a cuckoo filter set is constructed based on the target object; wherein the fingerprint data of the target object is stored in the cuckoo filter set; the source objects in the source hash mapping table are traversed concurrently, and for each source object in the source hash mapping table, the fingerprint data of the source object is searched in the cuckoo filter set, and the data synchronization verification result is determined based on the search result. Utilizing the technical solution provided by the present invention, a source hash mapping table can be constructed during the data synchronization process, and a search can be performed in the cuckoo filter set based on the fingerprint data of all source data, thereby quickly achieving synchronization verification of all data, greatly improving the accuracy of verification, and ensuring consistency and integrity before and after data migration.

[0051] Figure 2 FIG. 1 shows a flow chart of a data synchronization method according to an embodiment of the present invention. Figure 2 As shown, where:

[0052] Step S201: reading source data from a source database, performing data conversion processing on the source data, and generating source objects and target objects corresponding to the source data.

[0053] Specifically, a thread is started to read source data in the source database in batches, and the number of source data items read at a time is determined according to a preset reading quantity.

[0054] Preferably, when reading data, the data is read in ascending order of the sequence ID of the primary key, and the preset number of reads may be 1000 or the like.

[0055] Specifically, a data converter is used to perform field type conversion to generate source objects and target objects corresponding to the source data; and source database records are converted into target database records.

[0056] Step S202: adding the source object to a source message queue, and consuming the source message queue to construct a source hash map.

[0057] Specifically, the source objects in the source message queue are consumed, the source objects are serialized and stored, and a hash algorithm is set; the serialized and stored source objects are processed using a linked list and a red-black tree model to construct a source hash mapping table.

[0058] Preferably, the source object is encapsulated as a message entity and delivered to the source message queue, wherein the source message queue can adopt the Kafka framework (Kafka distributed publish-subscribe message system), construct a topic (Topic) in it, and configure N partitions (Partition) for the topic to consume the source object in the source message queue. The structural diagram of the source message queue is as follows Figure 3 shown.

[0059] Figure 3 FIG. 4 shows a schematic diagram of a source message queue structure according to an embodiment of the present invention. Figure 3 As shown in the figure, the Kafka cluster contains multiple servers, namely Server 1 and Server 2. Server 1 further contains multiple partitions, namely P0 and P3; Server 2 further contains multiple partitions, namely P1 and P2. Different partitions correspond to different user groups, and users in the corresponding user groups consume messages in the partitions. Partitions and user groups are flexibly mapped, such as Figure 3 As shown in .

[0060] Preferably, the hash algorithm may be configured to use key fields or dynamically configured fields to participate in the calculation, thereby adding a strong random number UUID (Universally Unique Identifier) ​​factor to ensure that the HASH code encoding method does not conflict.

[0061] Preferably, a linked list and a red-black tree model are used to process the source objects after serialized storage, and a concurrent source hash map is constructed and stored in a memory cache. When the length of the linked list exceeds a preset threshold, the linked list is converted into a red-black tree model. The source hash map is based on a CAS (Compare and Swap) operation to obtain and store the source object, and a synchronization lock keyword is used to lock the corresponding data segment to improve the concurrency of the entire process, thereby improving the efficiency of acquisition and storage. The preset threshold of the linked list length can be 8, and the addressing time complexity of the linked list is 0(N); after being converted into a red-black tree, the addressing time complexity of the red-black tree is 0(long(N)).

[0062] Wherein, the linked list is as follows Figure 4A shown. Figure 4A A schematic diagram of a chain table according to one embodiment of the present invention is shown.

[0063] Each node in the linked list has a hash bucket with a serial number. Each bucket can be used as a data storage location to store multiple data items. The key value can be used to determine the data storage location (variable slot) according to the hash algorithm, improving addressing performance.

[0064] Figure 4A There are 7 hash buckets in the hash table, and the hash algorithm used for addressing is slot = key % 7. For example, the address corresponding to the value 21 is 21 % 7 = 0, that is, the storage location of the value 21 is bucket position 0.

[0065] The red-black tree is as follows Figure 4B shown. Figure 4B A schematic diagram of the red-black tree model structure according to an embodiment of the present invention is shown.

[0066] The red-black tree has the following properties:

[0067] 1. Nodes are either red or black. 2. The root node is black. 3. All leaves are black. (Leaves are empty nodes at the end.) 4. Both child nodes of each red node are black. (There cannot be two consecutive red nodes on all paths from each leaf to the root. That is, if a node is red, then its child nodes are either Null or black (that is, there cannot be two adjacent red nodes. That is, the parent, left child, and right child of a red node can only be black nodes). 5. All paths from any node to each of its leaves contain the same number of black nodes.

[0068] According to the above properties, the shortest possible path consists of all black nodes, the longest possible path has alternating red and black nodes, and all longest paths have the same number of black nodes. Therefore, no path can be more than twice as long as any other path, effectively improving addressing efficiency.

[0069] Step S203: adding the target object to a target message queue, writing the target object into a target database, and consuming the target message queue to construct a cuckoo filter set.

[0070] Specifically, the target object in the target message queue is consumed, data with a preset number of bits in the hash value of the target object is used as fingerprint data of the target object, and the fingerprint data is written into the cuckoo filter set.

[0071] Preferably, the preset number of bits may be 8, and the 8-bit data in the target object is used as its fingerprint data, thereby improving the space storage efficiency of the cuckoo filter set.

[0072] Specifically, based on the generated target object, the target object is encapsulated as a message entity and delivered to the target message queue, wherein the target message queue can adopt the Kafka framework, build a topic in it, and configure N partitions for the topic to consume the target object in the target message queue.

[0073] After all the target objects are added to the target message queue and their corresponding fingerprint data are written into the cuckoo filter set, a data comparison process will be performed. Figure 5 FIG. 1 shows a flow chart of a data comparison method according to an embodiment of the present invention. Figure 5 As shown, for each source object in the source hash map, searching for fingerprint data of the source object in the cuckoo filter set further includes:

[0074] Step S501: Calculate the fingerprint data of the source object.

[0075] Preferably, if a source object is given as x, its fingerprint data is f=fingerprint(x).

[0076] Step S502: Use a first hash algorithm to perform hash processing on the source object to obtain a first bucket address.

[0077] Specifically, the cuckoo filter set uses two hash functions (ie, hash algorithms) to determine the location indexes of two buckets that can store data.

[0078] Preferably, based on the first hash algorithm, the first bucket address is i1=hash(x).

[0079] Step S503: performing hash processing on the fingerprint data of the source object using a second hash algorithm to obtain a fingerprint hash value.

[0080] Preferably, based on the fingerprint data f, the fingerprint hash value is hash(f).

[0081] Step S504: performing an XOR operation on the first bucket address and the fingerprint hash value to obtain a second bucket address.

[0082] Preferably, based on the second hash algorithm, the first bucket address and the fingerprint hash value are subjected to an XOR process; wherein, when the two are the same, the result of the XOR process is 0, and when the two are different, the result is 1; and the result of the XOR of 0 and any number is equal to the number itself; that is, the second bucket address is

[0083] Preferably, when f and i1 are known, i2 can be calculated; according to the duality, when f and i2 are known, i1 can also be calculated.

[0084] Step S505: In the cuckoo filter set, search whether the bucket corresponding to the first bucket address or the bucket corresponding to the second bucket address stores the fingerprint data of the source object.

[0085] Specifically, if the given data is x, the fingerprint f=fingerprint(x) of x is calculated; and the positions of the two buckets corresponding to the fingerprint data, i.e., i1 and i2, are obtained.

[0086] Further:

[0087] Preferably, the fingerprint data of the data x is searched in the cuckoo filter set in the following manner.

[0088] f = fingerprint(x)

[0089] i1=hash(x)

[0090]

[0091] if bucket[i1]or bucket[i2]has f then

[0092] return true;

[0093] return false;

[0094] Further, after obtaining the search results, the data synchronization verification results are determined according to the search results;

[0095] If the fingerprint data of the source object is found, determining that the data synchronization verification result of the source data corresponding to the source object is normal, and continuing to traverse other source objects in the source hash map;

[0096] If the fingerprint data of the source object is not found, it is determined that the data synchronization verification result of the source data corresponding to the source object is an abnormal result, and the source data corresponding to the source object is stored as abnormal data.

[0097] Specifically, the abnormal data is processed according to the preset abnormality handling plan, that is, corresponding remedial measures are implemented.

[0098] Preferably, the preset exception handling solution includes discarding, re-correcting source data for resynchronization, directly triggering resynchronization, etc.

[0099] According to a data comparison method provided in this embodiment, it is possible to concurrently traverse the fingerprint data of the synchronized data in the source hash mapping table, and perform concurrent search and comparison in the cuckoo filter set, efficiently achieving 100% verification and comparison of the entire data, fully ensuring the consistency and integrity before and after data migration.

[0100] Figure 6 FIG. 2 shows a schematic diagram of a data comparison process according to another embodiment of the present invention. Figure 6 As shown, where:

[0101] Step S601: Concurrently traverse the source objects in the source hash map.

[0102] Step S602: Determine whether the traversal is completed.

[0103] Specifically, if yes, execute step S607; if no, execute step S603.

[0104] Step S603: Searching for the source object in the Cuckoo filter set according to the hash algorithm.

[0105] Step S604: Based on the search result, determine whether the source object is found in the Cuckoo filter set.

[0106] Specifically, if yes, execute step S605; if no, execute step S606.

[0107] Step S605: traverse the next source object in the source hash map.

[0108] Specifically, return to step S601 to continue traversing the source data in the source hash mapping table.

[0109] Step S606: storing abnormal data based on failure to find the source object.

[0110] Specifically, after storing the abnormal data, the process returns to step S601 to continue traversing the source data in the source hash mapping table.

[0111] Step S607: Determine whether there is abnormal data.

[0112] Specifically, if the stored abnormal data is detected, step S608 is executed; if no abnormal data exists, step S609 is executed.

[0113] Step S608: Take remedial measures for abnormal data.

[0114] Step S609: Complete the data synchronization and proofreading task.

[0115] Figure 7FIG. 1 shows a structural block diagram of a data synchronization verification device according to an embodiment of the present invention. Figure 7 As shown, the device includes: a synchronization module 701 and a verification module 702; wherein,

[0116] The synchronization module 701 is used to generate source objects and target objects corresponding to the source data during the process of synchronizing source data from the source database to the target database, and to construct a source hash mapping table based on the source object and a cuckoo filter set based on the target object; wherein the cuckoo filter set stores the fingerprint data of the target object.

[0117] Specifically, the synchronization module 701 is further used to: read source data from the source database, perform data conversion processing on the source data, and generate source objects and target objects corresponding to the source data; add the source object to the source message queue, consume the source message queue to build a source hash mapping table; add the target object to the target message queue, write the target object to the target database, and consume the target message queue to build a cuckoo filter set.

[0118] Specifically, the synchronization module 701 is further used to: consume the source objects in the source message queue, serialize and store the source objects, and set a hash algorithm; use a linked list and a red-black tree model to process the serialized and stored source objects and construct a source hash mapping table.

[0119] Specifically, the synchronization module 701 is further used to: consume the target object in the target message queue, use the data of preset bits in the hash value of the target object as the fingerprint data of the target object, and write the fingerprint data into the cuckoo filter set.

[0120] The verification module 702 is configured to concurrently traverse the source objects in the source hash mapping table, search for fingerprint data of each source object in the source hash mapping table in the cuckoo filter set, and determine a data synchronization verification result based on the search result.

[0121] Specifically, the verification module 702 is further used to: calculate the fingerprint data of the source object; use a first hash algorithm to hash the source object to obtain a first bucket address; use a second hash algorithm to hash the fingerprint data of the source object to obtain a fingerprint hash value; perform an XOR process on the first bucket address and the fingerprint hash value to obtain a second bucket address; and search in the cuckoo filter set whether the fingerprint data of the source object is stored in the bucket corresponding to the first bucket address or the bucket corresponding to the second bucket address.

[0122] Specifically, the verification module 702 is further used to: determine the data synchronization verification result based on the search result; if the fingerprint data of the source object is found, determine that the data synchronization verification result of the source data corresponding to the source object is a normal result, and continue to traverse other source objects in the source hash mapping table; if the fingerprint data of the source object is not found, determine that the data synchronization verification result of the source data corresponding to the source object is an abnormal result, and store the source data corresponding to the source object as abnormal data.

[0123] Specifically, the verification module 702 is further configured to process abnormal data according to a preset abnormality processing solution.

[0124] According to a data synchronization verification device provided by this embodiment, in the process of synchronizing source data from a source database to a target database, a source object and a target object corresponding to the source data are generated, and a source hash mapping table is constructed based on the source object, and a cuckoo filter set is constructed based on the target object; wherein the fingerprint data of the target object is stored in the cuckoo filter set; the source objects in the source hash mapping table are traversed concurrently, and for each source object in the source hash mapping table, the fingerprint data of the source object is searched in the cuckoo filter set, and the data synchronization verification result is determined based on the search result. Utilizing the technical solution provided by the present invention, a source hash mapping table can be constructed during the data synchronization process, and a search can be performed in the cuckoo filter set based on the fingerprint data of the source data for all source data, thereby quickly achieving synchronization verification of all data, greatly improving the accuracy of verification, and ensuring consistency and integrity before and after data migration.

[0125] The present invention also provides a non-volatile computer storage medium, which stores at least one executable instruction. The executable instruction can execute the data synchronization verification method in any of the above method embodiments.

[0126] Figure 8 A schematic structural diagram of a computing device according to an embodiment of the present invention is shown. The specific embodiment of the present invention does not limit the specific implementation of the computing device.

[0127] like Figure 8 As shown, the computing device may include: a processor 802 , a communications interface 804 , a memory 806 , and a communication bus 808 .

[0128] in:

[0129] The processor 802 , the communication interface 804 , and the memory 806 communicate with each other via a communication bus 808 .

[0130] The communication interface 804 is used to communicate with other devices such as clients or other servers.

[0131] The processor 802 is configured to execute the program 810 , and specifically to execute the relevant steps in the above-mentioned data synchronization verification method embodiment.

[0132] Specifically, the program 810 may include program codes, which include computer operation instructions.

[0133] Processor 802 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention. The one or more processors included in a computing device may be processors of the same type, such as one or more CPUs, or processors of different types, such as one or more CPUs and one or more ASICs.

[0134] The memory 806 is used to store the program 810. The memory 806 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0135] Program 810 can be specifically used to cause processor 802 to execute the data synchronization verification method in any of the above-mentioned method embodiments. The specific implementation of each step in program 810 can refer to the corresponding descriptions of the corresponding steps and units in the above-mentioned data synchronization verification method embodiments, and will not be repeated here. Those skilled in the art will clearly understand that for the convenience and brevity of description, the specific working processes of the above-mentioned devices and modules can refer to the corresponding process descriptions in the above-mentioned method embodiments, and will not be repeated here.

[0136] The algorithm and display provided herein are not inherently related to any particular computer, virtual system or other device. Various general-purpose systems can also be used together with the teachings based on this. According to the above description, it is obvious that the structure required for constructing this type of system. In addition, the present invention is not directed to any specific programming language. It should be understood that various programming languages ​​can be utilized to realize the content of the present invention described herein, and the above description of specific languages ​​is for the purpose of disclosing the best mode of the present invention.

[0137] In the description provided herein, numerous specific details are described. However, it is understood that embodiments of the present invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques are not shown in detail so as not to obscure the understanding of this description.

[0138] Similarly, it should be understood that in order to streamline the present disclosure and aid understanding of one or more of the various inventive aspects, in the above description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, this disclosed method should not be interpreted as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as reflected in the claims, inventive aspects lie in less than all the features of the individual embodiments disclosed above. Accordingly, the claims that follow the detailed description are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of the invention.

[0139] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition may be divided into multiple submodules or subunits or subcomponents. All features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed herein may be combined in any combination, except that at least some of such features and / or processes or units are mutually exclusive. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0140] Furthermore, those skilled in the art will appreciate that although some embodiments described herein include certain features included in other embodiments but not other features, combinations of features from different embodiments are intended to be within the scope of the present invention and to form different embodiments. For example, in the claims, any of the claimed embodiments may be used in any combination.

[0141] The various component embodiments of the present invention can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It will be appreciated by those skilled in the art that a microprocessor or digital signal processor (DSP) can be used in practice to implement some or all of the functions of some or all of the components in accordance with the embodiments of the present invention. The present invention can also be implemented as a device or apparatus program (e.g., a computer program and a computer program product) for executing a portion or all of the methods described herein. Such a program implementing the present invention can be stored on a computer-readable medium, or can have the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0142] It should be noted that the above embodiments illustrate rather than limit the invention, and that those skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between brackets should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention may be implemented by means of hardware comprising several different elements and by means of appropriately programmed computers. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third etc. does not indicate any order. These words may be interpreted as names.

Claims

1. A data synchronization verification method, comprising: In the process of synchronizing source data from a source database to a target database, a data converter is used to convert the field types of the source data, a source object and a target object corresponding to the source data are generated, a source hash map is constructed based on the source object, and a cuckoo filter set is constructed based on the target object; wherein a hash algorithm is set, and key fields are selected to participate in the calculation, or dynamically configured fields are selected to participate in the calculation; the source hash map obtains and stores the source object based on a CAS operation, and a synchronization lock keyword is used to lock the corresponding data segment; the cuckoo filter set stores the fingerprint data of the target object; Concurrently traverse the source objects in the source hash mapping table, search for fingerprint data of each source object in the source hash mapping table in the cuckoo filter set, and determine a data synchronization verification result according to the search result.

2. The method according to claim 1, characterized in that The constructing of a source hash map according to the source object and the constructing of a cuckoo filter set according to the target object further include: Add the source object to the source message queue, consume the source message queue to build a source hash map; Add the target object to a target message queue, write the target object to a target database, and consume the target message queue to construct a cuckoo filter set.

3. The method according to claim 2, characterized in that The consuming the source message queue to construct a source hash mapping table further includes: Consume the source object in the source message queue, serialize and store the source object, and set a hash algorithm; The linked list and red-black tree models are used to process the serialized stored source objects and construct the source hash map.

4. The method according to claim 2, characterized in that The consuming the target message queue to construct a cuckoo filter set further includes: The target object in the target message queue is consumed, data with a preset number of bits in the hash value of the target object is used as fingerprint data of the target object, and the fingerprint data is written into the cuckoo filter set.

5. The method according to any one of claims 1 to 4, characterized in that For each source object in the source hash map, searching for fingerprint data of the source object in the cuckoo filter set further comprises: Calculating fingerprint data of the source object; Hash the source object using a first hash algorithm to obtain a first bucket address; Performing hash processing on the fingerprint data of the source object using a second hash algorithm to obtain a fingerprint hash value; Performing an XOR operation on the first bucket address and the fingerprint hash value to obtain a second bucket address; In the cuckoo filter set, it is searched whether the bucket corresponding to the first bucket address or the bucket corresponding to the second bucket address stores the fingerprint data of the source object.

6. The method according to any one of claims 1 to 5, characterized in that Determining the data synchronization verification result according to the search result further includes: If the fingerprint data of the source object is found, determining that the data synchronization verification result of the source data corresponding to the source object is normal, and continuing to traverse other source objects in the source hash map; If the fingerprint data of the source object is not found, it is determined that the data synchronization verification result of the source data corresponding to the source object is an abnormal result, and the source data corresponding to the source object is stored as abnormal data.

7. The method according to claim 6, characterized in that The method further comprises: Process abnormal data according to the preset abnormality handling plan.

8. A data synchronization verification device, comprising: Synchronization module and verification module; wherein, The synchronization module is configured to, during the process of synchronizing source data from a source database to a target database, convert the field types of the source data using a data converter, generate source objects and target objects corresponding to the source data, construct a source hash map based on the source objects, and construct a cuckoo filter set based on the target objects; wherein a hash algorithm is set, and either key fields or dynamically configured fields are selected for participation in the calculation; the source hash map acquires and stores the source objects based on a CAS operation, and locks corresponding data segments using a synchronization lock keyword; and the cuckoo filter set stores fingerprint data of the target object; The verification module is used to concurrently traverse the source objects in the source hash mapping table, search for fingerprint data of each source object in the source hash mapping table in the cuckoo filter set, and determine a data synchronization verification result according to the search result.

9. A computing device comprising: A processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform an operation corresponding to the data synchronization verification method according to any one of claims 1 to 7.

10. A computer storage medium, wherein at least one executable instruction is stored in the storage medium, and wherein the executable instruction enables a processor to execute an operation corresponding to the data synchronization verification method according to any one of claims 1 to 7.

Images