Data verification method and device, electronic equipment, medium and product

Through hash function division and bronz filter verification, combined with cyclic redundancy verification, the problem of low efficiency of large-scale data verification after data migration in the existing technology is solved, and fast and accurate data verification is achieved.

CN119088791APending Publication Date: 2024-12-06CHINA MOBILE GRP BEIJING +1
View PDF 0 Cites 6 Cited by

Patent Information

Application Number
CN202411077851.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-07
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

The prior art is difficult to efficiently verify the accuracy of large-scale data after data migration, especially in the presence of tables without primary keys and unique indexes, resulting in inefficient verification.

Method used

The data is divided into row collection buckets through a hash function, and is checked based on hash value calculation and Bloom filter, combined with cyclic redundancy verification to ensure the accuracy and efficiency of data verification.

Benefits of technology

It realizes fast and accurate data verification, avoids hash collisions and matching errors, improves the efficiency of data migration verification, and is suitable for large-scale data verification.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119088791A_ABST
    Figure CN119088791A_ABST
Patent Text Reader

Abstract

The invention provides a data verification method and device, electronic equipment, a medium and a product, and belongs to the technical field of Internet, the method comprises the following steps: if record numbers of verification tables in a source end database and a target end database are consistent, obtaining first to-be-verified data from the source end database and obtaining second to-be-verified data from the target end database; determining a hash function corresponding to each row of data in the first to-be-verified data; performing row set bucket division on the first to-be-verified data, and then performing hash value calculation on corresponding row data in the divided row set buckets based on each hash function to obtain at least one first row set bucket; performing row set bucket division on the second to-be-verified data, and then performing hash value calculation on corresponding row data in the divided row set buckets based on each hash function to obtain at least one second row set bucket; and performing data verification according to each first row set bucket and each second row set bucket to obtain a verification result. According to the invention, the data verification efficiency can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of Internet technology, and in particular to a data verification method, device, electronic device, medium and product. Background Art

[0002] With the development of Internet technology, traditional centralized databases have gradually encountered bottlenecks. In particular, the telecommunications industry has been gradually migrating to distributed databases with high concurrency and low latency characteristics. At this time, how to verify the accuracy and efficiency of data after data migration is particularly important; The current verification method is to verify the data at the source and destination ends row by row. Although it can meet the verification time requirements when the data volume is small, for some industries, the verification data volume is large and the downtime window is short. The database may have a large number of tables without primary keys and unique indexes at the same time. If the existing technical solution is used, the efficiency of data verification will be greatly reduced. Summary of the invention

[0003] The present application provides a data verification method, device, electronic device, medium and product to solve the problem that the efficiency of data verification will be greatly reduced by adopting the existing technical solutions, so as to improve the efficiency of data verification.

[0004] The present application provides a data verification method, which includes: if the number of records in the verification table in the source database is consistent with the number of records in the verification table in the destination database, obtaining first data to be verified from the source database and obtaining second data to be verified from the destination database; the destination database stores data migrated from the source database; determining the hash functions corresponding to each row of data in the first data to be verified; after dividing the first data to be verified into row set buckets, based on each hash function, respectively calculate hash values ​​for the corresponding row data in the divided row set buckets to obtain at least one first row set bucket; after dividing the second data to be verified into row set buckets, based on each hash function, respectively calculate hash values ​​for the corresponding row data in the divided row set buckets to obtain at least one second row set bucket; performing data verification according to each of the first row set buckets and each of the second row set buckets to obtain a verification result.

[0005] According to a data verification method provided by the present application, when performing data verification based on each of the first row collection buckets and each of the second row collection buckets, for a row collection bucket pair consisting of a first row collection bucket and a second row collection bucket with the same number, the following steps are respectively performed: a Bloom filter is constructed based on each hash value of the first row collection bucket in the current row collection bucket pair; if all hash values ​​of the second row collection bucket in the current row collection bucket pair can find a hash value with the same value in the Bloom filter, it is determined that the data verification of the current row collection bucket pair is successful, and it is determined that the migration data verification is successful when the data of all row collection bucket pairs are verified successfully.

[0006] According to a data verification method provided by the present application, after constructing a Bloom filter based on the hash values ​​of the first row set bucket in the current row set bucket pair, it also includes: if any hash value of the second row set bucket in the current row set bucket pair does not have a hash value with the same value in the Bloom filter, the current row set bucket pair is determined as a row set bucket pair that fails data verification, and the two row set buckets in the row set bucket pair that fails data verification are respectively divided into row set buckets to form two new row set bucket pairs; if the data verification of any new row set bucket pair fails, the corresponding new row set bucket pair is determined as a row set bucket pair that fails data verification, and the step of performing row set bucket division on the two row set buckets in the row set bucket pair that fails data verification and forming two new row set bucket pairs is returned to execute, until the data of the two new row set bucket pairs are verified successfully or the number of executions reaches a preset threshold.

[0007] According to a data verification method provided by the present application, if the number of executions reaches a preset threshold, the following steps are performed: two row data corresponding to two hash values ​​with the same row number but inconsistent numerical values ​​are obtained from the two row set buckets of the row set bucket pair that failed the data verification; a cyclic redundancy check is performed on the two row data; if the two row data pass the cyclic redundancy check, it is determined that the data verification of the corresponding row set bucket pair is successful; if the two row data fail the cyclic redundancy check, it is determined that the migration data verification has failed.

[0008] According to a data verification method provided by the present application, the method of obtaining two row data corresponding to two hash values ​​with the same row number but inconsistent numerical values ​​from two row set buckets of a row set bucket pair that failed data verification includes: hash mapping the data of the rows with the same number in the two row set buckets of the row set bucket pair that failed data verification through the same hash function, respectively, to obtain hash pairs with the same number as the number of rows in the row set bucket pair; wherein each hash pair includes hash values ​​corresponding to the rows with the same number in the two row set buckets of the row set bucket pair that failed data verification; determining a target hash pair from each hash pair; the two hash values ​​in the target hash pair are different; and obtaining two row data corresponding to the target hash pair.

[0009] According to a data verification method provided by the present application, after dividing the first data to be verified into row set buckets, hash value calculations are performed on the corresponding row data in the divided row set buckets based on each hash function to obtain at least one first row set bucket, including: dividing each row data into row set buckets according to a preset bucket depth to obtain at least one row set bucket; and hash value calculations are performed on the corresponding row data in each row set bucket based on each hash function to obtain at least one first row set bucket.

[0010] According to a data verification method provided by the present application, determining the hash functions corresponding to each row of data in the first data to be verified includes: inputting each row of data in the first data to be verified into a hash function matching model, and obtaining the hash functions corresponding to the corresponding row of data output by the hash function matching model; wherein the hash function matching model is obtained by model training based on sample data, data type annotation information of the sample data, and hash function annotation information of the sample data.

[0011] The present application also provides a data verification device, including: an acquisition module, which is used to obtain first data to be verified from the source database and second data to be verified from the destination database if the number of records in the verification table in the source database is consistent with the number of records in the verification table in the destination database; the destination database stores data migrated from the source database; a determination module, which is used to determine a hash function according to the data type of the source database; a first partitioning module, which is used to perform row set bucket partitioning after performing hash value calculations on corresponding row data in the first data to be verified based on each hash function, to obtain at least one first row set bucket; a second partitioning module, which is used to perform row set bucket partitioning after performing hash value calculations on corresponding row data in the second data to be verified based on the hash function, to obtain at least one second row set bucket; a verification module, which is used to perform data verification based on each of the first row set buckets and each of the second row set buckets to obtain a verification result.

[0012] The present application also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the above-described data verification methods when executing the computer program.

[0013] The present application also provides a medium, which is a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, any of the data verification methods described above is implemented.

[0014] The present application also provides a product, which is a computer program product, including a computer program, and when the computer program is executed by a processor, it implements any of the data verification methods described above.

[0015] The data verification method, device, electronic device, medium and product provided by the present application, after determining that the number of records in the verification table in the source database is consistent with the number of records in the verification table in the destination database, and obtaining the first data to be verified from the source database and the second data to be verified from the destination database storing the data migrated from the source database, by determining the hash functions corresponding to each row of data in the first data to be verified, it is possible to divide the first data to be verified into row set buckets, and then calculate the hash values ​​of the corresponding row data in the divided row set buckets based on each hash function to obtain at least one first row set bucket, and after dividing the second data to be verified into row set buckets, calculate the hash values ​​of the corresponding row data in the divided row set buckets based on each hash function to obtain at least one second row set bucket; thereby, data verification can be further quickly performed according to each first row set bucket and each second row set bucket to obtain a verification result. Since the hash function is selected according to the type of data in the database, and the hash value is calculated according to the hash function after the row set buckets are divided, and the multiple row set buckets containing the hash values ​​are verified, the data can be divided in sequence and different hash values ​​can be generated to avoid hash conflicts and matching errors. Even if there is a table without a primary key or unique index, data verification can be completed quickly, thereby improving the efficiency of data verification. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present application or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0017] Figure 1 It is a flow chart of the data verification method provided by this application.

[0018] Figure 2 It is a schematic diagram of the structure of the electronic device provided by this application. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in this application. Obviously, the described embodiments are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0020] It should be noted that in the description of the present application, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "include one..." do not exclude the existence of other identical elements in the process, method, article or device including the elements. The orientation or position relationship indicated by the terms "upper", "lower" and the like is based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present application. Unless otherwise clearly specified and limited, the terms "installed", "connected" and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be directly connected, or indirectly connected through an intermediate medium, or it can be the internal communication of two elements. For those skilled in the art, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0021] The terms "first", "second", etc. in this application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0022] Combine the following Figure 1-Figure 2 Describe the data verification method, device, electronic device, medium and product of the present application.

[0023] Figure 1 It is a flow chart of the data verification method provided by this application, such as Figure 1 As shown, the method includes the following steps 100 to 500.

[0024] Step 100: If the number of records in the verification table in the source database is consistent with the number of records in the verification table in the target database, obtain the first data to be verified from the source database and obtain the second data to be verified from the target database; the target database stores the data migrated from the source database.

[0025] Step 200: Determine the hash function corresponding to each row of data in the first data to be verified.

[0026] Step 300: After dividing the first data to be checked into row set buckets, hash value calculations are performed on corresponding row data in the divided row set buckets based on respective hash functions to obtain at least one first row set bucket.

[0027] Step 400: After dividing the second data to be checked into row set buckets, hash value calculations are performed on corresponding row data in the divided row set buckets based on respective hash functions to obtain at least one second row set bucket.

[0028] Step 500: Perform data verification on each first row set bucket and each second row set bucket to obtain a verification result.

[0029] It should be noted that the executor of the data verification method provided in the embodiment of the present application can be a server, a computer device, etc., such as a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a wearable device, an ultra-mobile personal computer (UMPC), a netbook or a personal digital assistant (PDA), etc.

[0030] The server or computer device of the present application may be provided with a verification program, and the data verification device may be controlled to execute the data verification method of the present application through the execution of the verification program.

[0031] With the development of Internet technology, traditional centralized databases have gradually encountered bottlenecks. Therefore, data in traditional centralized databases are being gradually migrated to distributed databases with high concurrency and low latency characteristics. This application defines the database before migration as the source database and the database after migration as the destination database. On this basis, it is necessary to verify the data before and after data migration to determine whether there are any abnormalities in the data migration process.

[0032] The user of this application can initiate a data verification instruction after completing the data migration, and the verification program can obtain the number of records in the verification table in the source database and the number of records in the verification table in the destination database based on this.

[0033] Furthermore, the two record numbers may be verified to determine whether the two record numbers are consistent.

[0034] If it is determined that the number of records in the verification table in the source database is inconsistent with the number of records in the verification table in the target database, it is determined that the migration data verification fails, and a verification result is obtained.

[0035] If it is determined that the number of records in the verification table in the source database is consistent with the number of records in the verification table in the destination database, the first data to be verified can be obtained from the source database and the second data to be verified can be obtained from the destination database.

[0036] Specifically, the database is divided according to the timestamp of the changed data: in response to the data verification instruction, the historical verification record of the source database is queried, and if the source database has the corresponding historical verification record, the timestamp of the last data verification of the database is obtained.

[0037] Furthermore, the change data of the source database and the target database are sampled respectively: according to the timestamp, query statements or application programming interface (API) calls are used to query and sort the source database after the timestamp as the first data to be verified, and the synchronization data of the target database after the timestamp is obtained as the second data to be verified. When obtaining data, if the data in the database has no primary key / unique index, the operator can select the sorted and extracted fields as rows (for example, select fields with large discreteness, such as mobile phone numbers).

[0038] Furthermore, a hash function can be matched to each row of data according to the data characteristics of each row of data in the first data to be verified. For example, MD5, SHA-1, SHA-256 and other hash functions are matched for row data whose data type is text data; MD5, SHA-1, SHA-256, CRC32 and other hash functions are matched for row data whose data type is digital data; SHA-256, Perceptual Hashing and other hash functions are matched for row data whose data type is image data. Among them, MD5 is a widely used hash function that generates a 128-bit (16-byte) hash value. SHA-1 generates a 160-bit (20-byte) hash value, which is used in security applications such as data integrity verification and digital signatures. SHA-256 generates a 256-bit (32-byte) hash value and is part of the SHA-2 family. CRC32 is a cyclic redundancy check algorithm that generates a 32-bit (4-byte) checksum.

[0039] Furthermore, hash value calculations can be performed on corresponding row data in the first data to be verified according to each hash function, and row set buckets can be divided according to the obtained hash values ​​and row data to obtain at least one first row set bucket.

[0040] Furthermore, hash values ​​of corresponding row data in the second data to be verified may be calculated according to each hash function, and then row set buckets may be divided to obtain at least one second row set bucket.

[0041] Furthermore, data verification is performed on each row set bucket pair consisting of a first row set bucket and a second row set bucket with the same number to obtain a verification result.

[0042] Specifically, when the data of all row set bucket pairs are verified successfully, it can be determined that the migration data verification is successful, and a verification result is obtained.

[0043] If the data verification of any row set bucket pair fails, it is determined that the migration data verification fails, and the verification result is obtained.

[0044] It should be noted that successful migration data verification means that the data before migration is completely consistent with the data after migration, and failed migration data verification means that the data before migration is not completely consistent with the data after migration.

[0045] The data verification method provided by the embodiment of the present application determines that the number of records in the verification table in the source database is consistent with the number of records in the verification table in the destination database, and obtains the first data to be verified from the source database and the second data to be verified from the destination database storing the data migrated from the source database. Then, by determining the hash functions corresponding to each row of data in the first data to be verified, the first data to be verified can be divided into row set buckets, and then hash values ​​of the corresponding row data in the divided row set buckets can be calculated based on each hash function to obtain at least one first row set bucket, and the second data to be verified can be divided into row set buckets, and then hash values ​​of the corresponding row data in the divided row set buckets can be calculated based on each hash function to obtain at least one second row set bucket; thereby, data verification can be further quickly performed according to each first row set bucket and each second row set bucket to obtain a verification result. Since the hash function is selected according to the type of data in the database, and the hash value is calculated according to the hash function after the row set buckets are divided, and the multiple row set buckets containing the hash values ​​are verified, the data can be divided in sequence and different hash values ​​can be generated to avoid hash conflicts and matching errors. Even if there is a table without a primary key or unique index, data verification can be completed quickly, thereby improving the efficiency of data verification.

[0046] The present application also automatically identifies changed data based on the timestamp, and only rechecks the updated data after the timestamp. The data that has been checked (the data before the timestamp) does not need to be rechecked, which improves the efficiency of the check and greatly saves computing resources.

[0047] In one embodiment, determining the hash functions corresponding to each row of data in the first data to be verified includes: inputting each row of data in the first data to be verified into a hash function matching model, and obtaining the hash functions corresponding to the corresponding row of data output by the hash function matching model; wherein the hash function matching model is obtained by model training based on sample data, data type annotation information of the sample data, and hash function annotation information of the sample data.

[0048] The present application can use existing sample data, data type annotation information of the sample data, and hash function annotation information of the sample data as a data set to train a multimodal model, and obtain a multimodal hash function matching model, so that the trained model can automatically identify the hash functions matched by different data types in the data. The data set includes various types of data, such as text data, digital data, image data, etc.

[0049] In this application, one or more matching hash functions can be set for different data types in advance. For example, matching hash functions such as MD5, SHA-1, and SHA-256 can be set for data types; matching hash functions such as MD5, SHA-1, SHA-256, and CRC32 can be set for data types; matching hash functions such as SHA-256 and Perceptual Hashing can be set for data types.

[0050] As a result, the appropriate hash function can be automatically selected according to data of different data types, and the hash function can be automatically replaced when a hash conflict occurs; or, when there is mixed data in the data, such as text data and image data, the corresponding hash function can be automatically identified and called to calculate the hash value.

[0051] The present application determines the hash functions corresponding to each row of data in the first data to be verified, so that after dividing the first data to be verified into row set buckets, the hash values ​​of the corresponding row data in the divided row set buckets are calculated based on each hash function to obtain at least one first row set bucket, and after dividing the second data to be verified into row set buckets, the hash values ​​of the corresponding row data in the divided row set buckets are calculated based on each hash function to obtain at least one second row set bucket; thus, data verification can be further performed quickly based on each first row set bucket and each second row set bucket to obtain a verification result. Since the hash function is selected according to the type of data in the database, and the hash value is calculated according to the hash function after dividing the row set buckets, and the multiple row set buckets containing the hash values ​​are verified, the data can be divided in order and different hash values ​​can be generated to avoid hash conflicts and matching errors. Even if there is a table without a primary key or unique index, data verification can be completed quickly, thereby improving the efficiency of data verification.

[0052] In one embodiment, after the first data to be checked is divided into row set buckets, hash values ​​are calculated for the corresponding row data in the divided row set buckets based on each hash function to obtain at least one first row set bucket, including: dividing each row data into row set buckets according to a preset bucket depth to obtain at least one row set bucket; and calculating hash values ​​for the corresponding row data in each row set bucket based on each hash function to obtain at least one first row set bucket.

[0053] For the first data to be verified, the present application may obtain a preset bucket depth, where the bucket depth refers to the maximum row value of a bucket. For example, in one embodiment, 5000 rows are set as the depth of a bucket.

[0054] Furthermore, the present application can divide the first to-be-checked data into one or more row set buckets according to the preset bucket depth and the order of data acquisition, wherein each row set bucket includes the same number of row data as the preset bucket depth, and each row data can have a corresponding number. And each row set bucket also has a corresponding bucket number.

[0055] Furthermore, for each row of data in the row set bucket, a hash function corresponding to the row of data may be obtained, and a hash value of the row of data may be calculated according to the hash function.

[0056] Therefore, for each divided row set bucket, a first row set bucket can be formed by the bucket number, the hash value of each row, the row number, the row data and other information, thereby obtaining one or more first row set buckets.

[0057] It should be noted that when calculating the hash value of a row set bucket, the overall calculation task can be divided into multiple subtasks, each subtask corresponds to a row set bucket, and the subtasks can be evenly divided to different cores according to the number of row set buckets and the amount of calculation.

[0058] The divided subtasks are assigned to multiple different cores for parallel computing. Each core is responsible for calculating the hash value of one or more row set buckets. Threads or processes can be used to implement parallel computing.

[0059] The data of the row set bucket is allocated to each core. Since the data in each row set bucket is inconsistent, the progress of calculating the hash value is also different (for example, it is more difficult to calculate the hash value of an image than to calculate the hash value of a text). Therefore, a combination of static allocation and dynamic allocation can be used. Before starting to calculate the hash value, the row set bucket is allocated to each core through static allocation. Each core is responsible for calculating a fixed row set bucket. During the calculation process, the progress of the hash value calculation is obtained in real time through log recording. When a core completes the calculation of the hash value of a row set bucket, the log progress will feedback the information that the calculation is completed. The core that has completed the hash value calculation will be idle. At this time, the row set bucket waiting to be processed is dynamically allocated to the idle core, and the idle core performs the hash value calculation processing, thereby improving the hash value calculation efficiency while saving computing resources. In this process, the hash function used by each row set bucket can be flexibly switched.

[0060] Similarly, when verifying the hash values ​​of the row set buckets at the source and destination ends, a multi-core processor or multi-threading may be used to improve processing efficiency.

[0061] It should be further explained that due to the huge amount of data, when calculating hash values ​​for different rows of data, the same hash value may be obtained, resulting in a hash conflict, thus affecting the correctness of the data verification. For example, if the source row a1 corresponds to the migration data of the destination row b1, the hash values ​​of a1 and b1 are the same; the source row a2 corresponds to the migration data of the destination row b2, and the hash values ​​of a2 and b2 are the same; row a1 may produce the same hash value as row a2, and in the process of verifying the destination row, if the destination fails to migrate the row a1, resulting in the destination end not having a migration row b1 corresponding to a1, but having a migration row b2 corresponding to a2, then, in the process of verifying the migration of a1, the hash value of a1 may match the hash value of b2, thereby affecting the correctness of the a1 verification result. In this regard, the present application also performs the following steps when performing hash value calculations: a. Define a hash table, which includes a hash function and an array for storing hash values; b. When calculating the hash value for each row, store the calculated hash value. If the hash value position in the hash table is empty, store the hash value in the hash table and continue to calculate the hash value of the next row; if the hash value position in the hash table is not empty, recalculate the hash value of the row data using a different hash function as a mirrored hash value and store the mirrored hash value in the hash table; c. If the position of the hash value is not empty when storing the mirrored hash value in the hash table, it means that there is already a row in the database with the same hash value, and a hash conflict occurs. At this time, the hash value function should be matched on the hash value, and other hash functions should be used to calculate the hash value of the row to obtain a new hash value and store it in the hash table as a mirrored new hash value.

[0062] For example, the hash value of "abc" is calculated to be 2, and 2 is stored at index position 2 in the hash table; then, the hash value of "def" is calculated to be 2, and it is found that index position 2 is already occupied, so a hash conflict occurs. Another hash function is used to calculate the hash value of "def", such as the original hash function is SHA-256, which can be converted to MD5 for calculation. If the new hash value of "def" is 5, index position 5 in the hash table is empty, and 5 is stored at index position 5. In this way, hash values ​​can be calculated in real time while avoiding hash conflicts.

[0063] d. After the hash value of each row is calculated, the hash value of each row in the row collection bucket is merged and concatenated into a string as the hash value of the corresponding row collection bucket; the row collection buckets on the source and destination ends are operated in the same row order. If the data migration is correct, the hash value of the row collection bucket on the source end should be consistent with the hash value of the row collection bucket on the destination end.

[0064] Based on the same processing method, the second data to be verified can be divided into row set buckets and then hash values ​​of the corresponding row data in the divided row set buckets can be calculated based on each hash function to obtain at least one second row set bucket, which will not be repeated in this application.

[0065] This application uses a hash bucket verification algorithm to sequentially divide data into row set buckets of fixed size, and can select hash functions for data in the row set bucket according to different data types. By establishing a hash table, data with repeated hash values ​​are matched with different hash functions to recalculate the hash values, and then the hash values ​​of each row are concatenated to form the hash value of the row set bucket. This solves the problem of hash conflicts that may occur when using hash value calculations, resulting in matching errors, and can improve the accuracy of data verification, thereby improving the efficiency of data verification.

[0066] The present application also uses a multi-core processor to generate hash values ​​for row set buckets, and performs parallel calculations on the hash values ​​of each row set bucket to improve the calculation efficiency of the hash values; and, uses log data to record completed cores and provide feedback, and uses a combination of static allocation and dynamic allocation to allocate row set buckets waiting to be processed to idle cores for processing, thereby making full use of computing resources and improving processing efficiency.

[0067] In one embodiment, when performing data verification based on each first row collection bucket and each second row collection bucket, the following steps are performed respectively for the row collection bucket pair consisting of the first row collection bucket and the second row collection bucket with the same number: a Bloom filter is constructed based on the hash values ​​of the first row collection bucket in the current row collection bucket pair; if all the hash values ​​of the second row collection bucket in the current row collection bucket pair can find the hash value with the same value in the Bloom filter, it is determined that the data verification of the current row collection bucket pair is successful, and it is determined that the migration data verification is successful when the data of all row collection bucket pairs are verified successfully.

[0068] Furthermore, after constructing a Bloom filter based on the hash values ​​of the first row set bucket in the current row set bucket pair, it also includes: if any hash value of the second row set bucket in the current row set bucket pair does not have the same hash value in the Bloom filter, the current row set bucket pair is determined to be a row set bucket pair that has failed data verification, and the two row set buckets in the row set bucket pair that has failed data verification are respectively divided into row set buckets to form two new row set bucket pairs; if the data verification of any new row set bucket pair fails, the corresponding new row set bucket pair is determined to be a row set bucket pair that has failed data verification, and the step of performing row set bucket division on the two row set buckets in the row set bucket pair that has failed data verification and forming two new row set bucket pairs is returned to execute, until the data of the two new row set bucket pairs are verified successfully or the number of executions reaches a preset threshold.

[0069] Furthermore, if the number of executions reaches a preset threshold, the following steps are performed: from the two row set buckets of the row set bucket pair where data verification failed, obtain two row data corresponding to two hash values ​​with the same row number but inconsistent values; perform a cyclic redundancy check on the two row data; if the two row data pass the cyclic redundancy check, it is determined that the data verification of the corresponding row set bucket pair is successful; if the two row data fail the cyclic redundancy check, it is determined that the migration data verification has failed.

[0070] Furthermore, two row data corresponding to two hash values ​​with the same row number but inconsistent numerical values ​​are obtained from the two row set buckets of the row set bucket pair where data verification fails, including: hash mapping the data of the rows with the same number in the two row set buckets of the row set bucket pair where data verification fails through the same hash function, to obtain hash pairs with the same number as the rows in the row set bucket pair; wherein each hash pair includes hash values ​​corresponding to the rows with the same number in the two row set buckets of the row set bucket pair where data verification fails; determining a target hash pair from each hash pair; the two hash values ​​in the target hash pair are different; and obtaining two row data corresponding to the target hash pair.

[0071] Specifically, during the verification process, the row set bucket pair with the same number on the source and destination ends is found according to the bucket number corresponding to the row set bucket, and then the hash values ​​of the two set buckets in the row set bucket pair are verified. The verification process uses Bloom Filter. When comparing and verifying the hash values ​​of the row set bucket on the source end (i.e., the first row set bucket) and the row set bucket on the destination end (i.e., the second row set bucket), the hash value of the row set bucket on the source end is constructed into a Bloom Filter, and then the hash values ​​of the row set bucket on the destination end are queried one by one. If the query results all exist in BloomFilter, it means that the data corresponding to the row set bucket has been correctly migrated and the hash value verification is successful. If the data verification of all row set buckets is successful, the verification result can be returned to be consistent, confirming that all migrated data has been verified successfully; otherwise, the row set bucket that failed the verification proceeds to the next step for further verification. Furthermore, the data corresponding to the row set buckets that failed the verification are divided, and the source and destination are re-divided using the same division height, gradually narrowing the verification range. For example, if it is divided into half of the original bucket depth, the source generates two row set buckets, numbered as 1-a bucket and 1-b bucket; the destination generates two corresponding row set buckets, numbered as 2-a bucket and 2-b bucket. In theory, since the data on the destination end is migrated from the data on the source end, and the data in the row set buckets are arranged in order according to rows, they are re-divided according to the same division height during division. Therefore, the data and height in buckets 1-a and 2-a should be the same, and the data and height in buckets 1-b and 2-b should be the same. If buckets 1-a and 2-a are consistent during verification in step 201, buckets 1-b and 2-b can be directly verified at this time to improve the efficiency of verification.

[0072] Regenerate the corresponding row set bucket number, calculate the hash value of the row set bucket after division, and verify the row set bucket again according to the hash value. Continuing with the above example, verify the hash values ​​of buckets 1-a and 2-a. If the verification results are consistent, return the verification result and the verification is complete. If the verification results are inconsistent, filter out the rows with consistent hash values, divide buckets 1-a and 2-a again, and calculate the hash value for verification. If the verification still fails after executing the preset number of times (the preset number of times can be set according to needs, such as 3 times, 4 times, etc.), proceed to the next step; Furthermore, if the hash value comparison between the row set bucket at the source end and the row set bucket at the destination end is still inconsistent, since the hash value of the row set bucket is formed by concatenating the hash values ​​of each row, after multiple divisions, the number of rows in the row set bucket is small and the hash value is also short. At this time, the rows of the two row set buckets can be hash mapped according to the same hash function, such as the hash value of bucket A at the source end: (hash(row a1)+hash(row b1)+hash(row c1))=111; the hash value of bucket B at the destination end: (hash(row a2)+hash(row b2)+hash(row f2))=119. In the hash mapping process, the hash values ​​of row a1 and row a2 are the same, the hash values ​​of row b1 and row b2 are the same, and the hash values ​​of row c1 and row f2 are different. Based on this, the next step of related calculations is performed on row c1 at the source end and row f2 at the destination end. Furthermore, the CRC32 value is calculated for the row data where the hash value match fails, and a list of {matching column values, matching column values ​​corresponding to CRC32 values} is obtained for the source and destination ends. Using the above example, the CRC32 value is calculated for row c1 on the source end and row f2 on the destination end, and the matching column value on the source end is used as the input condition to compare the CRC32 values ​​of the source and destination ends in turn; if the verification and comparison are normal, the data in the two tables are consistent, otherwise the row records with inconsistent verification results are thrown out.

[0073] This application uses a Bloom filter to verify the hash value, and re-divides the row set buckets that fail the verification, generates different row set bucket numbers, gradually narrows the division range, and filters out rows with consistent hash values; when the number of divisions reaches the division threshold, a hash map is used to find the rows that fail to match; the CRC32 value is calculated for the rows that fail to match for further verification, which can improve data verification efficiency.

[0074] This application uses a hash bucket verification algorithm to sequentially divide the data in the verification table into row set buckets of fixed size. By calculating and verifying the hash value corresponding to each row set bucket on the source and destination ends, it can quickly filter out consistent rows and improve the overall verification speed without considering whether the table has a primary key / unique index.

[0075] The data verification device provided by the present application is described below. The data verification device described below and the data verification method described above can be referenced to each other.

[0076] Furthermore, the present application also provides a data verification device.

[0077] The data verification device may also include: an acquisition module, which is used to obtain the first data to be verified from the source database and the second data to be verified from the destination database if the number of records in the verification table in the source database is consistent with the number of records in the verification table in the destination database; the destination database stores data migrated from the source database; a determination module, which is used to determine the hash functions corresponding to each row of data in the first data to be verified; a first partitioning module, which is used to divide the first data to be verified into row set buckets and then calculate the hash values ​​of the corresponding row data in the divided row set buckets based on each hash function to obtain at least one first row set bucket; a second partitioning module, which is used to divide the second data to be verified into row set buckets and then calculate the hash values ​​of the corresponding row data in the divided row set buckets based on each hash function to obtain at least one second row set bucket; a verification module, which is used to perform data verification based on each of the first row set buckets and each of the second row set buckets to obtain a verification result.

[0078] In one embodiment, the determination module is specifically used to: input each row of data in the first data to be verified into a hash function matching model respectively, and obtain a hash function corresponding to the corresponding row of data output by the hash function matching model; wherein the hash function matching model is obtained by model training based on sample data, data type annotation information of the sample data, and hash function annotation information of the sample data.

[0079] In one embodiment, the first partitioning module is specifically used to: partition each row of data into row set buckets according to a preset bucket depth to obtain at least one row set bucket; and calculate hash values ​​for the corresponding row data in each row set bucket based on each hash function to obtain at least one first row set bucket.

[0080] In one embodiment, the verification module is specifically used to perform the following steps, respectively, for a row set bucket pair consisting of a first row set bucket and a second row set bucket with the same number, when performing data verification based on each of the first row set buckets and each of the second row set buckets: construct a Bloom filter based on the hash values ​​of the first row set bucket in the current row set bucket pair; if all hash values ​​of the second row set bucket in the current row set bucket pair can find hash values ​​with the same value in the Bloom filter, it is determined that the data verification of the current row set bucket pair is successful, and it is determined that the migration data verification is successful when the data of all row set bucket pairs are verified successfully.

[0081] In one embodiment, the verification module is also used for: if any hash value of the second row set bucket in the current row set bucket pair does not have a hash value with the same value in the Bloom filter, then the current row set bucket pair is determined as a row set bucket pair that fails data verification, and the two row set buckets in the row set bucket pair that fails data verification are respectively divided into row set buckets to form two new row set bucket pairs; if the data verification of any new row set bucket pair fails, then the corresponding new row set bucket pair is determined as a row set bucket pair that fails data verification, and the step of performing row set bucket division on the two row set buckets in the row set bucket pair that fails data verification and forming two new row set bucket pairs is returned to execute, until the data of the two new row set bucket pairs are verified successfully or the number of executions reaches a preset threshold.

[0082] In one embodiment, the verification module is also used to: obtain two row data corresponding to two hash values ​​with the same row number but inconsistent numerical values ​​from the two row set buckets of the row set bucket pair that failed data verification; perform a cyclic redundancy check on the two row data; if the two row data pass the cyclic redundancy check, it is determined that the data verification of the corresponding row set bucket pair is successful; if the two row data fail the cyclic redundancy check, it is determined that the migration data verification has failed.

[0083] In one embodiment, the verification module is also used to: perform hash mapping on the data of the same numbered rows in the two row set buckets of the row set bucket pair where data verification failed respectively through the same hash function to obtain hash pairs with the same number of rows in the row set bucket pair; wherein each hash pair includes hash values ​​corresponding to the same numbered rows in the two row set buckets of the row set bucket pair where data verification failed; determine a target hash pair from each hash pair; the two hash values ​​in the target hash pair are different; and obtain two row data corresponding to the target hash pair.

[0084] The data verification device provided by the embodiment of the present application, after determining that the number of records in the verification table in the source database is consistent with the number of records in the verification table in the destination database, and obtaining the first data to be verified from the source database and the second data to be verified from the destination database storing the data migrated from the source database, determines the hash functions corresponding to each row of data in the first data to be verified, so that the first data to be verified can be divided into row set buckets, and then hash values ​​are calculated for the corresponding row data in the divided row set buckets based on each hash function to obtain at least one first row set bucket, and the second data to be verified can be divided into row set buckets, and then hash values ​​are calculated for the corresponding row data in the divided row set buckets based on each hash function to obtain at least one second row set bucket; thereby, data verification can be further quickly performed according to each first row set bucket and each second row set bucket to obtain a verification result. Since the hash function is selected according to the type of data in the database, and the hash value is calculated according to the hash function after the row set buckets are divided, and the multiple row set buckets containing the hash values ​​are verified, the data can be divided in sequence and different hash values ​​can be generated to avoid hash conflicts and matching errors. Even if there is a table without a primary key or unique index, data verification can be completed quickly, thereby improving the efficiency of data verification.

[0085] Figure 2 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 2 As shown, the electronic device may include: a processor (processor) 210, a communication interface (Communications Interface) 220, a memory (memory) 230 and a communication bus 240, wherein the processor 210, the communication interface 220, and the memory 230 communicate with each other through the communication bus 240. The processor 210 can call the logic instructions in the memory 230 to execute the data verification method, which includes: if the number of records in the verification table in the source database is consistent with the number of records in the verification table in the destination database, obtaining the first data to be verified from the source database and obtaining the second data to be verified from the destination database; the destination database stores the data migrated from the source database; determining the hash functions corresponding to each row of data in the first data to be verified; after dividing the first data to be verified into row set buckets, based on each hash function, respectively calculate the hash values ​​of the corresponding row data in the divided row set buckets to obtain at least one first row set bucket; after dividing the second data to be verified into row set buckets, based on each hash function, respectively calculate the hash values ​​of the corresponding row data in the divided row set buckets to obtain at least one second row set bucket; performing data verification according to each of the first row set buckets and each of the second row set buckets to obtain a verification result.

[0086] In addition, the logic instructions in the above-mentioned memory 230 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0087] On the other hand, the present application also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the data verification method provided by the above methods, and the method includes: if the number of records in the verification table in the source database is consistent with the number of records in the verification table in the destination database, then obtaining the first data to be verified from the source database and obtaining the second data to be verified from the destination database; the destination database stores the data migrated from the source database; determining the hash function corresponding to each row of data in the first data to be verified; after dividing the first data to be verified into row set buckets, based on each hash function, respectively calculate the hash value of the corresponding row data in the divided row set buckets, to obtain at least one first row set bucket; after dividing the second data to be verified into row set buckets, based on each hash function, respectively calculate the hash value of the corresponding row data in the divided row set buckets, to obtain at least one second row set bucket; perform data verification according to each of the first row set buckets and each of the second row set buckets to obtain a verification result.

[0088] On the other hand, the present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the data verification method provided by the above-mentioned methods, the method comprising: if the number of records in the verification table in the source database is consistent with the number of records in the verification table in the destination database, obtaining the first data to be verified from the source database and obtaining the second data to be verified from the destination database; the destination database stores the data migrated from the source database; determining the hash functions corresponding to each row of data in the first data to be verified; after dividing the first data to be verified into row set buckets, performing hash value calculations on the corresponding row data in the divided row set buckets based on each hash function to obtain at least one first row set bucket; after dividing the second data to be verified into row set buckets, performing hash value calculations on the corresponding row data in the divided row set buckets based on each hash function to obtain at least one second row set bucket; performing data verification according to each of the first row set buckets and each of the second row set buckets to obtain a verification result.

[0089] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0090] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0091] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A data verification method, characterized in that: The method comprises: If the number of records in the check table in the source database is consistent with the number of records in the check table in the target database, obtaining first data to be checked from the source database and obtaining second data to be checked from the target database; the target database stores data migrated from the source database; Determine the hash functions corresponding to each row of data in the first data to be verified; After dividing the first to-be-checked data into row set buckets, performing hash value calculations on corresponding row data in the divided row set buckets based on respective hash functions to obtain at least one first row set bucket; After dividing the second to-be-checked data into row set buckets, performing hash value calculations on corresponding row data in the divided row set buckets based on each hash function to obtain at least one second row set bucket; Data verification is performed according to each of the first row set buckets and each of the second row set buckets to obtain a verification result.

2. The data verification method according to claim 1, characterized in that: When data verification is performed according to each of the first row set buckets and each of the second row set buckets, the following steps are respectively performed for a row set bucket pair consisting of a first row set bucket and a second row set bucket with the same number: Construct a Bloom filter based on the hash values ​​of the first row bucket in the current row bucket pair; If all hash values ​​of the second row bucket in the current row bucket pair can find the same hash value in the Bloom filter, the data verification of the current row bucket pair is determined to be successful, and the migration data verification is determined to be successful when the data of all row bucket pairs are verified successfully.

3. The data verification method according to claim 2, characterized in that: After constructing a Bloom filter based on the hash values ​​of the first row bucket in the current row bucket pair, it also includes: If any hash value of the second row set bucket in the current row set bucket pair does not have the same hash value in the Bloom filter, the current row set bucket pair is determined as a row set bucket pair that has failed data verification, and the two row set buckets in the row set bucket pair that has failed data verification are divided into row set buckets to form two new row set bucket pairs; If the data verification of any new row set bucket pair fails, the corresponding new row set bucket pair is determined as a row set bucket pair that has failed data verification, and the process returns to the step of dividing the two row set buckets in the row set bucket pair that has failed data verification and forming two new row set bucket pairs, until the data of the two new row set bucket pairs are verified successfully or the number of executions reaches a preset threshold.

4. The data verification method according to claim 3, characterized in that: If the number of executions reaches the preset threshold, the following steps are performed: From the two row set buckets of the row set bucket pair that failed data verification, obtain two row data corresponding to two hash values ​​with the same row number but inconsistent values; Perform cyclic redundancy check on two rows of data; If two rows of data pass the cyclic redundancy check, it is determined that the data verification of the corresponding row set bucket pair is successful; If the two rows of data fail the cyclic redundancy check, it is determined that the migration data verification has failed.

5. The data verification method according to claim 4, characterized in that: The method of obtaining two row data corresponding to two hash values ​​with the same row number but inconsistent values ​​from the two row set buckets of the row set bucket pair that failed data verification includes: The data of the rows with the same number in the two row set buckets of the row set bucket pair that failed data verification are hash mapped respectively through the same hash function to obtain hash pairs with the same number of rows in the row set bucket pair; wherein each hash pair includes the hash values ​​corresponding to the rows with the same number in the two row set buckets of the row set bucket pair that failed data verification; Determine a target hash pair from each hash pair; two hash values ​​in the target hash pair are different; Get two rows of data corresponding to the target hash pair.

6. The data verification method according to claim 1, characterized in that: After dividing the first to-be-checked data into row set buckets, performing hash value calculations on corresponding row data in the divided row set buckets based on respective hash functions to obtain at least one first row set bucket includes: According to the preset bucket depth, each row of data is divided into row set buckets to obtain at least one row set bucket; Based on each hash function, hash value calculations are performed on corresponding row data in each row set bucket to obtain at least one first row set bucket.

7. The data verification method according to any one of claims 1 to 6, characterized in that: The determining of the hash functions corresponding to the respective rows of data in the first data to be verified includes: Input each row of data in the first data to be verified into a hash function matching model respectively to obtain a hash function corresponding to the corresponding row of data output by the hash function matching model; wherein the hash function matching model is obtained by model training based on sample data, data type annotation information of the sample data, and hash function annotation information of the sample data.

8. A data verification device, characterized in that: include: an acquisition module, configured to acquire first data to be verified from the source database and second data to be verified from the destination database if the number of records in the verification table in the source database is consistent with the number of records in the verification table in the destination database; The target database stores data migrated from the source database; A determination module, used to determine the hash functions corresponding to each row of data in the first data to be verified; A first partitioning module, configured to partition the first to-be-checked data into row set buckets, and then respectively calculate hash values ​​of corresponding row data in the partitioned row set buckets based on respective hash functions, to obtain at least one first row set bucket; A second partitioning module, configured to partition the second to-be-checked data into row set buckets, and then calculate hash values ​​of corresponding row data in the partitioned row set buckets based on respective hash functions, to obtain at least one second row set bucket; The verification module is used to perform data verification according to each of the first row set buckets and each of the second row set buckets to obtain a verification result.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the data verification method according to any one of claims 1 to 7 is implemented.

10. A medium, the medium being a non-transitory computer-readable storage medium, on which a computer program is stored, characterized in that: When the computer program is executed by a processor, the data verification method according to any one of claims 1 to 7 is implemented.

11. A product, the product being a computer program product, comprising a computer program, characterized in that: When the computer program is executed by a processor, the data verification method according to any one of claims 1 to 7 is implemented.

Citation Information

Cited By

  • Data consistency verification method and system for remote heterogeneous terminal

    CN120578670A

  • A method and system for data consistency check for remote heterogeneous ends

    CN120578670B

  • Self-adaptive multi-stage filtering and accurate decoding data consistency checking system and self-adaptive multi-stage filtering and accurate decoding data consistency checking method

    CN120723783A

  • System and method for adaptive multi-stage filtering and data consistency check with exact decoding

    CN120723783B

  • Password similarity judgment method and device, electronic equipment and storage medium

    CN121765712A