Data verification method, device and equipment of database

Abstract

The application relates to a data verification method, device and equipment of a database, and relates to the technical field of databases.The method comprises the following steps: acquiring to-be-verified set information of a database, wherein the to-be-verified set information comprises source set information and target set information; performing hash verification according to the source set information and the target set information to obtain hash verification information; performing verification processing according to a preset verification rule according to the hash verification information to obtain a data verification result corresponding to the to-be-verified set information; and quickly determining the data verification result by performing hash verification on the source set and the target set, thereby improving the overall verification speed and enabling the accuracy verification before and after data migration to be completed in a short time.

Description

Technical Field

[0001] This application relates to the field of database technology, and in particular to a data verification method, apparatus and device for databases. Background Technology

[0002] Currently, with the development of Internet technology, traditional centralized databases have gradually encountered bottlenecks. In particular, databases in the financial and telecommunications industries are gradually migrating to distributed databases with high concurrency and low latency. At this time, how to verify the accuracy of data after migration is particularly important.

[0003] Existing database verification methods typically involve using verification software to retrieve the record counts of the verification table (i.e., the database table) from both the source and destination databases. If the record counts match, the source database's primary key value and the 32-bit Cyclic Redundancy Check (CRC) value of the primary key row are retrieved. Using the source primary key value as a condition, the CRC32 values ​​of the corresponding primary key rows in the destination database are then retrieved and compared with the CRC32 values ​​of the source primary key rows. If the comparison is successful, an error message indicating that the two tables are consistent is displayed; otherwise, an error message indicating that the verification results are inconsistent is displayed. However, existing database verification methods require verifying each row of data one by one, meaning every row in every database table needs to be verified. This results in slow verification speeds, which is unacceptable for industries with large data volumes, such as finance and telecommunications, where shorter verification times are crucial. Summary of the Invention

[0004] This application provides a database data verification method, apparatus, and device to quickly determine the data verification result by performing hash verification on the source set and the target set, thereby improving the overall verification speed and enabling the accuracy verification before and after data migration to be completed in a short time. This solves the problem caused by existing database data verification methods that can only verify each row of data one by one.

[0005] Firstly, this application provides a data verification method for a database, including:

[0006] Obtain the database information of the set to be verified, which includes source set information and target set information;

[0007] Hash verification is performed based on the source set information and the target set information to obtain hash verification information;

[0008] Based on the hash verification information, verification processing is performed according to preset verification rules to obtain the data verification result corresponding to the set information to be verified.

[0009] Optionally, obtaining the database's set of data to be verified includes:

[0010] The data is segmented based on the verification table information of the database to obtain row set information and the corresponding identifier information of the row set information;

[0011] The row set information is processed according to a preset hash generation method to obtain the hash information corresponding to the row set information;

[0012] Based on the row set information, combined with the identifier information and the hash information, a set information to be verified is generated.

[0013] Optionally, the database includes a source database and a destination database, and the segmentation process based on the verification table information of the database to obtain row set information includes:

[0014] Obtain the source verification table of the source database and the target verification table of the destination database respectively;

[0015] The record count of the source verification table and the target verification table is verified according to the preset verification method to obtain the record count verification result.

[0016] If the record count verification result is a consistent record count result, then the source verification table and the target verification table are extracted according to the preset extraction method to obtain the source field row information and the target field row information.

[0017] The row set information is obtained by segmenting the row information based on the source field row information and the target field row information respectively.

[0018] Optionally, the step of segmenting based on the source field row information and the target field row information to obtain the row set information includes:

[0019] Based on the source field row information, the source verification table is segmented according to the preset depth configuration information to obtain at least one source row set information;

[0020] Based on the target field row information, the target verification table is segmented according to the preset depth configuration information to obtain at least one target row set information;

[0021] The source row set information and the target row set information are determined as the row set information.

[0022] Optionally, the step of performing hash verification based on the source set information and the target set information to obtain hash verification information includes:

[0023] Extract source identifier information and source hash information from the source set information, and extract target identifier information and target hash information from the target set information;

[0024] For the source identifier information and the target identifier information, determine whether the source hash information matches the target hash information;

[0025] If the source hash information matches the target hash information, then the hash verification is determined to be successful, and the hash verification success information is determined as the hash verification information;

[0026] If the source hash information does not match the target hash information, the hash verification is determined to have failed, and the hash verification failure information is determined as the hash verification information.

[0027] Optionally, the step of performing verification processing according to the hash verification information and a preset verification rule to obtain the data verification result corresponding to the set information to be verified includes:

[0028] If the hash verification information is a hash verification failure, then the source set information and the target set information are iteratively verified according to the preset verification rules to obtain an abnormal row set, and the number of iterations is recorded.

[0029] Determine whether the number of iterations has reached a preset iteration threshold;

[0030] If the number of iterations reaches a preset iteration threshold, then a cyclic redundancy check is performed based on the abnormal row set to obtain the data check result.

[0031] Optionally, the step of iteratively verifying the source set information and the target set information according to preset verification rules to obtain an abnormal row set includes:

[0032] The source set information and the target set information are re-divided according to the preset verification rules to obtain the source partition set information corresponding to the source set information and the target partition set information corresponding to the target set information.

[0033] Determine whether the source partition set information and the target partition set information match;

[0034] If the source partition set information and the target partition set information do not match, then an abnormal row set is determined based on the source partition set information and / or the target partition set information.

[0035] Optionally, the method further includes:

[0036] If the hash verification information is a hash verification success message, then based on the hash verification success message, the data verification success result is determined as the data verification result.

[0037] Secondly, this application provides a data verification device for a database, comprising:

[0038] The module for obtaining information on the set to be verified is used to obtain information on the set to be verified from the database. The information on the set to be verified includes source set information and target set information.

[0039] The hash verification module is used to perform hash verification based on the source set information and the target set information to obtain hash verification information;

[0040] The verification processing module is used to perform verification processing according to the hash verification information and preset verification rules to obtain the data verification result corresponding to the set information to be verified.

[0041] Thirdly, this application provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0042] Memory, used to store computer programs;

[0043] When a processor executes a program stored in memory, it implements the steps of the database data verification method as described in any embodiment of the first aspect.

[0044] In summary, this application embodiment obtains the database's data set information to be verified, which includes source set information and target set information. Hash verification is performed based on the source set information and target set information to obtain hash verification information. Based on the hash verification information, verification processing is performed according to preset verification rules to obtain the data verification result corresponding to the data set information to be verified. By performing hash verification on the source set and target set, the data verification result is quickly determined, improving the overall verification speed. It can complete the accuracy verification after data migration in a short time, solving the problem caused by existing database data verification methods that can only verify each row of data one by one. Attached Figure Description

[0045] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0046] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 A flowchart illustrating a database data verification method provided in this application embodiment;

[0048] Figure 2 This is a flowchart illustrating the steps of a database data verification method provided in an optional embodiment of this application;

[0049] Figure 3 This is a schematic diagram of a database verification rule provided in an optional embodiment of this application;

[0050] Figure 4 A structural block diagram of a database data verification device provided in this application embodiment;

[0051] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0053] In related technologies, existing database verification methods obtain the record counts of verification tables from the source and destination databases using verification software, and then verify the record counts. If the record counts of the two verification tables match, a CRC32 check is performed on the verification table with a primary key or unique index. However, existing database verification methods cannot verify verification tables without a primary key or unique index. Therefore, existing database verification methods have two problems: ① They require verification row by row, resulting in slow verification speed; ② They cannot verify tables without a primary key / unique index.

[0054] To address the aforementioned issues, this application provides a database data verification method, apparatus, and device. By acquiring the database's data set information to be verified, performing hash verification based on the source and target data set information within the data set information, hash verification information is obtained. Subsequently, based on the hash verification information, verification processing is performed according to preset verification rules to obtain the data verification result corresponding to the data set information to be verified. By performing hash verification on the source and target sets, the data verification result is quickly determined, improving the overall verification speed. This allows for accurate verification before and after data migration in a relatively short time. Furthermore, by utilizing hash verification, data tables without primary keys or unique indexes can be verified, solving the problem caused by existing database data verification methods that can only verify each row of data one by one.

[0055] To facilitate understanding of the embodiments of this application, further explanations and descriptions will be provided below in conjunction with the accompanying drawings and specific embodiments. These embodiments do not constitute a limitation on the embodiments of this application.

[0056] Figure 1 This is a flowchart illustrating a database data verification method provided in an embodiment of this application. Figure 1 As shown, the database data verification method provided in this application embodiment may specifically include the following steps:

[0057] Step 110: Obtain the database information of the set to be verified, which includes source set information and target set information.

[0058] Specifically, the source set information may include a source row set, source identifier information corresponding to the source row set, and source hash information. The source row set may contain row data from tables in the source database, such as one or more rows from a specific table within the source database; this embodiment does not impose any limitations on this. The source identifier information can be used to identify the source row set, and the source hash information can be the hash value corresponding to the source row set; this embodiment does not impose any limitations on this. Similarly, the target set information may include a target row set, target identifier information corresponding to the target row set, and target hash information. The target row set may contain row data from tables in the destination database, such as one or more rows from a specific table within the destination database; this embodiment does not impose any limitations on this. The target identifier information can be used to identify the target row set, and the target hash information can be the hash value corresponding to the target row set; this embodiment does not impose any limitations on this.

[0059] Specifically, in the embodiments of this application, when performing database verification, source set information can be obtained from the source database and target set information can be obtained from the destination database. Then, the source set information and target set information can be used as set information to be verified, so that hash verification can be performed based on the source set information and target set information in the set information to be verified, i.e., step 120 is executed.

[0060] In specific implementations, the source database in this application embodiment can refer to the database before data migration, and the destination database can refer to the database after data migration. The data verification method for the database provided in this application embodiment can be used for data verification after migration across heterogeneous databases, and can also be used for data verification before and after migration of non-heterogeneous databases. For example, the source database and the destination database can be databases with the same structure, such as both the source database and the destination database being distributed databases; or they can be heterogeneous databases, such as the source database being a traditional centralized database and the destination database being a distributed database, to achieve data consistency verification when migrating from a centralized database to a distributed database. This application embodiment does not impose any limitations on this. Specifically, after migrating data from the source database to the destination database, data verification can be performed on the data in the destination database in conjunction with the data in the source database. For example, the data tables in the source database and the data tables in the destination database can be split according to a preset splitting method, dividing the data tables in the source database and the destination database into one or more row sets. Then, a preset hash algorithm can be used to generate a hash value corresponding to each row set, and an identifier corresponding to each row set can be determined. Based on the row set, the hash value corresponding to the row set, and the identifier corresponding to the row set, source set information and target set information are generated so that hash verification can be performed on the source set information and target set information later.

[0061] Step 120: Perform hash verification based on the source set information and the target set information to obtain hash verification information.

[0062] Specifically, hash verification information can include hash verification success information and hash verification failure information, and this application embodiment does not limit this. Specifically, after determining the source set information and the target set information, this application embodiment can perform hash verification based on the source set information and the target set information. For example, hash values ​​and identifiers can be extracted from the source set information and the target set information respectively. It is then determined whether the source set information and hash value with the same identifier are consistent with the hash value of the target set information. If the hash value of the source set information is consistent with the hash value of the target set information, hash verification success information can be generated. Subsequently, based on the hash verification success information, it can be determined that the source set information and the target set information are consistent, that is, the data table corresponding to the source set information is consistent with the data table corresponding to the target set information, thereby determining that the data in the source database is consistent with the data in the destination database. If the hash value of the source set information is inconsistent with the hash value of the target set information, hash verification failure information can be generated.

[0063] Step 130: Based on the hash verification information, perform verification processing according to the preset verification rules to obtain the data verification result corresponding to the set information to be verified.

[0064] Specifically, the data verification result includes successful data verification and failed data verification results, and this application embodiment does not limit this. Specifically, this application embodiment can perform verification processing on the source set information and target set information according to the hash verification information and preset verification rules to obtain the data verification result corresponding to the set information to be verified. By performing hash verification on the source set information and target set information, it is possible to verify the accuracy of the migrated data without having to verify each row of data in the data table, thus improving the verification speed. Furthermore, the hash verification method can be used to verify data tables without primary keys or unique indexes.

[0065] For example, if the hash verification information is a successful hash verification message, it can be determined that the source set information and the target set information are consistent, and then a successful data verification result can be generated. If the hash verification information indicates a hash verification failure, it can be determined that the source set information and the target set information are inconsistent. This could be due to inconsistent rows within the source and target sets. In this case, the source and target sets can be verified according to preset verification rules. For example, the source and target sets can be re-divided, each into half of its original size. The hash verification is then re-executed based on these re-divided sets. The re-divided sets with hash verification failures are then cyclically divided, with a set number of iterations. After reaching the set number of iterations, the failed rows are used as the failed row set. Cyclic redundancy checks are then performed on these failed rows to determine if the source and target rows in the failed row set are consistent. If they are inconsistent, the cyclic redundancy check fails, and the data verification fails. If they are consistent, the cyclic redundancy check succeeds, and the data verification succeeds.

[0066] As can be seen, this embodiment of the application obtains the database's data set information to be verified, which includes source set information and target set information. Hash verification is performed based on the source set information and target set information to obtain hash verification information. Based on the hash verification information, verification processing is performed according to preset verification rules to obtain the data verification result corresponding to the data set information to be verified. By performing hash verification on the source set and target set, the data verification result is quickly determined, improving the overall verification speed. It can complete the accuracy verification after data migration in a short time. Furthermore, by using hash verification, it can verify data tables without primary keys or unique indexes, solving the problem caused by existing database data verification methods that can only verify each row of data one by one.

[0067] Reference Figure 2 This diagram illustrates a step-by-step flowchart of a database data verification method according to an optional embodiment of this application. The database data verification method specifically includes the following steps:

[0068] Step 210: Obtain the database information of the set to be verified, which includes source set information and target set information.

[0069] In specific implementations, both the source set information and the target set information can be row set buckets, which can contain one or more rows of data (wherein, one or more rows of data can originate from a data table). This application embodiment does not impose any restrictions on this. Specifically, this application embodiment can use a hash bucket verification algorithm to sequentially divide the data of the verification tables (which can contain data tables that need to be verified) in the source database and the target database into row set buckets of fixed size. The row set buckets of the source database can serve as the source set information, and the row set buckets of the target database can serve as the target set information. This application embodiment does not impose any restrictions on this.

[0070] Optionally, obtaining the database set information to be verified as described above may specifically include the following sub-steps:

[0071] Sub-step 2101 involves segmenting the data based on the verification table information of the database to obtain row set information.

[0072] Specifically, the verification table information may include the data table to be verified, and this application embodiment does not limit this; the row set information may be a row set bucket, which may contain the segmented data of the data table to be verified, and this application embodiment does not limit this. Specifically, this application embodiment can segment the data table to be verified in the source database and the destination database respectively to obtain the row set bucket corresponding to the source database and the row set bucket corresponding to the destination database.

[0073] In an optional embodiment, when the database includes a source database and a destination database, the step of segmenting based on the verification table information of the database to obtain row set information may specifically include: obtaining the source verification table of the source database and the target verification table of the destination database respectively; verifying the number of records in the source verification table and the target verification table according to a preset verification method to obtain a record count verification result; if the record count verification result is a consistent result, then extracting fields from the source verification table and the target verification table according to a preset extraction method to obtain source field row information and target field row information; and segmenting based on the source field row information and the target field row information to obtain the row set information.

[0074] Specifically, in this embodiment, a source verification table and a target verification table can be obtained from the source database and the target database, respectively. Then, the record count of the source and target verification tables can be verified according to a preset verification method. For example, the record count verification can involve obtaining the number of data rows (i.e., the number of records) of the source verification table and the number of data rows of the target verification table, and determining whether the number of data rows in the source and target verification tables is consistent. That is, determining whether the number of data rows in the source verification table is the same as the number of data rows in the target verification table. If the number of data rows is determined to be consistent, then the record count of the two verification tables is determined to be consistent, and the record count consistency result is obtained as the record count verification result. Subsequently, the record count can be verified according to a preset extraction method. The formula extracts fields from the source and target validation tables respectively. For example, it can determine whether a primary key / unique index key exists in the source and target validation tables. If it is determined that a primary key / unique index key exists in the source and target validation tables, the fields in the source and target validation tables can be sorted according to the primary key / unique index key to obtain the source field row information and the target field row information of the target validation table. The source validation table is then segmented based on the source field row information to obtain the source row set information, and the target validation table is segmented based on the target field information to obtain the target row set information. The source row set information and the target row set information are used as the row set information. If it is determined that neither the source nor the target validation table has a primary key / unique index key, meaning both the source and target validation tables are validation tables without a primary key / unique index key, then a preset field sort or a field sort selected by the developer can be obtained as the source and target field row information. This enables data validation of the validation table without a primary key / unique index key, thus solving the problem caused by the inability of existing database data validation methods to validate tables without a primary key / unique index key.

[0075] In specific implementations, embodiments of this application can preset depth configuration information, which can be used to determine the size of the segmented row set. Specifically, both the source check table and the target check table can contain one or more rows of data. The depth configuration information can be used to determine the amount of segmented row data. For example, for the source check table and the target check table, each row set can be set to 50,000 rows of data. Embodiments of this application do not impose this limitation.

[0076] In one optional embodiment, the present application embodiments segment the row information based on the source field row information and the target field row information respectively to obtain the row set information. Specifically, it may include: segmenting the source verification table according to preset depth configuration information based on the source field row information to obtain at least one source row set information; segmenting the target verification table according to preset depth configuration information based on the target field row information to obtain at least one target row set information; and determining the source row set information and the target row set information as the row set information.

[0077] As an example, refer to Figure 3 A verification program can be pre-set to check the number of records in the verification tables of the source and destination databases. If the number of records is consistent, the verification program can extract data from the source and destination databases according to the primary key / unique index key. If there is no primary key / unique index, the operator can select the sorting and extraction fields (such as fields with high dispersion, such as mobile phone numbers). Then, the depth configuration information can be obtained. The verification program can determine the depth of the row set bucket based on the depth configuration information, such as 50,000 rows per bucket. According to the bucket depth and the field order, the source verification table and the target verification table are divided to obtain the source row set information and the target row set information.

[0078] Sub-step 2102: Process the row set information according to a preset hash generation method to obtain the hash information and the identifier information corresponding to the row set information.

[0079] In specific implementations, embodiments of this application can preset a hash algorithm as a preset hash generation method. After determining the row set information, the row set information can be processed according to the preset hash generation method. For example, the source hash value corresponding to the source row set and the target hash value corresponding to the target row set can be generated according to the hash algorithm, so that the source hash value is used as the source hash information corresponding to the source row set, and the target hash value is used as the target hash information corresponding to the target row set. Source identifier information corresponding to the source row set and target identifier information corresponding to the target row set can also be generated. Both the source identifier information and the target identifier information can be numbers, such as numbers 1 to N (N is an integer), and embodiments of this application do not impose any restrictions on this.

[0080] Sub-step 2103: Based on the row set information, combined with the identifier information and the hash information, generate the set information to be verified.

[0081] Specifically, embodiments of this application can generate verification set information based on row set information, combined with identification information and hash information. For example, source set information can be generated based on source row set information, combined with source identification information and source hash information; target set information can be generated based on target row set information, combined with target identification information and target hash information. Then, the source set information and target set information are used as verification set information. By reasonably dividing the verification table, row sets are obtained, and then hash values ​​corresponding to the row sets are generated using hashing. Subsequently, hash verification can be performed on the entire row set, thereby avoiding the problem of slow verification speed caused by verifying row by row, improving the verification speed, and meeting the requirement of short verification time in cases of small data volume or large data volume.

[0082] As an example, refer to Figure 3 After obtaining the row collection buckets, the verification program can calculate the hash value corresponding to each row collection bucket according to the preset hash generation method. That is, each row collection bucket contains a bucket number, the hash value corresponding to the bucket record, and the row record (e.g., 50,000 rows).

[0083] Step 220: Extract source identifier information and source hash information from the source set information, and extract target identifier information and target hash information from the target set information.

[0084] Specifically, when performing hash verification in this application embodiment, source identifier information and source hash information can be extracted from source set information, and target identifier information and target hash information can be extracted from target set information, so that hash verification can be performed on source set information and target set information with the same identifier information in the future.

[0085] Step 230: For the source identifier information and the target identifier information, determine whether the source hash information matches the target hash information.

[0086] Specifically, in this application embodiment, source identifier information and target identifier information can be compared to determine whether they are consistent. If the source identifier information and target identifier information are consistent, then the source hash information and target hash information can be matched. For example, it can be determined whether the hash value corresponding to the source set information is consistent with the hash value corresponding to the target set information. If the hash value corresponding to the source set information is consistent with the hash value corresponding to the target set information, then the source hash information and target hash information are matched. If the hash value corresponding to the source set information is inconsistent with the hash value corresponding to the target set information, then the source hash information and target hash information are not matched.

[0087] Step 240: If the source hash information matches the target hash information, then the hash verification is determined to be successful, and the hash verification success information is determined as the hash verification information.

[0088] Specifically, in this embodiment of the application, when the source hash information matches the target hash information, the hash verification is determined to be successful. Subsequently, hash verification success information can be generated and the hash verification success information can be determined as hash verification information.

[0089] In the specific implementation, the source hash information matches the target hash information, which can determine that the source set information and the target set information are the same, and the table data corresponding to the source set information and the target set information are consistent. Subsequently, based on the hash verification success information, a data verification success result can be generated as the data verification result, i.e., step 290 is executed.

[0090] Step 250: If the source hash information does not match the target hash information, then the hash verification is determined to have failed, and the hash verification failure information is determined as the hash verification information.

[0091] In the specific implementation, if the source hash information and the target hash information do not match, it means that there are rows of inconsistent data in the source set information and the target set information. There may be one or more rows of inconsistent data, which will cause the hash value corresponding to the source set information to be inconsistent with the hash value corresponding to the target set information during hash generation. It can be determined that the hash verification has failed, and a hash verification failure message can be generated. The hash verification failure message is used as the hash verification information. Subsequently, the source set information and the target set information can be iteratively verified based on the hash verification failure message, so as to find the inconsistent rows of data in the source set information and the target set information, realize the rapid filtering of consistent and inconsistent rows of data, and improve the overall verification speed.

[0092] Step 260: If the hash verification information is a hash verification failure, then the source set information and the target set information are iteratively verified according to the preset verification rules to obtain an abnormal row set, and the number of iterations is recorded.

[0093] Specifically, the number of iterations can be used to stop iterative verification, and this application embodiment does not limit this; the abnormal row set can contain inconsistent row data, and this application embodiment does not limit this. Specifically, this application embodiment can determine that there may be inconsistent row data in the source set information and the target set information when the hash verification information is a hash verification failure information. Iterative verification can be performed on the source set information and the target set information according to preset verification rules. For example, the source row set corresponding to the source set information and the target row set corresponding to the target set information can be re-divided into multiple corresponding row sets. Hash verification can be performed again according to the divided row sets, thereby determining the abnormal row set, and the number of re-divisions can be recorded as the number of iterations.

[0094] In the specific implementation, after re-dividing the source row set and the target row set into multiple corresponding row sets, the resulting source partition set information and target partition set information can be obtained. Subsequently, the source partition set information and target partition set information can be re-hash-verified. Based on the hash verification result, it can be determined whether the re-divided source partition set information and target partition set information are consistent. If they are consistent, it can be determined that the row data in the re-divided source partition set information and target partition set information are consistent. If they are inconsistent, it can be determined that the row data in the re-divided source partition set information and target partition set information are inconsistent, and there are inconsistent row data in the source partition set information and target partition set information. Multiple re-divisions and hash verifications can be performed on the source partition set information and target partition set information with inconsistent row data, thereby effectively identifying inconsistent row data and obtaining an abnormal row set.

[0095] In one optional embodiment, this application embodiment iteratively verifies the source set information and the target set information according to a preset verification rule to obtain an abnormal row set. Specifically, this may include: re-dividing the source set information and the target set information according to the preset verification rule to obtain source partition set information corresponding to the source set information and target partition set information corresponding to the target set information; determining whether the source partition set information and the target partition set information match; if the source partition set information and the target partition set information do not match, then determining the abnormal row set based on the source partition set information and / or the target partition set information. By iteratively verifying the source set information and target set information that fail hash verification, the abnormal row data can be located, and consistent and inconsistent row data can be quickly filtered out, meeting the requirement of a short verification time.

[0096] Step 270: Determine whether the number of iterations has reached a preset iteration threshold.

[0097] In specific implementations, to avoid excessive iteration verification leading to a slowdown in the overall verification speed, embodiments of this application can preset an iteration number threshold. After determining the number of iterations, it can be determined whether the number of iterations has reached the preset iteration number threshold, so that iteration verification can be stopped when the number of iterations reaches the preset iteration number threshold.

[0098] Step 280: If the number of iterations reaches a preset iteration threshold, then perform cyclic redundancy check based on the abnormal row set to obtain the data check result.

[0099] Specifically, in this application embodiment, when the number of iterations reaches a preset iteration threshold, the iterative verification can be stopped. Subsequently, the abnormal row set can be split, and cyclic redundancy verification can be performed on each row of data in the abnormal row set to determine the row data that is inconsistent between the source set information and the target set information, which can be used as the data verification result.

[0100] As an example, after partitioning the row collection buckets, a hash check can be performed on the row collection buckets in the source and destination databases based on their corresponding numbers and hash values. If the hash check is successful, a hash check success message can be generated, and a check consistency result can be returned. If the hash check fails, a hash check failure message can be generated, and a check inconsistency result can be returned. At this point, it can be determined that the row collection bucket check failed. Subsequently, the row collection buckets that failed the check can be repartitioned, which can be done by repartitioning them by half the depth of the row collection bucket, resulting in two row collection buckets, each with its corresponding number and hash value. For example, if both source row bucket A and target row bucket B are row buckets with a depth of 50,000 rows, source row bucket A can be divided into source row bucket A1 and source row bucket A2 with a depth of 25,000 rows, and target row bucket B can be divided into target row bucket B1 and target row bucket B2. Hash values ​​and numbers are generated for source row bucket A1, source row bucket A2, target row bucket B1, and target row bucket B2 respectively. Source and target row buckets with the same number are hash-verified. For source and target row buckets with inconsistent hash verification results, the partitioning and verification process is repeated. If multiple partitioning and verifications fail, the repeated partitioning and verification process is stopped, resulting in a verification failure record. This verification failure record can contain inconsistent row sets, thus effectively identifying inconsistent row data. Subsequently, cyclic redundancy check (CRC) can be performed on the inconsistent row set in the verification failure records. A CRC32 value is calculated for each row in the inconsistent row set to obtain a matching list for the source and destination databases. This matching list contains the matching column value and its corresponding CRC32 value for each row in the inconsistent row set. Then, the matching column value of each row in the source database is used as input conditions to compare the CRC32 values ​​of the source and destination databases sequentially. If the verification comparison for each row is consistent, the data in the verification tables of the source and destination databases is considered consistent, resulting in a successful data verification result. If there are rows with inconsistent verification comparisons, the data in the verification tables of the source and destination databases is considered consistent, resulting in a failed data verification result. The inconsistent rows can then be discarded, thus quickly filtering out consistent and inconsistent rows and improving the overall verification speed.

[0101] Step 290: If the hash verification information is hash verification success information, then based on the hash verification success information, the data verification success result is determined as the data verification result.

[0102] Specifically, in this application embodiment, when the hash verification information is a hash verification success message, it can be determined that there are no inconsistent rows in the source set information and the target set information. Based on the hash verification success message, a data verification success result can be generated and determined as the data verification result. This eliminates the need to perform data verification row by row, improving the data verification speed. Furthermore, it can perform verification on tables without primary keys or unique indexes, solving the problem caused by existing database data verification methods that can only verify each row of data one by one.

[0103] In summary, this application embodiment obtains the set information to be verified from the database, extracts source identifier information and source hash information from the source set information in the set information to be verified, and extracts target identifier information and target hash information from the target set information in the set information to be verified. For the source identifier information and target identifier information, it determines whether the source hash information and target hash information match. When the source hash information and target hash information match, the hash verification is determined to be successful, and the successful hash verification information is determined as the hash verification information. Furthermore, based on the successful hash verification information, the successful data verification result is determined as the data verification result. When the hash verification information is a failed hash verification information, the source set information and target set information are verified according to preset verification rules. Information is iteratively verified to obtain a set of abnormal rows, and the number of iterations is recorded. When the number of iterations reaches a preset threshold, cyclic redundancy verification is performed based on the set of abnormal rows to obtain the data verification result. By performing hash verification on the source set and the target set, the data verification result is quickly determined, improving the overall verification speed. It can complete the accuracy verification after data migration in a short time, and can perform data verification on tables without primary keys / unique indexes. This solves the problem caused by existing database data verification methods that can only verify each row of data one by one, and also solves the problem caused by existing database data verification methods that cannot perform data verification on tables without primary keys / unique indexes.

[0104] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of actions. However, those skilled in the art should know that the embodiments of this application are not limited to the described order of actions, because according to the embodiments of this application, some steps may be performed in other orders or simultaneously.

[0105] like Figure 4 As shown in the figure, this application embodiment also provides a database data verification device 400, including:

[0106] The module 410 for obtaining the set information to be verified is used to obtain the set information to be verified from the database. The set information to be verified includes source set information and target set information.

[0107] Hash verification module 420 is used to perform hash verification based on the source set information and the target set information to obtain hash verification information;

[0108] The verification processing module 430 is used to perform verification processing according to the hash verification information and preset verification rules to obtain the data verification result corresponding to the set information to be verified.

[0109] Optionally, the module 410 for obtaining the set information to be verified includes:

[0110] The segmentation processing submodule is used to perform segmentation processing based on the verification table information of the database to obtain row set information;

[0111] The row set processing submodule is used to process the row set information according to a preset hash generation method to obtain the hash information and the identifier information corresponding to the row set information.

[0112] The submodule for generating information to be verified is used to generate information to be verified based on the row set information, combined with the identifier information and the hash information.

[0113] Optionally, the database includes a source database and a destination database, and the segmentation processing submodule includes:

[0114] The verification table acquisition unit is used to acquire the source verification table of the source database and the target verification table of the destination database, respectively.

[0115] The record count verification unit is used to verify the record count of the source verification table and the target verification table according to a preset verification method, and obtain the record count verification result.

[0116] The field extraction unit is used to extract fields from the source verification table and the target verification table according to a preset extraction method when the record count verification result is a record count consistency result, so as to obtain source field row information and target field row information.

[0117] The segmentation unit is used to segment the row information based on the source field row information and the target field row information respectively to obtain the row set information.

[0118] Optionally, the segmentation unit is specifically used to segment the source verification table according to preset depth configuration information based on the source field row information to obtain at least one source row set information; to segment the target verification table according to preset depth configuration information based on the target field row information to obtain at least one target row set information; and to determine the source row set information and the target row set information as the row set information.

[0119] Optionally, the hash verification module 420 includes:

[0120] The extraction submodule is used to extract source identifier information and source hash information from the source set information, and to extract target identifier information and target hash information from the target set information;

[0121] The judgment submodule is used to determine whether the source hash information matches the target hash information based on the source identifier information and the target identifier information;

[0122] The hash verification information determination submodule is used to determine that the hash verification is successful when the source hash information and the target hash information match, and to determine the hash verification success information as the hash verification information; and to determine that the hash verification fails when the source hash information and the target hash information do not match, and to determine the hash verification failure information as the hash verification information.

[0123] Optionally, the verification processing module 430 includes:

[0124] The iterative verification submodule is used to perform iterative verification on the source set information and the target set information according to a preset verification rule when the hash verification information is a hash verification failure information, to obtain an abnormal row set and record the number of iterations;

[0125] The judgment submodule is used to determine whether the number of iterations has reached a preset iteration number threshold;

[0126] The cyclic redundancy check submodule is used to perform cyclic redundancy check based on the abnormal row set when the number of iterations reaches a preset iteration number threshold, and obtain the data check result.

[0127] Optionally, the iterative verification submodule is specifically used for: re-dividing the source set information and the target set information according to preset verification rules to obtain source partition set information corresponding to the source set information and target partition set information corresponding to the target set information; determining whether the source partition set information and the target partition set information match; if the source partition set information and the target partition set information do not match, then determining an abnormal row set based on the source partition set information and / or the target partition set information.

[0128] Optionally, the data verification device 400 for the database further includes:

[0129] The data verification success result determination module is used to determine the data verification success result as the data verification result based on the hash verification success information when the hash verification information is hash verification success information.

[0130] It should be noted that the database data verification device provided in this application embodiment can execute the database data verification method provided in any embodiment of this application, and has the corresponding functions and beneficial effects of executing the database data verification method.

[0131] In a specific implementation, the data verification device for the aforementioned database can be integrated into the device, enabling the device to perform hash verification based on the information of the set to be verified, and obtain the data verification result. As an electronic device, it can achieve data verification for data tables without primary keys or unique indexes without performing data verification row by row. The electronic device can be composed of two or more physical entities, or it can be composed of a single physical entity. For example, the electronic device can be a personal computer (PC), a computer, a server, etc. This application embodiment does not impose specific limitations in this regard.

[0132] like Figure 5 As shown, this application embodiment provides an electronic device including a processor 111, a communication interface 112, a memory 113, and a communication bus 114. The processor 111, communication interface 112, and memory 113 communicate with each other via the communication bus 114. The memory 113 stores computer programs. The processor 111, when executing the program stored in the memory 113, implements the steps of the database data verification method provided in any of the aforementioned method embodiments. For example, the steps of the database data verification method may include the following: obtaining database set information to be verified, the database set information including source set information and target set information; performing hash verification based on the source set information and the target set information to obtain hash verification information; and performing verification processing according to the hash verification information and preset verification rules to obtain the data verification result corresponding to the database set information to be verified.

[0133] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the database data verification method provided in any of the foregoing method embodiments.

[0134] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0135] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A data verification method for a database, characterized in that, include: Obtain the database information of the set to be verified, which includes source set information and target set information; Hash verification is performed based on the source set information and the target set information to obtain hash verification information; Based on the hash verification information, verification processing is performed according to preset verification rules to obtain the data verification result corresponding to the set information to be verified; The step of obtaining the database's set of information to be verified includes: Based on the verification table information of the database, the data is segmented to obtain row set information; The row set information is processed according to a preset hash generation method to obtain the hash information and the identifier information corresponding to the row set information; Based on the row set information, combined with the identifier information and the hash information, a set of information to be verified is generated; The step of performing verification processing according to the hash verification information and preset verification rules to obtain the data verification result corresponding to the set information to be verified includes: If the hash verification information is a hash verification failure, then the source set information and the target set information are iteratively verified according to the preset verification rules to obtain an abnormal row set, and the number of iterations is recorded. Determine whether the number of iterations has reached a preset iteration threshold; If the number of iterations reaches a preset iteration threshold, then a cyclic redundancy check is performed based on the abnormal row set to obtain the data check result.

2. The method according to claim 1, characterized in that, The database includes a source database and a destination database. The process of segmenting based on the verification table information of the database to obtain row set information includes: Obtain the source verification table of the source database and the target verification table of the destination database respectively; The record count of the source verification table and the target verification table is verified according to the preset verification method to obtain the record count verification result. If the record count verification result is a consistent record count result, then the source verification table and the target verification table are extracted according to the preset extraction method to obtain the source field row information and the target field row information. The row set information is obtained by segmenting the row information based on the source field row information and the target field row information respectively.

3. The method according to claim 2, characterized in that, The process of segmenting based on the source field row information and the target field row information to obtain the row set information includes: Based on the source field row information, the source verification table is segmented according to the preset depth configuration information to obtain at least one source row set information; Based on the target field row information, the target verification table is segmented according to the preset depth configuration information to obtain at least one target row set information; The source row set information and the target row set information are determined as the row set information.

4. The method according to claim 1, characterized in that, The step of performing hash verification based on the source set information and the target set information to obtain hash verification information includes: Extract source identifier information and source hash information from the source set information, and extract target identifier information and target hash information from the target set information; For the source identifier information and the target identifier information, determine whether the source hash information matches the target hash information; If the source hash information matches the target hash information, then the hash verification is determined to be successful, and the hash verification success information is determined as the hash verification information; If the source hash information does not match the target hash information, the hash verification is determined to have failed, and the hash verification failure information is determined as the hash verification information.

5. The method according to claim 1, characterized in that, The step of iteratively verifying the source set information and the target set information according to preset verification rules to obtain an abnormal row set includes: The source set information and the target set information are re-divided according to the preset verification rules to obtain the source partition set information corresponding to the source set information and the target partition set information corresponding to the target set information. Determine whether the source partition set information and the target partition set information match; If the source partition set information and the target partition set information do not match, then an abnormal row set is determined based on the source partition set information and / or the target partition set information.

6. The method according to claim 1, characterized in that, Also includes: If the hash verification information is a hash verification success message, then based on the hash verification success message, the data verification success result is determined as the data verification result.

7. A data verification device for a database, characterized in that, include: The module for obtaining information on the set to be verified is used to obtain information on the set to be verified from the database. The information on the set to be verified includes source set information and target set information. The hash verification module is used to perform hash verification based on the source set information and the target set information to obtain hash verification information; The verification processing module is used to perform verification processing according to the hash verification information and preset verification rules to obtain the data verification result corresponding to the set information to be verified. The step of obtaining the database's set of information to be verified includes: Based on the verification table information of the database, the data is segmented to obtain row set information; The row set information is processed according to a preset hash generation method to obtain the hash information and the identifier information corresponding to the row set information; Based on the row set information, combined with the identifier information and the hash information, a set of information to be verified is generated; The step of performing verification processing according to the hash verification information and preset verification rules to obtain the data verification result corresponding to the set information to be verified includes: If the hash verification information is a hash verification failure, then the source set information and the target set information are iteratively verified according to the preset verification rules to obtain an abnormal row set, and the number of iterations is recorded. Determine whether the number of iterations has reached a preset iteration threshold; If the number of iterations reaches a preset iteration threshold, then a cyclic redundancy check is performed based on the abnormal row set to obtain the data check result.

8. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the steps of the data verification method for the database as described in any one of claims 1-6.

Images