An intelligent data verification method, device and related equipment

Abstract

The application discloses an intelligent data verification method and device and related equipment, and the method comprises the following steps: acquiring the field type, partition key and partition data size of a target Hive table; generating a field filtering expression based on a rule engine and the field type of the target Hive table; generating a cascaded verification condition based on the partition key of the target Hive table; generating a verification rule based on the field filtering expression and the cascaded verification condition; based on the storage space of a storage block on a distributed file system and the size of each partition data, each partition data is fragmented; and each fragmented data is verified to obtain a target global verification code. In the application, when verifying the Hive data before and after migration, the non-scalar fields in the Hive table are filtered out by setting the field filtering expression, and the tolerance to slight errors during verification is improved; when performing verification, each partition data is fragmented, and each fragmented data is verified in parallel based on multiple computing nodes of a distributed cluster, so that the verification efficiency is improved.

Description

Technical Field

[0001] This application relates to the field of big data, and in particular to an intelligent data verification method, apparatus, and related equipment. Background Technology

[0002] In the field of big data, massive amounts of data are often stored in Hive data warehouses. Hive tables are used to store metadata, while the actual business data is stored in the distributed file system (Hadoop Distributed File System, HDFS). When migrating a Hive data warehouse, it is necessary to verify the Hive data warehouse before and after the migration to ensure data consistency.

[0003] Existing verification methods employ a single-threaded, serial computation of the MD5 (Message-Digest Algorithm 5) hash of the entire Hive database. If the MD5 hashes before and after migration are identical, it indicates that the Hive data warehouse is consistent. However, this method is inefficient for petabyte-scale Hive data warehouses due to its serial computation, resulting in long verification times. Furthermore, verification methods based on the overall Hive data warehouse's MD5 hash are extremely sensitive to changes in metadata. Differences in the representation of Hive tables often lead to verification failures due to minute errors. Summary of the Invention

[0004] The main purpose of this application is to propose an intelligent data verification method, apparatus, and related equipment, which aims to solve the problems of low efficiency and low tolerance for small errors in existing verification methods.

[0005] To achieve the above objectives, this application proposes an intelligent data verification method, comprising:

[0006] Obtain the metadata of the target Hive table, including field types, partition keys, and partition data sizes;

[0007] Based on the preset rule engine and the field types of the target Hive table, a field filtering expression is generated. The field filtering expression is used to filter non-scalar fields in the target Hive table.

[0008] Based on the partition key of the target Hive table, generate cascading verification conditions;

[0009] Generate validation rules based on the field filtering expression and the cascading validation conditions;

[0010] Based on the storage space of the storage blocks on the distributed file system corresponding to the target Hive table, and the size of the data in each partition of the target Hive table, the data in each partition of the target Hive table is sharded to obtain multiple sharded data.

[0011] Based on the verification rules, each of the data segments is verified to obtain the target global verification code.

[0012] In this embodiment of the application, the metadata also includes the number of fields and the order of fields. Before generating the field filtering expression, the intelligent data validation method further includes:

[0013] Determine whether the number of fields, field order, field type, and partition key of the target Hive table are consistent with those of the source Hive table;

[0014] If not, the verification ends and the first verification information is returned. The first verification information is used to indicate the parts of the target Hive table that are inconsistent with the number of fields, field order, field type, and partition key of the target Hive table.

[0015] In this embodiment of the application, before verifying each of the data segments based on the verification rules, the intelligent data verification method further includes:

[0016] Determine whether the total size of the business data corresponding to the target Hive table is consistent with the total size of the business data corresponding to the source Hive table;

[0017] If not, the verification ends and the second verification information is returned. The second verification information is used to indicate that the total size of the business data corresponding to the target Hive table is inconsistent with the total size of the business data corresponding to the source Hive table.

[0018] If they match, the sampling partition is determined based on the partition key, and the business data of a preset proportion in the sampling partition is sampled based on the verification rules to obtain the target sampling verification code.

[0019] Determine whether the target sampling verification code and the source sampling verification code are consistent, wherein the source sampling verification code is obtained by sampling the business data and verification rules of the corresponding partition in the source Hive table;

[0020] If not, the verification ends and a third verification message is returned, indicating that the target sampling verification code and the source sampling verification code are inconsistent.

[0021] In this embodiment of the application, the storage space of the storage blocks on the distributed file system corresponding to the target Hive table, and the size of the data in each partition of the target Hive table, are used to shard the data in each partition of the target Hive table to obtain multiple sharded data, including:

[0022] Determine the size of the data files contained in each partition;

[0023] Data files within the same partition are split and / or merged according to the storage space size of the storage block to obtain multiple data fragments.

[0024] In this embodiment of the application, based on the verification rules, each of the data segments is verified to obtain a global verification code, including:

[0025] Each data shard is sent to each computing node, and each computing node executes the verification rules on each data shard to obtain the data shard verification code corresponding to each data shard. The data shards correspond one-to-one with the computing nodes.

[0026] The target global checksum is obtained based on the checksums of each data segment.

[0027] In this embodiment of the application, the fragmented data verification code is obtained based on the following method:

[0028] The verification rules are executed on each field of the fragmented data to obtain the row-level checksum corresponding to each field.

[0029] Based on the order of each field of the fragmented data, the row-level checksums of the fragmented data are merged to obtain the fragmented data checksum corresponding to the fragmented data.

[0030] In this embodiment of the application, after obtaining the target global checksum, the intelligent data verification method further includes:

[0031] Determine whether the target global checksum and the source global checksum are consistent, wherein the source global checksum is obtained based on the source Hive table and the checksum rules;

[0032] If so, the verification ends and the fourth verification information is returned, which indicates that the target Hive table has been verified without error.

[0033] If not, then the check codes of each partition data contained in the target global check code and the check codes of each partition data contained in the source global check code are compared one by one based on the correspondence of the partition data, and the partition data with different check codes are re-verified.

[0034] This application also proposes an intelligent data verification device, comprising:

[0035] The acquisition module is used to acquire the metadata of the target Hive table, which includes field types, partition keys, and partition data sizes.

[0036] The processing module is used to generate field filtering expressions based on a preset rule engine and the field types of the target Hive table. The field filtering expressions are used to filter non-scalar fields in the target Hive table.

[0037] Based on the partition key of the target Hive table, generate cascading verification conditions;

[0038] Generate validation rules based on the field filtering expression and the cascading validation conditions;

[0039] Based on the storage space of each storage block on the distributed file system where the partition data is located, and the file size of each partition data, the partition data of the target Hive table is sharded to obtain multiple shard data.

[0040] Based on the verification rules, each of the data segments is verified to obtain the target global verification code.

[0041] This application also proposes a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements any of the above embodiments.

[0042] This application also proposes a computing device, including a processor, on which a computer program is stored, and the computer program, when executed, implements any of the above embodiments.

[0043] In this embodiment of the application, when verifying Hive data before and after migration, non-scalar fields in the Hive table can be filtered out by setting field filtering expressions when constructing verification rules, thereby avoiding false positive results and improving the tolerance for small errors during verification. When performing verification, the verification efficiency can be improved by sharding the data in each partition and verifying the data in each partition in parallel based on multiple computing nodes of the distributed cluster. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0045] Figure 1 This is a flowchart illustrating the steps of an intelligent data verification method in one embodiment of this application;

[0046] Figure 2 This is a block diagram of an intelligent data verification device according to an embodiment of this application;

[0047] Figure 3 This is a block diagram of a computer-readable storage medium in one embodiment of this application;

[0048] Figure 4 This is a block diagram of a computing device according to one embodiment of this application.

[0049] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0050] The principles and spirit of this application will now be described with reference to several exemplary embodiments. It should be understood that these embodiments are provided merely to enable those skilled in the art to better understand and implement this application, and are not intended to limit the scope of this application in any way. Rather, these embodiments are provided to make this application more thorough and complete, and to fully convey the scope of this application to those skilled in the art.

[0051] Those skilled in the art will understand that embodiments of this application can be implemented as a system, device, method, or computer program product. Therefore, this application can be specifically implemented in the following forms: entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0052] According to the embodiments of this application, an intelligent data verification method, apparatus and related equipment are proposed.

[0053] In this article, it is important to understand that any number of elements in the accompanying figures is for illustrative purposes and not for limitation, and any naming is for distinction only and has no limiting meaning.

[0054] The principles and spirit of this application will be explained in detail below with reference to several representative embodiments.

[0055] Exemplary methods

[0056] like Figure 1 As shown in the embodiments of this application, the intelligent data verification method includes the following steps S100-S600:

[0057] S100: Obtain the metadata of the target Hive table, including field types, partition keys, and partition data sizes.

[0058] In this embodiment, the Hadoop Distributed File System (HDFS) is a distributed, scalable, and highly fault-tolerant file storage system. The Hive data warehouse is a data warehouse built on top of HDFS. The Hive data warehouse itself does not store data files; the data files are stored on HDFS. The Hive data warehouse is responsible for mapping the data files stored on HDFS to a database table, namely the Hive table. Based on the Hive table, specific data files can be queried from HDFS.

[0059] In this embodiment of the application, the Hive table includes a TBLS table, a COLUMNS_V2 table, and an SDS table. The TBLS table is used to store the ID of the Hive data warehouse (db_id) and the ID of the Hive table (tbl_id). The COLUMNS_V2 table is used to store information such as the field type, field name, field order, number of fields, and partition key contained in each Hive table. The SDS table is used to store the storage path of the data file corresponding to the field in each Hive table in HDFS.

[0060] In this embodiment, the target Hive table is the migrated Hive table that needs to be verified, while the source Hive table is the original Hive table before migration. For example, the data file corresponding to the source Hive table is stored in the source HDFS. After migrating the data file corresponding to the source Hive table to the target HDFS, the target Hive table is mapped. At this time, it is necessary to compare the target Hive table obtained after migration and the data file in the target HDFS with the source Hive table and the data file in the source HDFS to see if they are consistent. If they are consistent, it means that the data file was not damaged or erroneous during the migration process. If they are inconsistent, it means that the data file may have been damaged or erroneous during the migration process.

[0061] Therefore, in step S100, the field types and partition keys of the target Hive table can be determined from the COLUMNS_V2 table of the target Hive table, and the data storage path and size of the partition data of the target Hive table can be obtained based on the SDS table of the target Hive table. Similarly, the field types and partition keys of the source Hive table can be determined from the COLUMNS_V2 table of the source Hive table, and the data storage path and size of the partition data of the source Hive table can be obtained based on the SDS table of the source Hive table.

[0062] S200: Based on the preset rule engine and the field types of the target Hive table, generate a field filtering expression, which is used to filter non-scalar fields in the target Hive table.

[0063] A nonscalar field is a field whose value cannot be represented by a simple, indivisible data. The value of a nonscalar field is itself a composite structure containing multiple elements or attributes. In the embodiments of this application, in the data consistency verification scenario before and after migration, for nonscalar fields, such as an ordered element sequence ARRAY, its serialization output may contain non-substantial differences (such as format differences). In this case, even if the logical semantics of two nonscalar fields are completely equivalent, their physical byte representations may be different. Therefore, when verifying based on the physical bytes of the field, the MD5 verification results corresponding to the two physical bytes (source physical byte and target physical byte) will inevitably be different, leading to false positives.

[0064] In addition, most of the non-scalar fields in Hive tables are auxiliary and descriptive data. Slight differences in these fields do not affect the consistency of the core data. If all such non-scalar fields are validated, on the one hand, a large amount of noise data will be introduced, which will cause false positives. On the other hand, it will also lead to greater resource consumption and longer validation time.

[0065] In this embodiment, a rule engine is a software system that separates business decisions from application code and manages them using predefined rule files. This is achieved by setting up a rule engine and pre-setting non-scalar types, such as ordered element sequences (ARRAY), within it. <t>MAP (Map of Key-Value Pairs)<K, V> Named field sets such as STRUCT<...> are used. Field filtering expressions are generated using the rule engine and the field types of the target Hive table. These expressions are then added when constructing validation rules, creating a whitelist mechanism for field types. This allows ARRAY fields in the target Hive table to be filtered out before validation is performed. <t>MAP<K, V> Non-scalar fields such as STRUCT<...>.

[0066] For example, for any field in the target Hive table, before validation, the whitelist mechanism is used to determine whether the field type is a scalar type. If it is, the validation rules are applied to the field parameter; otherwise, the field is not validated. This avoids false positives caused by non-scalar fields, saves resources, and shortens validation time.

[0067] S300: Generate cascading verification conditions based on the partition key of the target Hive table.

[0068] In this embodiment, the key name and value of the partition key in the metadata of the target Hive table can be read based on the rule engine. Assuming the partition key in the metadata of the target Hive table is (dt, region), and the rule engine reads the partition key's partition value as ('20000101', 'BJ'), cascading validation conditions can be generated:

[0069] "WHERE dt = '20240527'AND region ='BJ'"

[0070] This cascading check condition indicates that when performing a check based on this condition, only data with the time "20000101" and the region "BJ" will be checked.

[0071] S400: Generate validation rules based on the field filtering expression and the cascading validation conditions.

[0072] In this embodiment of the application, when constructing the verification rules, the field filtering expression in step S200 and the cascading verification conditions in step S300 can be combined and encapsulated into an SQL query statement, and the query verification can be performed based on the SQL query statement in the final execution stage.

[0073] In other embodiments, when encapsulating field filtering expressions and cascading validation conditions into SQL query statements, a corresponding template can be selected based on the format of the target Hive table.

[0074] For example, if the target Hive table is Hive 2.x, then a dialect template for Hive 2.x is selected from the rule library of the rule engine for encapsulation; if the target Hive table is Hive 3.x, then a dialect template for Hive 3.x is selected from the rule library of the rule engine for encapsulation.

[0075] In this application embodiment, query statement templates for different Hive versions are preset in the rule template of the rule engine. When constructing SQL query statements, the corresponding template can be selected according to the version of the target Hive table, so that query statements compatible with the target Hive table environment can be dynamically generated.

[0076] S500: Based on the storage space of the storage blocks on the distributed file system corresponding to the target Hive table, and the size of each partition of the target Hive table, the data of each partition of the target Hive table is sharded to obtain multiple sharded data.

[0077] In this embodiment, partitioned data refers to the specific business data corresponding to each partition of the target Hive table on HDFS. For a Hive table, each partition contains a large amount of business data. If a serial verification mechanism is used to verify each partition one by one, the verification time is long and the efficiency is low.

[0078] Therefore, in this embodiment of the application, the partition data corresponding to each partition can be sharded, and each shard data can be verified in parallel based on each computing node in the distributed cluster, thereby improving the verification efficiency.

[0079] In this embodiment of the application, fragmentation can be performed based on the following method:

[0080] Determine the size of the data files contained in each partition;

[0081] Data files within the same partition are split and / or merged according to the storage space size of the storage block to obtain multiple data fragments.

[0082] In this embodiment of the application, the sharding controller can be used to determine the storage path and size of each partition of the target Hive table in HDFS based on the SDS table in the Hive table.

[0083] For example, in this embodiment of the application, assuming that a partition of the target Hive table has three data files: file1, file2, and file3, where file1 is 200MB in size, file2 is 150MB in size, and file3 is 50MB in size, and assuming that the HDFS storage block size corresponding to the target Hive table is 128MB, then, based on the sharding controller, the three data files of the partition data can be divided into the following shards:

[0084] Fragment data 1: file1 (0-128MB)

[0085] Data fragment 2: file1 (129MB-200MB) + file2 (0-56MB)

[0086] Data fragment 3: file2 (57MB-150MB) + file3 (0-35MB)

[0087] Fragment data 4: file3 (36-50MB)

[0088] S600: Based on the verification rules, each of the data segments is verified to obtain the target global verification code.

[0089] In this embodiment, the verification rules are determined in step S400, and the partition data corresponding to each partition is sharded in step S500 to obtain the shard data corresponding to each partition. In step S600, parallel verification can be performed based on each computing node in the distributed cluster. It should be noted that when performing parallel verification based on the distributed cluster, each computing node participating in the computation uses the same verification rules to verify the shard data corresponding to each computing node.

[0090] In this embodiment of the application, based on the verification rules, each of the data segments is verified to obtain the target global checksum, which may include the following steps S610-S620:

[0091] S610: Send each data shard to each computing node. Each computing node executes the verification rule on each data shard to obtain the data shard verification code corresponding to each data shard. The data shards and computing nodes are in one-to-one correspondence.

[0092] Taking file1, file2, and file3 as examples, including fragmented data 1-4, fragmented data 1-4 can be sent to four computing nodes in the same distributed cluster. The four computing nodes verify fragmented data 1-4 respectively. The verification rules are all the verification rules obtained in step S400. After each computing node executes the verification rules on its corresponding fragmented data, it can obtain a fragmented data verification code.

[0093] S620: Obtain the target global checksum based on the checksums of each data segment.

[0094] In this embodiment of the application, in a distributed cluster, a master computing node and sub-computing nodes can be set. After each sub-computing node verifies and obtains the shard data checksum, it can send each shard data checksum to the master computing node. The master computing node then merges the shard data checksums corresponding to each shard data to obtain the final target global checksum.

[0095] In this embodiment of the application, a global checksum can be obtained based on a result aggregator and the MapReduce algorithm. For example, for any computing node, when executing the checksum rules, it can be done in the following way:

[0096] The verification rules are executed on each field of the fragmented data to obtain the row-level checksum corresponding to each field.

[0097] Based on the order of each field of the fragmented data, the row-level checksums of the fragmented data are merged to obtain the fragmented data checksum corresponding to the fragmented data.

[0098] In this embodiment of the application, for any computing node, after obtaining the sharded data, for each row of the sharded data, a filtering expression is first executed. If it is a scalar field, it is verified, such as by performing CRC32 verification or MD5 verification, and the row-level check code corresponding to that row is obtained. The row-level check code is the calculation result corresponding to CRC32 or MD5.

[0099] After all rows have been verified, the row-level checksums of each row are first merged into a local fragment data checksum. Then, the local fragment data checksum is sent to the master compute node. When the master compute node receives the fragment data checksums, it aggregates the fragment data checksums to obtain the target global checksum. This method can reduce the amount of data transmitted between compute nodes and save bandwidth overhead.

[0100] In this embodiment of the application, after obtaining the target global checksum, the intelligent data verification method further includes:

[0101] Determine whether the target global checksum and the source global checksum are consistent, wherein the source global checksum is obtained based on the source Hive table and the checksum rules;

[0102] If so, the verification ends and the fourth verification information is returned, which indicates that the target Hive table has been verified without error.

[0103] If not, then the check codes of each partition data contained in the target global check code and the check codes of each partition data contained in the source global check code are compared one by one based on the correspondence of the partition data, and the partition data with different check codes are re-verified.

[0104] In this embodiment, after obtaining the target global checksum, it can be compared with the source global checksum. If they match, it indicates that the migrated target Hive data is consistent with the original source Hive data. At this point, the fourth checksum is returned, indicating that the Hive data before and after the migration is consistent. It should be noted that the steps for obtaining the source global checksum are the same as those for obtaining the target global checksum. The checksum rules for obtaining the source global checksum based on the source Hive table are also the same as those for obtaining the target global checksum. For example, the filtering expression, cascading checksum conditions, and sharding methods are all the same as those for processing the target Hive table, and will not be elaborated upon here.

[0105] If the target global checksum and the source global checksum are inconsistent, the checksums of each shard data corresponding to the source Hive table and the checksums of each shard data corresponding to the target Hive table can be compared one by one to determine the shard data that are different.

[0106] In addition, in this embodiment of the application, after determining the fragmented data with differences, the fault-tolerant actuator can also re-perform the verification of the fragmented data with differences. When re-performing the verification, the verification model can also be changed. For example, the first verification is based on CRC32 verification, and the second verification can be based on MD5 verification. If the verification codes of the fragmented data in the two verifications are different, it can be determined that the fragmented data has an error during migration. If the verification codes of the fragmented data after the second verification are the same, the verification is correct. After all the erroneous fragmented data has been successfully verified twice, the fourth verification information is returned.

[0107] In this embodiment of the application, when verifying Hive data before and after migration, non-scalar fields in the Hive table can be filtered out by setting field filtering expressions when constructing verification rules, thereby avoiding false positive results and improving the tolerance for small errors during verification. When performing verification, the verification efficiency can be improved by sharding the data in each partition and verifying the data in each partition in parallel based on multiple computing nodes of the distributed cluster.

[0108] In this embodiment of the application, the metadata also includes the number of fields and the order of fields. Before generating the field filtering expression, the intelligent data validation method further includes:

[0109] Determine whether the number of fields, field order, field type, and partition key of the target Hive table are consistent with those of the source Hive table;

[0110] If not, the verification ends and the first verification information is returned. The first verification information is used to indicate the parts of the target Hive table that are inconsistent with the number of fields, field order, field type, and partition key of the target Hive table.

[0111] In this embodiment of the application, in order to avoid wasting computing resources, a pre-verification can be performed first. For example, the number of fields, field order, field type, and partition key of the target Hive table can be obtained, as well as the number of fields, field order, field type, and partition key of the source Hive table. If the number of fields, field order, field type, and partition key of the target Hive table are the same as those of the source Hive table, then the subsequent verification steps can be performed. If they are different, the verification can be terminated and the first verification information can be returned.

[0112] For example, if the number of fields in the target Hive table is different from the number of fields in the source Hive table, the validation will end directly and the first validation information will be returned, indicating that the number of fields in the target Hive table is different from the number of fields in the source Hive table.

[0113] If the number of fields, field order, field type, and partition key of the target Hive table are different from those of the source Hive table, then the target global checksum and the source global checksum obtained from subsequent verifications will inevitably be different. Therefore, by performing pre-verification, under the premise that the number of fields, field order, field type, and partition key of the target Hive table are the same as those of the source Hive table, the subsequent verification process can be performed, which can avoid wasting computing resources.

[0114] In this embodiment of the application, before verifying each of the data segments based on the verification rules, the intelligent data verification method further includes:

[0115] Determine whether the total size of the business data corresponding to the target Hive table is consistent with the total size of the business data corresponding to the source Hive table;

[0116] If not, the verification ends and the second verification information is returned. The second verification information is used to indicate that the total size of the business data corresponding to the target Hive table is inconsistent with the total size of the business data corresponding to the source Hive table.

[0117] If they match, the sampling partition is determined based on the partition key, and the business data of a preset proportion in the sampling partition is sampled based on the verification rules to obtain the target sampling verification code.

[0118] Determine whether the target sampling verification code and the source sampling verification code are consistent, wherein the source sampling verification code is obtained by sampling the business data and verification rules of the corresponding partition in the source Hive table;

[0119] If not, the verification ends and a third verification message is returned, indicating that the target sampling verification code and the source sampling verification code are inconsistent.

[0120] In the embodiments of this application, in addition to the above-mentioned pre-verification method which can avoid wasting computing resources, a progressive verification method can also be adopted to further avoid wasting computing resources.

[0121] For example, after generating the verification rules, first determine whether the total size of the data file (business data) on the HDFS corresponding to the target Hive table is consistent with the total size of the data file (business data) on the HDFS corresponding to the source Hive table. If they are inconsistent, the subsequent target global checksum and source global checksum will necessarily be different. Therefore, the verification can be terminated and the second verification information can be returned. The second verification information can indicate that the total size of the data file on the HDFS corresponding to the target Hive table is inconsistent with the total size of the data file on the HDFS corresponding to the source Hive table.

[0122] If they match, a small-scale data sampling inspection will be conducted.

[0123] For example, during sampling inspection, corresponding partitions of the target Hive table and the source Hive table can be sampled. That is, the sampled partitions of the target Hive table and the sampled partitions of the source Hive table are corresponding. Moreover, after determining the sampled partition, only the first 10% of the business data in the sampled partition can be verified. The same verification rules are executed during the verification, and the source sampling verification code and the target sampling verification code are obtained respectively. If the source sampling verification code and the target sampling verification code are consistent, the subsequent global verification steps are performed. If they are inconsistent, the verification can be terminated and the third verification information can be returned. The third verification information can indicate that the sampling verification has failed.

[0124] In this embodiment of the application, by adopting this progressive verification, the waste of computing resources can be further avoided.

[0125] In this embodiment of the application, when verifying Hive data before and after migration, non-scalar fields in the Hive table can be filtered out by setting field filtering expressions when constructing verification rules, thereby avoiding false positive results and improving the tolerance for small errors during verification. When performing verification, the verification efficiency can be improved by sharding the data in each partition and verifying the data in each partition in parallel based on multiple computing nodes of the distributed cluster.

[0126] Exemplary device

[0127] This application also proposes an intelligent data verification device 100, such as... Figure 2 As shown in this embodiment, the intelligent data verification device includes:

[0128] The acquisition module 110 is used to acquire the metadata of the target Hive table, including field types, partition keys, and partition data sizes.

[0129] Processing module 120 is used to generate field filtering expressions based on a preset rule engine and the field types of the target Hive table. The field filtering expressions are used to filter non-scalar fields in the target Hive table.

[0130] Based on the partition key of the target Hive table, generate cascading verification conditions;

[0131] Generate validation rules based on the field filtering expression and the cascading validation conditions;

[0132] Based on the storage space of the storage blocks on the distributed file system corresponding to the target Hive table, and the size of each partition of the target Hive table, the data of each partition of the target Hive table is sharded to obtain multiple sharded data.

[0133] Based on the verification rules, each of the data segments is verified to obtain the target global verification code.

[0134] In this embodiment of the application, the metadata also includes the number of fields and the order of fields. Before generating the field filtering expression, the processing module 120 is further configured to:

[0135] Determine whether the number of fields, field order, field type, and partition key of the target Hive table are consistent with those of the source Hive table;

[0136] If not, the verification ends and the first verification information is returned. The first verification information is used to indicate the parts of the target Hive table that are inconsistent with the number of fields, field order, field type, and partition key of the target Hive table.

[0137] In this embodiment of the application, before verifying each of the data fragments based on the verification rules, the processing module 120 is further configured to:

[0138] Determine whether the total size of the business data corresponding to the target Hive table is consistent with the total size of the business data corresponding to the source Hive table;

[0139] If not, the verification ends and the second verification information is returned. The second verification information is used to indicate that the total size of the business data corresponding to the target Hive table is inconsistent with the total size of the business data corresponding to the source Hive table.

[0140] If they match, the sampling partition is determined based on the partition key, and the business data of a preset proportion in the sampling partition is sampled based on the verification rules to obtain the target sampling verification code.

[0141] Determine whether the target sampling verification code and the source sampling verification code are consistent, wherein the source sampling verification code is obtained by sampling the business data and verification rules of the corresponding partition in the source Hive table;

[0142] If not, the verification process ends and the third verification information is returned.

[0143] In this embodiment of the application, the processing module 120 is further configured to:

[0144] Determine the size of the data files contained in each partition;

[0145] Data files within the same partition are split and / or merged according to the storage space size of the storage block to obtain multiple data fragments.

[0146] In this embodiment of the application, the processing module is further configured to:

[0147] Each data shard is sent to each computing node, and each computing node executes the verification rules on each data shard to obtain the data shard verification code corresponding to each data shard. The data shards correspond one-to-one with the computing nodes.

[0148] The target global checksum is obtained based on the checksums of each data segment.

[0149] In this embodiment of the application, the processing module 120 is further configured to:

[0150] The verification rules are executed on each field of the fragmented data to obtain the row-level checksum corresponding to each field.

[0151] Based on the order of each field of the fragmented data, the row-level checksums of the fragmented data are merged to obtain the fragmented data checksum corresponding to the fragmented data.

[0152] In this embodiment of the application, the processing module 120 is further configured to:

[0153] Determine whether the target global checksum and the source global checksum are consistent, wherein the source global checksum is obtained based on the source Hive table and the checksum rules;

[0154] If so, the verification ends and the fourth verification information is returned, which indicates that the target Hive table has been verified without error.

[0155] If not, then the check codes of each partition data contained in the target global check code and the check codes of each partition data contained in the source global check code are compared one by one based on the correspondence of the partition data, and the partition data with different check codes are re-verified.

[0156] In this embodiment of the application, when verifying Hive data before and after migration, the processing module 120 can filter out non-scalar fields in the Hive table by setting field filtering expressions when constructing verification rules, thereby avoiding false positive results and improving the tolerance for small errors during verification. When performing verification, the verification efficiency can be improved by sharding the data in each partition and verifying the data in each partition in parallel based on multiple computing nodes of the distributed cluster.

[0157] Exemplary media

[0158] After introducing the methods, media, and systems of exemplary embodiments of this application, the following references are made. Figure 3 The computer-readable storage medium of exemplary embodiments of this application will be described, please refer to... Figure 3 The computer-readable storage medium shown is an optical disc 70, on which a computer program (i.e., a program product) is stored. When the computer program is run by a processor, it performs the steps described in the above method implementation, such as obtaining the metadata of the target Hive table, which includes field types, partition keys, and partition data sizes.

[0159] Based on the preset rule engine and the field types of the target Hive table, a field filtering expression is generated. The field filtering expression is used to filter non-scalar fields in the target Hive table.

[0160] Based on the partition key of the target Hive table, generate cascading verification conditions;

[0161] Generate validation rules based on the field filtering expression and the cascading validation conditions;

[0162] Based on the storage space of the storage blocks on the distributed file system corresponding to the target Hive table, and the size of the data in each partition of the target Hive table, the data in each partition of the target Hive table is sharded to obtain multiple sharded data.

[0163] Based on the aforementioned verification rules, each of the data segments is verified to obtain the target global checksum. The specific implementation methods of each step will not be repeated here. It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory, or other optical and magnetic storage media, which will not be elaborated upon here.

[0164] Exemplary computing device

[0165] After introducing the methods, systems, and media of exemplary embodiments of this application, the following references are made. Figure 4 Computing devices according to exemplary embodiments of this application.

[0166] Figure 4 A block diagram is shown of an exemplary computing device 80 suitable for implementing embodiments of the present application. The computing device 80 may be a computer system or a server. Figure 4 The computing device 80 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0167] like Figure 4 As shown, the components of the computing device 80 may include, but are not limited to: one or more processors or processing units 801, system memory 802, and bus 803 connecting different system components (including system memory 802 and processing unit 801).

[0168] The computing device 80 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the computing device 80, including volatile and non-volatile media, removable and non-removable media.

[0169] System storage 802 may include computer system readable media in the form of volatile storage, such as random access memory (RAM) 8021 and / or cache storage 8022. Computing device 80 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, ROM 8023 may be used to read and write non-removable, non-volatile magnetic media (…). Figure 4 Not shown in the image (usually referred to as "hard drive"). Although not shown in Figure 4 The diagram illustrates that a disk drive for reading and writing to removable non-volatile disks (e.g., "floppy disks") and an optical disk drive for reading and writing to removable non-volatile optical disks (e.g., CD-ROMs, DVD-ROMs, or other optical media) can be provided. In these cases, each drive can be connected to bus 803 via one or more data media interfaces. System storage 802 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of this application.

[0170] A program / utility 8025 having a set (at least one) of program modules 8024 may be stored, for example, in system storage 802, and such program modules 8024 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment. Program modules 8024 typically perform the functions and / or methods described in the embodiments of this application.

[0171] The computing device 80 can also communicate with one or more external devices 804 (such as a keyboard, pointing device, display, etc.). This communication can be performed through the I / O interface 805. Furthermore, the computing device 80 can also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 806. Figure 4 As shown, network adapter 806 communicates with other modules of computing device 80 (such as processing unit 801) via bus 803. It should be understood that, although... Figure 4 As not shown, it can be used in conjunction with computing device 80 with other hardware and / or software modules.

[0172] The processing unit 801 executes various functional applications and data processing by running programs stored in the system storage 802, such as obtaining metadata of the target Hive table, including field types, partition keys, and partition data sizes.

[0173] Based on the preset rule engine and the field types of the target Hive table, a field filtering expression is generated. The field filtering expression is used to filter non-scalar fields in the target Hive table.

[0174] Based on the partition key of the target Hive table, generate cascading verification conditions;

[0175] Generate validation rules based on the field filtering expression and the cascading validation conditions;

[0176] Based on the storage space of the storage blocks on the distributed file system corresponding to the target Hive table, and the size of the data in each partition of the target Hive table, the data in each partition of the target Hive table is sharded to obtain multiple sharded data.

[0177] Based on the aforementioned verification rules, each of the data segments is verified to obtain the target global verification code. The specific implementation methods of each step will not be repeated here. It should be noted that although several units / modules or sub-units / sub-modules of the computing device are mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided and embodied by multiple units / modules.

[0178] In the description of this application, it should be noted that the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0179] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, units, and processes described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0180] In the several embodiments provided in this application, it should be understood that the disclosed systems, methods, and approaches can be implemented in other ways. The system embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the shown or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between systems or units may be electrical, mechanical, or other forms.

[0181] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0182] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0183] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computing device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0184] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The protection scope of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the technical scope disclosed in this application. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be covered within the protection scope of this application. Therefore, the protection scope of this application should be determined by the protection scope of the claims.

[0185] Furthermore, although the operations of the method of this application are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.< / t> < / t>

Claims

1. An intelligent data verification method, comprising: Obtain the metadata of the target Hive table, including field types, partition keys, and partition data sizes; Based on the preset rule engine and the field types of the target Hive table, a field filtering expression is generated. The field filtering expression is used to filter non-scalar fields in the target Hive table. Based on the partition key of the target Hive table, generate cascading verification conditions; Generate validation rules based on the field filtering expression and the cascading validation conditions; Based on the storage space of the storage blocks on the distributed file system corresponding to the target Hive table, and the size of the data in each partition of the target Hive table, the data in each partition of the target Hive table is sharded to obtain multiple sharded data. Based on the verification rules, each of the data segments is verified to obtain the target global verification code. The metadata also includes the number of fields and the field order. Before generating the field filtering expression, the intelligent data validation method also includes: Determine whether the number of fields, field order, field type, and partition key of the target Hive table are consistent with those of the source Hive table; If not, the verification ends and the first verification information is returned. The first verification information is used to indicate the part of the target Hive table that is inconsistent with the number of fields, field order, field type, and partition key of the target Hive table. Before verifying each of the data segments based on the verification rules, the intelligent data verification method further includes: Determine whether the total size of the business data corresponding to the target Hive table is consistent with the total size of the business data corresponding to the source Hive table; If not, the verification ends and the second verification information is returned. The second verification information is used to indicate that the total size of the business data corresponding to the target Hive table is inconsistent with the total size of the business data corresponding to the source Hive table. If they match, the sampling partition is determined based on the partition key, and the business data of a preset proportion in the sampling partition is sampled based on the verification rules to obtain the target sampling verification code. Determine whether the target sampling verification code and the source sampling verification code are consistent, wherein the source sampling verification code is obtained by sampling the business data and verification rules of the corresponding partition in the source Hive table; If not, the verification ends and the third verification information is returned, which indicates that the target sampling verification code and the source sampling verification code are inconsistent. Based on the storage space of the storage blocks on the distributed file system corresponding to the target Hive table, and the size of the data in each partition of the target Hive table, the data in each partition of the target Hive table is sharded to obtain multiple sharded data, including: Determine the size of the data files contained in each partition; Data files within the same partition are split and / or merged according to the storage space size of the storage block to obtain multiple data fragments.

2. The intelligent data verification method as described in claim 1, wherein, Based on the aforementioned verification rules, each of the data segments is verified to obtain a global verification code, including: Each data shard is sent to each computing node, and each computing node executes the verification rules on each data shard to obtain the data shard verification code corresponding to each data shard. The data shards correspond one-to-one with the computing nodes. The target global checksum is obtained based on the checksums of each data segment.

3. The intelligent data verification method as described in claim 2, wherein the fragmented data verification code is obtained based on the following method: The verification rules are executed on each field of the fragmented data to obtain the row-level checksum corresponding to each field. Based on the order of each field of the fragmented data, the row-level checksums of the fragmented data are merged to obtain the fragmented data checksum corresponding to the fragmented data.

4. The intelligent data verification method as described in claim 2, further comprising, after obtaining the target global checksum, the intelligent data verification method: Determine whether the target global checksum and the source global checksum are consistent, wherein the source global checksum is obtained based on the source Hive table and the checksum rules; If so, the verification ends and the fourth verification information is returned, which indicates that the target Hive table has been verified without error. If not, then the check codes of each partition data contained in the target global check code and the check codes of each partition data contained in the source global check code are compared one by one based on the correspondence of the partition data, and the partition data with different check codes are re-verified.

5. An intelligent data verification device, comprising: The acquisition module is used to acquire the metadata of the target Hive table, which includes field types, partition keys, and partition data sizes. The processing module is used to generate field filtering expressions based on a preset rule engine and the field types of the target Hive table. The field filtering expressions are used to filter non-scalar fields in the target Hive table. Based on the partition key of the target Hive table, generate cascading verification conditions; Generate validation rules based on the field filtering expression and the cascading validation conditions; Based on the storage space of each storage block on the distributed file system where the partition data is located, and the file size of each partition data, the partition data of the target Hive table is sharded to obtain multiple shard data. Based on the verification rules, each of the data segments is verified to obtain the target global verification code. The metadata also includes the number of fields and the order of fields. Before generating the field filtering expression, the processing module is also used for: Determine whether the number of fields, field order, field type, and partition key of the target Hive table are consistent with those of the source Hive table; If not, the verification ends and the first verification information is returned. The first verification information is used to indicate the part of the target Hive table that is inconsistent with the number of fields, field order, field type, and partition key of the target Hive table. Before verifying each of the data segments based on the verification rules, the processing module is further configured to: Determine whether the total size of the business data corresponding to the target Hive table is consistent with the total size of the business data corresponding to the source Hive table; If not, the verification ends and the second verification information is returned. The second verification information is used to indicate that the total size of the business data corresponding to the target Hive table is inconsistent with the total size of the business data corresponding to the source Hive table. If they match, the sampling partition is determined based on the partition key, and the business data of a preset proportion in the sampling partition is sampled based on the verification rules to obtain the target sampling verification code. Determine whether the target sampling verification code and the source sampling verification code are consistent, wherein the source sampling verification code is obtained by sampling the business data and verification rules of the corresponding partition in the source Hive table; If not, the verification ends and the third verification information is returned, which indicates that the target sampling verification code and the source sampling verification code are inconsistent. The processing module is further configured to partition the data of each partition of the target Hive table based on the storage space of the storage blocks on the distributed file system corresponding to the target Hive table and the size of the data of each partition of the target Hive table, to obtain multiple partitioned data: Determine the size of the data files contained in each partition; Data files within the same partition are split and / or merged according to the storage space size of the storage block to obtain multiple data fragments.

6. A computer-readable storage medium comprising instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1-4.

7. A computing device comprising a processor having a computer program stored thereon, the computer program executing to implement the method as described in any one of claims 1-4.

Images