Data consistency verification method and device, computer equipment, storage medium and computer program product

Abstract

The invention relates to a data consistency verification method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: acquiring source data and target data, and determining verification demand information; determining to-be-verified data according to the verification demand information, determining a verification item according to the data type of each to-be-verified data, and determining a traversal sequence according to a target data directory where each to-be-verified data is located; traversing the target data directories according to the traversing sequence, determining corresponding to-be-compared data of the to-be-verified data in the source data in the target data directories, and performing comparison processing on the verification values of the verification items corresponding to the to-be-verified data and the verification values of the verification items corresponding to the to-be-compared data to obtain a consistency verification result of the target data directories; and determining a consistency verification result of the target data according to the consistency verification result of each target data directory. By adopting the method, the data verification efficiency can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data consistency verification method, device, computer equipment, storage medium and computer program product. Background Art

[0002] With the vigorous development of the Internet and informatization, various types of enterprise, office, and personal applications have increased, and the types and amounts of data generated have also increased. If complex and massive data are abnormal due to improper migration, the application will not be able to be used normally, so it is particularly important to ensure the integrity and accuracy of the migrated data.

[0003] In the related art, when verifying data integrity, the source data and the migrated target data are usually fully verified in sequence according to the data directory structure, which leads to the problem of low verification efficiency. Summary of the invention

[0004] Based on this, it is necessary to provide a data consistency verification method, device, computer equipment, storage medium and computer program product to address the above technical problems.

[0005] In a first aspect, the present application provides a data consistency verification method. The method comprises:

[0006] Acquire source data and target data, and determine verification requirement information for the target data;

[0007] According to the verification requirement information, determine the data to be verified in the target data, determine the verification items of each data to be verified according to the data type of each data to be verified, and determine the traversal order for each target data directory according to the target data directory where each data to be verified is located;

[0008] Traversing each of the target data directories in the traversal order, determining, for the target data directory currently traversed, the data to be compared corresponding to each of the data to be verified in the target data directory in the source data, and performing a comparison process on the check value of each of the data to be verified corresponding to the check item and the check value of the data to be compared corresponding to the check item, to obtain a consistency check result of the target data directory;

[0009] The consistency check result of the target data is determined according to the consistency check result of each of the target data directories.

[0010] In one of the embodiments, the verification requirement information includes at least one of the following: verification time, verification data type, and verification data ratio.

[0011] In one embodiment, when the verification requirement information includes the verification data ratio, determining the data to be verified in the target data according to the verification requirement information includes:

[0012] Determining initial data to be verified in the target data according to information in the verification requirement information except the verification data ratio;

[0013] From the initial data to be verified, data of the verification data ratio is randomly determined as the data to be verified.

[0014] In one embodiment, before determining the verification requirement information for the target data, the method further includes:

[0015] Obtaining a directory tree of the target data and a directory tree of the source data;

[0016] A comparison process is performed on the directory tree of the target data and the directory tree of the source data, and when the directory tree of the target data is different from the directory tree of the source data, a consistency check result of the target data is determined as an inconsistent result.

[0017] In one embodiment, the verification item includes a hash value of the data to be verified, and before comparing the verification value of each of the data to be verified corresponding to the verification item with the verification value of the data to be compared corresponding to the verification item, the method further includes:

[0018] When the size of the data to be verified is greater than a preset threshold, the data to be verified and the data to be compared corresponding to the data to be verified are respectively processed into slices to obtain slices of the data to be verified and slices of the data to be compared;

[0019] Determine in parallel the hash value of each of the data to be verified slices and the hash value of each of the data to be compared, construct a hash tree of the data to be verified according to the hash value of each of the data to be verified slices, and construct a hash tree of the data to be compared according to the hash value of each of the data to be compared slices;

[0020] The value of the root node of the hash tree of the data to be verified is used as the verification value of the verification item corresponding to the data to be verified, and the value of the root node of the hash tree of the data to be compared is used as the verification value of the verification item corresponding to the data to be verified.

[0021] In one embodiment, the consistency check result of the target data directory includes a comparison result of each pair of the data to be checked and the data to be compared, and the method further includes:

[0022] In the event that any comparison result of the data to be verified and the data to be compared is an inconsistent result, obtaining the path information of the data to be verified, and generating a verification report of the target data directory according to the path information of the data to be verified;

[0023] A verification report for the target data is generated according to the verification report for each of the target data directories.

[0024] In a second aspect, the present application also provides a data consistency verification device. The device comprises:

[0025] A first determination module is used to obtain source data and target data, and determine verification requirement information for the target data;

[0026] A second determination module is used to determine the data to be verified in the target data according to the verification requirement information, determine the verification items of each of the data to be verified according to the data type of each of the data to be verified, and determine the traversal order for each of the target data directories according to the target data directories where each of the data to be verified is located;

[0027] A comparison module is used to traverse each of the target data directories in the traversal order, determine the data to be compared corresponding to each of the data to be verified in the target data directory in the source data for the target data directory currently traversed, compare the check value of each of the data to be verified corresponding to the check item with the check value of the data to be compared corresponding to the check item, and obtain the consistency check result of the target data directory;

[0028] The third determination module is used to determine the consistency verification result of the target data according to the consistency verification results of each of the target data directories.

[0029] In one of the embodiments, the verification requirement information includes at least one of the following: verification time, verification data type, and verification data ratio.

[0030] In one embodiment, when the verification requirement information includes the verification data ratio, the second determination module is further configured to:

[0031] Determining initial data to be verified in the target data according to information in the verification requirement information except the verification data ratio;

[0032] From the initial data to be verified, data of the verification data ratio is randomly determined as the data to be verified.

[0033] In one embodiment, the device further comprises:

[0034] An acquisition module, used for acquiring a directory tree of the target data and a directory tree of the source data;

[0035] The fourth determination module is used to compare the directory tree of the target data and the directory tree of the source data, and when the directory tree of the target data is different from the directory tree of the source data, determine the consistency check result of the target data as an inconsistent result.

[0036] In one embodiment, the verification item includes a hash value of the data to be verified, and the device further includes:

[0037] A slicing module, configured to, when the size of the data to be verified is greater than a preset threshold, slice the data to be verified and the data to be compared corresponding to the data to be verified, respectively, to obtain slices of the data to be verified and slices of the data to be compared;

[0038] A construction module, used to determine the hash value of each of the data to be verified and the hash value of each of the data to be compared in parallel, construct a hash tree of the data to be verified according to the hash value of each of the data to be verified, and construct a hash tree of the data to be compared according to the hash value of each of the data to be compared;

[0039] The processing module is used to use the value of the root node of the hash tree of the data to be verified as the verification value of the verification item corresponding to the data to be verified, and use the value of the root node of the hash tree of the data to be compared as the verification value of the verification item corresponding to the data to be verified.

[0040] In one embodiment, the consistency check result of the target data directory includes a comparison result of each pair of the data to be checked and the data to be compared, and the device further includes:

[0041] A first generating module is used to obtain the path information of the data to be verified and generate a verification report of the target data directory according to the path information of the data to be verified when any comparison result of the data to be verified and the data to be compared is an inconsistent result;

[0042] The second generating module is used to generate a verification report of the target data according to the verification report of each target data directory.

[0043] In a third aspect, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, any one of the above methods is implemented.

[0044] In a fourth aspect, the present application further provides a computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, any of the above methods is implemented.

[0045] In a fifth aspect, the present application further provides a computer program product, wherein the computer program product comprises a computer program, and when the computer program is executed by a processor, any of the above methods is implemented.

[0046] The above-mentioned data consistency verification method, device, computer equipment, storage medium and computer program product determine the items that need to be verified for each data to be verified and the order of traversing each data to be verified according to the verification requirement information, traverse each data to be verified, compare the verification values ​​of the verification items corresponding to the data to be verified, and obtain the consistency verification result of the target data. Therefore, according to actual needs, the data range that needs to be verified in the target data and the order of verifying each data can be dynamically adjusted to improve the verification efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 A schematic diagram of a data consistency verification method in one embodiment;

[0048] Figure 2 A schematic diagram of a process of comparing directory trees in one embodiment;

[0049] Figure 3 A schematic diagram of a process of performing shard comparison on big data in one embodiment;

[0050] Figure 4 A schematic diagram of a data consistency verification method in one embodiment;

[0051] Figure 5 A schematic diagram of a test host in one embodiment;

[0052] Figure 6 is a structural block diagram of a data consistency verification device in one embodiment;

[0053] Figure 7 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0055] In one embodiment, Figure 1As shown, a data consistency verification method is provided. This embodiment takes the method applied to a server as an example. It can be understood that the method can also be applied to a terminal, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0056] Step 102: Acquire source data and target data, and determine verification requirement information for the target data.

[0057] In an embodiment of the present application, the source data is the original data before migration, and the target data is the data after migration. The verification requirement information is used to indicate the requirements for verifying the target data, such as which types of data to verify, which directories to verify, and so on. For example, the verification requirement information may include at least one of the following: verification time, verification data type, and verification data ratio. The verification time refers to the time required to complete the verification of the target data, such as 10 minutes, 1 hour, etc. The verification data type refers to the data type specified for verification, such as business data, metadata, ADS stream files (Alternate Data Streams), Sparse files, etc. The verification data ratio refers to the percentage of the data to be verified that needs to be verified after the data to be verified is determined based on other verification requirement information.

[0058] The verification requirement information can be specified by the user for each target data separately, or it can be determined based on the characteristic information of the target data (including but not limited to the application corresponding to the target data, the size of the target data, and the database where the target data is located). For example, when the size of the target data is large, or the read and write efficiency of the database where the target data is located is low, the verification data ratio can be set lower, or the verification data type can be set to important types of data such as business data to improve the verification efficiency.

[0059] In one embodiment, to improve verification efficiency, a directory tree comparison between the source data and the target data may be performed before determining the verification requirement information. Figure 2 As shown, including:

[0060] Step 202, obtaining a directory tree of target data and a directory tree of source data;

[0061] Step 204 , compare the directory tree of the target data and the directory tree of the source data, and if the directory tree of the target data is different from the directory tree of the source data, determine the consistency check result of the target data as an inconsistent result.

[0062] In the embodiment of the present application, each data directory in the data and the inclusion relationship of each data directory are recorded in the directory tree. Most operating systems provide the function of exporting a directory tree (for example, in a Linux system, the tree command can be used to export a directory tree). Therefore, when obtaining source data and target data, the directory tree of the source data and the directory tree of the target data can be exported at the same time, and the two directory trees can be imported into the server.

[0063] The two directory trees can be traversed in the same way, and for the currently traversed node, the number of child nodes of the node is determined to be consistent to compare whether the two directory trees are consistent. For example, starting from the root nodes of the two directory trees, each node of the two directory trees can be traversed in a depth-first manner. For the currently traversed node, the number of child nodes of the node is first compared. If the number of child nodes is different, it is determined that the directory tree of the target data is different from the directory tree of the source data. If the number of child nodes is the same, continue to traverse the next node until the directory tree traversal is completed.

[0064] If the directory tree of the target data is different from the directory tree of the source data, the consistency check result of the target data can be output as an inconsistent result. If the directory tree of the target data is the same as the directory tree of the source data, the step of determining the verification requirement information for the target data is continued to compare the specific data in the directory.

[0065] Step 104, according to the verification requirement information, determine the data to be verified in the target data, determine the verification items of each data to be verified according to the data type of each data to be verified, and determine the traversal order for each target data directory according to the target data directory where each data to be verified is located.

[0066] In the embodiment of the present application, the data to be verified that actually needs to be verified can be determined in the target data according to the verification requirement information, and then the verification items and the traversal order can be determined according to the data to be verified.

[0067] When the verification requirement information includes verification time, the process of determining the data to be verified can be: sorting each data in the target data from high to low according to the importance of each data to obtain a data importance sequence; determining the expected verification time of each data according to the size of each data, and traversing the data importance sequence sequentially. When the expected verification time corresponding to the currently traversed data and the sum of the expected verification time of each data to be verified is less than or equal to the verification time, the currently traversed data is determined as the data to be verified.

[0068] The importance of each data can be set in advance. For example, the importance of metadata can be set to be greater than that of business data, and the importance of user data in business data can be set to be greater than that of log data. The expected verification time is positively correlated with the size of the data. The precise relationship between the expected verification time and the data size can be obtained in advance through experiments. After arranging the data according to their importance, the more important data can be prioritized as the data to be verified until the sum of the expected verification time of the data to be verified reaches the verification time.

[0069] When the verification requirement information includes the verification data type, the data in the target data whose data type matches the verification data type can be used as the data to be verified. When the verification requirement information includes both the verification duration and the verification data type, after filtering out the data whose data type matches the verification data type, these data can be sorted according to importance according to the method of the aforementioned embodiment, and the more important data can be preferentially used as the data to be verified, which will not be described in detail in the present embodiment of the application.

[0070] When the verification requirement information includes the verification data ratio, the data of the verification data ratio can be randomly determined from the initial data to be verified after the initial data to be verified is determined in the target data according to the information other than the verification data ratio in the verification requirement information as the data to be verified. It should be noted that the random determination method can be to obtain the target data directory to which each initial data to be verified belongs, determine the initial data to be verified contained in each target data directory, and then select the data of the verification data ratio in the initial data to be verified contained in each target data directory as the data to be verified under the target data directory. Alternatively, it can also be to obtain the data type of each initial data to be verified, determine the initial data to be verified corresponding to each data type, and then select the data of the verification data ratio in the initial data to be verified contained in each data type as the data to be verified under the data type. This method can ensure that a certain proportion of data is verified in each target data directory or each data type, so that the distribution of the verified data is more balanced.

[0071] Different data types may require different objects to be compared. For example, for business data with a generally large data volume, the object that needs to be compared may be whether the hash value of the business data is the same as the source data, while for metadata with a generally small data volume, the object that needs to be compared may be whether each byte of the metadata is the same as the metadata. Therefore, different verification items may be set for different data types. Verification items are used to indicate the values ​​that need to be compared when verifying a certain type of data.

[0072] For example, when the data type includes business data and metadata, the verification item corresponding to the business data includes a data hash value; and the verification item corresponding to the metadata includes a data attribute of the business data corresponding to the metadata.

[0073] Among them, the data hash value can be obtained by calculating the data content through any low-collision hash algorithm (such as the md5 algorithm (Message-Digest Algorithm)). Since the data hash value is calculated using a low-collision hash algorithm, the data hash values ​​corresponding to two different data are likely to be different. Therefore, when the data hash value is used to compare the business data, it can be considered that the business data and the corresponding data in the source file are consistent when the data hash values ​​are the same, and the business data and the corresponding data in the source file are inconsistent when the data hash values ​​are different. Using data hash values ​​to compare business data with large data volumes can improve the efficiency of consistency verification.

[0074] The data attributes may include standard attributes of business data. Different standard attributes are shown in Table 1:

[0075] Table 1

[0076]

[0077] In a specific file or operating system, metadata may also include other information. For example, in some operating systems, metadata may also include EA attributes (Extended Attributes), DACL (Discretionary Access Control List), SACL (System Access Control List), file creator, the group to which the file creator belongs, etc. If the metadata includes this information, this information can also be used as a verification item.

[0078] For other data types that need to be verified, verification items can also be defined for these data types. For example, for a backup data stream, the verification item of the backup data stream can be determined as all the data in the backup data stream. For sparse files, the verification item can be determined as the sparse file attributes and ranges. For hard link files, the verification item can be determined as the index node (inode) pointed to by the hard link. For soft link files, the verification item can be determined as the path of the soft link or the reparse point (Reparse Point) attribute.

[0079] Furthermore, the creation time and modification time can also be used as verification items for each data to be verified. The specific selection of values ​​as verification items can be determined by those skilled in the art according to actual needs, and the embodiments of the present application do not make specific limitations.

[0080] It is also possible to obtain the target data directory where each data to be verified is located, and determine the most efficient traversal order for each target data directory. The traversal order can be determined according to the position of each target data directory in the total directory tree of the target data. For example, when the depth of each target data directory in the directory tree is relatively shallow (the depth is the number of directory levels that need to be passed through when accessing the target data directory from the root directory), the traversal order of each target data directory can be determined using the depth-first principle to increase the speed of finding the target data directory, thereby increasing the speed of consistency verification. When the depth of the target data directory in the directory tree is relatively deep, the traversal order of each target data directory can be determined using the breadth-first principle to increase the speed of finding the target data directory, thereby increasing the speed of consistency verification.

[0081] Step 106, traverse each target data directory in the traversal order, determine the data to be compared in the source data corresponding to each data to be verified in the target data directory for the current traversal, compare the verification value of the verification item corresponding to each data to be verified with the verification value of the verification item corresponding to the data to be compared, and obtain the consistency verification result of the target data directory.

[0082] In the embodiment of the present application, after the traversal order is determined, each target data directory can be traversed in the traversal order, and each data to be verified in the target data directory currently traversed can be obtained. According to the name, attribute and other characteristic information of the data to be verified, data in the same target data directory and with the same characteristic information can be determined from the source data as the data to be compared with the data to be verified.

[0083] According to the verification items previously determined for the data to be verified, the verification value of the data to be verified on the verification items and the verification value of the data to be compared on the verification items can be determined respectively. The verification value is the specific numerical value calculated for the verification item. For example, when the verification item is a hash value, the verification value is the specific hash value calculated based on the data to be verified. If the verification values ​​of the data to be verified and the data to be compared are the same, it can be determined that the data to be verified and the data to be compared are consistent. If the verification values ​​of the data to be verified and the data to be compared are different, it can be determined that the data to be verified and the data to be compared are inconsistent.

[0084] In one embodiment, when the verification item includes the hash value of the data to be verified, in order to improve the comparison efficiency of data with a large amount of data, the data may be compared in segments. Figure 3 As shown, including:

[0085] Step 302: When the size of the data to be verified is greater than a preset threshold, the data to be verified and the data to be compared corresponding to the data to be verified are respectively processed into slices to obtain slices of the data to be verified and slices of the data to be compared;

[0086] Step 304, determining the hash value of each data slice to be verified and the hash value of each data to be compared in parallel, constructing a hash tree of the data to be verified according to the hash value of each data slice to be verified, and constructing a hash tree of the data to be compared according to the hash value of each data slice to be compared;

[0087] Step 306, using the value of the root node of the hash tree of the data to be verified as the verification value of the verification item corresponding to the data to be verified, and using the value of the root node of the hash tree of the data to be compared as the verification value of the verification item corresponding to the data to be verified.

[0088] In the embodiment of the present application, the preset threshold value can be determined by those skilled in the art through experiments. For example, a maximum acceptable time consumption when calculating a hash value can be preset, and the maximum data size that can meet the maximum acceptable time consumption can be obtained through experiments, and the maximum data size is used as the preset threshold value.

[0089] If the size of the data to be verified is larger than the preset threshold, the data to be verified needs to be sharded to calculate the hash value of the data to be verified. The same sharding rules are used to shard the data to be verified and the data to be compared, and then multi-threading is used in parallel to calculate the hash values ​​of the shards of the data to be verified and the hash values ​​of the shards of the data to be compared, and then the hash trees of the data to be verified and the hash trees of the data to be compared are constructed according to the hash values ​​of each shard.

[0090] In a hash tree, the value of each node is the sum of the values ​​of each child node of the node, so the value of the root node in the embodiment of the present application is the sum of the hash values ​​of each shard. Since the sharding rule will affect the hash value of each shard, the same sharding rule needs to be used to shard the data to be verified and the data to be compared in the embodiment of the present application. In the case of sharding using the same sharding rule, if the root node values ​​of the hash tree of the data to be verified and the hash tree of the data to be compared are different, it indicates that at least one shard of the data to be verified and the data to be compared is inconsistent. In this case, it can be determined that the data to be verified and the data to be compared are inconsistent, and the inconsistent shards can be determined by checking the hash values ​​of each shard, and the specific information of the shard (such as byte offset) is output, so that the technician can more intuitively determine which part of the data is inconsistent.

[0091] For other data to be verified whose size is less than or equal to the preset threshold, a hash value may be calculated for the entire data to be verified, and the hash value may be compared with the hash value of the data to be compared.

[0092] After comparing all the data to be verified and the data to be compared, the consistency check result of the target data directory can be obtained. The consistency check result of the target data directory may include the comparison results of all the data to be verified and the data to be compared, or it may only indicate whether there is inconsistent data in the target data directory. The server can output the consistency check result of each target data directory after traversing it during the traversal process, so that the technician can see the consistency check result of each target data directory in real time.

[0093] Step 108, determining the consistency check result of the target data according to the consistency check results of each target data directory.

[0094] In the embodiment of the present application, after traversing all target data directories, the consistency check result of the entire target data can be obtained according to the consistency check result of each target data directory. The consistency check result of the target data can indicate all target data directories with inconsistent consistency check results, or it can only indicate whether there is a target data directory with inconsistent consistency check results.

[0095] In one embodiment, the consistency check result of the target data directory includes the comparison result of each pair of data to be checked and data to be compared. After obtaining the consistency check result of the target data, a verification report for the target data can also be generated. Specifically, when any comparison result of the data to be checked and the data to be compared is an inconsistent result, the path information of the data to be checked is obtained, and a verification report of the target data directory is generated according to the path information of the data to be checked; and a verification report of the target data is generated according to the verification reports of each target data directory.

[0096] In the embodiment of the present application, when the data to be compared and the data to be verified are compared one by one, if the comparison result of the data to be verified and the data to be compared is inconsistent, the path information of the data to be verified can be recorded. The path information includes the data directory where the data to be verified is located and the name of the data to be verified.

[0097] After the comparison of each data to be verified in the target data directory is completed, a verification report of the target data directory can be generated, and the verification report includes the path information of all the data to be verified in the target data directory whose comparison results are inconsistent. If there is no data to be verified in the target data directory whose comparison results are inconsistent, the verification report may not be generated, or only indicate in the verification report that the consistency verification result of the target data directory is a consistent result.

[0098] After traversing all target data directories, the verification reports of the target data directories can be combined to obtain the verification report of the target data, that is, the verification report of the target data will include the path information of all the data to be verified with inconsistent comparison results. In addition, the verification report can also include analysis data of each path information, such as which directory has the most data to be verified with inconsistent comparison results, the distribution of the data to be verified with inconsistent comparison results in each directory, etc.

[0099] The data consistency verification method provided in the embodiment of the present application determines the items that need to be verified for each data to be verified according to the verification requirement information, and the order in which each data to be verified is traversed, and each data to be verified is traversed, and the verification values ​​of the verification items corresponding to the data to be verified are compared to obtain the consistency verification result of the target data. Therefore, the data range that needs to be verified in the target data and the order of verifying each data can be dynamically adjusted according to actual needs, thereby improving the verification efficiency.

[0100] In one embodiment, Figure 4 As shown, a data consistency verification method is provided. This method is applied to Figure 5 The test host shown. The test host includes a parameter control module, a directory tree management module, a data verification module, a metadata verification module, and a report generation module, wherein the parameter control module is used to store configuration files, and the configuration files are used to record the verification requirement information input by the tester. The directory tree management module is used to recursively scan the entire directory hierarchy starting from the root directory of the source data and the target data, and cache the directory tree of the source data and the directory tree of the target data obtained after the scan. The data verification module is used to compare and verify the business data in the source data and the business data in the target data. The metadata verification module is used to verify the metadata in the source data and the metadata in the target data. The report generation module is used to output the results of the consistency verification.

[0101] Specifically, the above method includes:

[0102] S1, after the source data is migrated, the source data and target data are mounted to the server through local mounting or network protocol mounting.

[0103] Among them, the SMB (Server Message Block) network protocol can be used for network protocol mounting. The specific mounting method can be determined according to the server, the storage location of the source data, and the storage location of the target data.

[0104] S2, the tester fills in the verification requirement information.

[0105] Testers can configure the expected test time, whether to verify each type of data, and the level of detail of the verification report based on the target data generation and usage scenarios, the data types in the target data, and the target data scale. The configured data is stored in the parameter control module as a parameter configuration file.

[0106] S3, the test host performs a directory tree scan, and scans the directory trees of the source data and the target data through the directory tree management module. During the scanning process, the directory trees are dynamically compared to see if they are consistent. If the directory trees are inconsistent, the consistency check result of the target data is used as an inconsistent result and a check report is output.

[0107] S4, through the data verification module, obtain the data under a single-level directory from top to bottom according to the directory tree hierarchy. Use multiple threads to process multiple directories and multiple data under the directory in parallel. For data with a size greater than the preset threshold, perform sharding, calculate the hash value of each shard to generate a hash tree, and compare the hash value of the root node.

[0108] S5, through the metadata verification module, also according to the directory tree level, obtains the data in a single-level directory from top to bottom and processes them in parallel.

[0109] S6, after the verification is completed, the report generation module can choose whether to print a detailed verification report according to the parameter configuration. The verification report may include the path of the inconsistent data, data blocks, node information of the inconsistent directory, etc.

[0110] The data consistency verification method provided by the embodiment of the present application covers all kinds of metadata verification, covers all practical data, reduces the problem of abnormalities caused by data loss due to insufficient metadata verification, and improves the integrity of data consistency testing. At the same time, directory tree caching, large directory sharding, large file sharding, parallel computing, hash trees and other methods are used to improve the verification speed, and the verification order is optimized to improve the efficiency of data consistency testing.

[0111] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0112] Based on the same inventive concept, the embodiment of the present application also provides a data consistency verification device for implementing the data consistency verification method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more data consistency verification device embodiments provided below can refer to the limitations of the data consistency verification method above, and will not be repeated here.

[0113] In one embodiment, Figure 6 As shown, a data consistency verification device 600 is provided, including: a first determination module 602, a second determination module 604, a comparison module 606, and a third determination module 608, wherein:

[0114] A first determination module 602 is used to obtain source data and target data, and determine verification requirement information for the target data;

[0115] A second determination module 604 is used to determine the data to be verified in the target data according to the verification requirement information, determine the verification items of each of the data to be verified according to the data type of each of the data to be verified, and determine the traversal order for each of the target data directories according to the target data directories where each of the data to be verified is located;

[0116] The comparison module 606 is used to traverse each of the target data directories in the traversal order, determine the data to be compared corresponding to each of the data to be verified in the target data directory in the source data for the target data directory currently traversed, compare the check value of each of the data to be verified corresponding to the check item with the check value of the data to be compared corresponding to the check item, and obtain the consistency check result of the target data directory;

[0117] The third determination module 608 is used to determine the consistency verification result of the target data according to the consistency verification results of each of the target data directories.

[0118] The data consistency verification device provided in the embodiment of the present application determines the items that need to be verified for each data to be verified and the order in which each data to be verified is traversed according to the verification requirement information, traverses each data to be verified, compares the verification values ​​of the verification items corresponding to the data to be verified, and obtains the consistency verification result of the target data. Therefore, the data range that needs to be verified in the target data and the order of verifying each data can be dynamically adjusted according to actual needs, thereby improving the verification efficiency.

[0119] In one of the embodiments, the verification requirement information includes at least one of the following: verification time, verification data type, and verification data ratio.

[0120] In one embodiment, when the verification requirement information includes the verification data ratio, the second determination module 604 is further configured to:

[0121] Determining initial data to be verified in the target data according to information in the verification requirement information except the verification data ratio;

[0122] From the initial data to be verified, data of the verification data ratio is randomly determined as the data to be verified.

[0123] In one embodiment, the device further comprises:

[0124] An acquisition module, used for acquiring a directory tree of the target data and a directory tree of the source data;

[0125] The fourth determination module is used to compare the directory tree of the target data and the directory tree of the source data, and when the directory tree of the target data is different from the directory tree of the source data, determine the consistency check result of the target data as an inconsistent result.

[0126] In one embodiment, the verification item includes a hash value of the data to be verified, and the device further includes:

[0127] A slicing module, configured to, when the size of the data to be verified is greater than a preset threshold, slice the data to be verified and the data to be compared corresponding to the data to be verified, respectively, to obtain slices of the data to be verified and slices of the data to be compared;

[0128] A construction module, used to determine the hash value of each of the data to be verified and the hash value of each of the data to be compared in parallel, construct a hash tree of the data to be verified according to the hash value of each of the data to be verified, and construct a hash tree of the data to be compared according to the hash value of each of the data to be compared;

[0129] The processing module is used to use the value of the root node of the hash tree of the data to be verified as the verification value of the verification item corresponding to the data to be verified, and use the value of the root node of the hash tree of the data to be compared as the verification value of the verification item corresponding to the data to be verified.

[0130] In one embodiment, the consistency check result of the target data directory includes a comparison result of each pair of the data to be checked and the data to be compared, and the device further includes:

[0131] A first generating module is used to obtain the path information of the data to be verified and generate a verification report of the target data directory according to the path information of the data to be verified when any comparison result of the data to be verified and the data to be compared is an inconsistent result;

[0132] The second generating module is used to generate a verification report of the target data according to the verification report of each target data directory.

[0133] Each module in the above device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module above.

[0134] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 7 As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data consistency verification method is implemented.

[0135] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0136] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0137] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0138] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0139] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0140] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0141] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0142] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A data consistency verification method, characterized in that: The method comprises: Acquire source data and target data, and determine verification requirement information for the target data; According to the verification requirement information, determine the data to be verified in the target data, determine the verification items of each data to be verified according to the data type of each data to be verified, and determine the traversal order for each target data directory according to the target data directory where each data to be verified is located; Traversing each of the target data directories in the traversal order, determining, for the target data directory currently traversed, the data to be compared corresponding to each of the data to be verified in the target data directory in the source data, and performing a comparison process on the check value of each of the data to be verified corresponding to the check item and the check value of the data to be compared corresponding to the check item, to obtain a consistency check result of the target data directory; The consistency check result of the target data is determined according to the consistency check result of each of the target data directories.

2. The method according to claim 1, characterized in that The verification requirement information includes at least one of the following: verification time, verification data type, and verification data ratio.

3. The method according to claim 2, characterized in that In a case where the verification requirement information includes the verification data ratio, determining the data to be verified in the target data according to the verification requirement information includes: Determining initial data to be verified in the target data according to information in the verification requirement information except the verification data ratio; From the initial data to be verified, data of the verification data ratio is randomly determined as the data to be verified.

4. The method according to claim 1, characterized in that: Before determining the verification requirement information for the target data, the method further includes: Obtaining a directory tree of the target data and a directory tree of the source data; A comparison process is performed on the directory tree of the target data and the directory tree of the source data, and when the directory tree of the target data is different from the directory tree of the source data, a consistency check result of the target data is determined as an inconsistent result.

5. The method according to claim 1, characterized in that: The verification item includes a hash value of the data to be verified. Before comparing the verification value of each of the data to be verified corresponding to the verification item with the verification value of the data to be compared corresponding to the verification item, the method further includes: When the size of the data to be verified is greater than a preset threshold, the data to be verified and the data to be compared corresponding to the data to be verified are respectively processed into slices to obtain slices of the data to be verified and slices of the data to be compared; Determine in parallel the hash value of each of the data to be verified slices and the hash value of each of the data to be compared, construct a hash tree of the data to be verified according to the hash value of each of the data to be verified slices, and construct a hash tree of the data to be compared according to the hash value of each of the data to be compared slices; The value of the root node of the hash tree of the data to be verified is used as the verification value of the verification item corresponding to the data to be verified, and the value of the root node of the hash tree of the data to be compared is used as the verification value of the verification item corresponding to the data to be verified.

6. The method according to claim 1, characterized in that The consistency check result of the target data directory includes a comparison result of each pair of the data to be checked and the data to be compared, and the method further includes: In the event that any comparison result of the data to be verified and the data to be compared is an inconsistent result, obtaining the path information of the data to be verified, and generating a verification report of the target data directory according to the path information of the data to be verified; A verification report for the target data is generated according to the verification report for each of the target data directories.

7. A data consistency verification device, characterized in that: The device comprises: A first determination module is used to obtain source data and target data, and determine verification requirement information for the target data; A second determination module is used to determine the data to be verified in the target data according to the verification requirement information, determine the verification items of each of the data to be verified according to the data type of each of the data to be verified, and determine the traversal order for each of the target data directories according to the target data directories where each of the data to be verified is located; A comparison module is used to traverse each of the target data directories in the traversal order, determine the data to be compared corresponding to each of the data to be verified in the target data directory in the source data for the target data directory currently traversed, compare the check value of each of the data to be verified corresponding to the check item with the check value of the data to be compared corresponding to the check item, and obtain the consistency check result of the target data directory; The third determination module is used to determine the consistency verification result of the target data according to the consistency verification results of each of the target data directories.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images