Data verification method and device, computer equipment and readable storage medium

Abstract

The invention relates to a data verification method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: determining first data in a source database and second data in a target database; field discreteness analysis is conducted on the first data, a discreteness analysis result is obtained, and the discreteness analysis result comprises a variable coefficient used for measuring the data discreteness degree, a unique value proportion used for measuring the data sparseness degree and an information entropy used for measuring the data complexity degree; determining a data sampling mode based on the variable coefficient, the unique value proportion and the information entropy; sampling from the first data according to the data sampling mode to obtain first sampling data, and sampling from the second data according to the data sampling mode to obtain second sampling data; and performing comparison verification on the first sampling data and the second sampling data to obtain a migration data verification result of the first data and the second data. By adopting the method, the accuracy of data verification can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data verification method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] In the field of information technology, data migration is a crucial process that involves transferring large amounts of data from one storage system, database, or application to another. With the continuous expansion of corporate business and the continuous advancement of technology, data migration has become a common requirement in scenarios such as information system upgrades, mergers and integrations, and cloud computing migration. After the data migration is completed, it is crucial to verify the correctness and integrity of the data, but the accuracy of data verification after data migration is currently low. Summary of the invention

[0003] Based on this, it is necessary to provide a data verification method, device, computer equipment, computer-readable storage medium and computer program product that can improve the accuracy of data verification in response to the above technical problems.

[0004] In a first aspect, the present application provides a data verification method, comprising:

[0005] Determine first data in a source database and second data in a target database, wherein the second data is obtained by migrating the first data from the source database to the target database;

[0006] Performing a field discreteness analysis on the first data to obtain a discreteness analysis result, wherein the discreteness analysis result includes a coefficient of variation for measuring the degree of data discreteness, a unique value ratio for measuring the degree of data sparsity, and an information entropy for measuring the degree of data complexity;

[0007] Determine the data sampling method based on the coefficient of variation, unique value ratio and information entropy;

[0008] Sampling the first data according to the data sampling method to obtain first sampled data, and sampling the second data according to the data sampling method to obtain second sampled data;

[0009] The first sampling data and the second sampling data are compared and verified to obtain migration data verification results of the first data and the second data.

[0010] In a second aspect, the present application also provides a data verification device, comprising:

[0011] A data determination module, used to determine first data in a source database and second data in a target database, wherein the second data is obtained by migrating the first data from the source database to the target database;

[0012] A discreteness analysis module, used for performing field discreteness analysis on the first data to obtain discreteness analysis results, wherein the discreteness analysis results include a coefficient of variation for measuring the degree of data discreteness, a unique value ratio for measuring the degree of data sparsity, and an information entropy for measuring the degree of data complexity;

[0013] A sampling method determination module is used to determine the data sampling method based on the coefficient of variation, the unique value ratio and the information entropy;

[0014] A data sampling module, used for sampling the first data according to the data sampling method to obtain the first sampled data, and sampling the second data according to the data sampling method to obtain the second sampled data;

[0015] The data comparison module is used to compare and verify the first sampling data and the second sampling data to obtain the migration data verification result of the first data and the second data.

[0016] In a third aspect, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above data verification method when executing the computer program.

[0017] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and the computer program implements the steps of the above data verification method when executed by a processor.

[0018] In a fifth aspect, the present application further provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the above data verification method are implemented.

[0019] The above-mentioned data verification method, device, computer equipment, computer-readable storage medium and computer program product, for the first data in the source database of data migration and the second data in the target database, perform field discreteness analysis on the first data, obtain discreteness analysis results including coefficient of variation, unique value ratio and information entropy, determine the data sampling method based on the coefficient of variation, unique value ratio and information entropy, collect the first sampling data and the second sampling data from the first data and the second data respectively according to the data sampling method, and obtain the migration data verification result of the first data and the second data by comparing and verifying the first sampling data and the second sampling data. The data sampling method is determined by the coefficient of variation, unique value ratio and information entropy in the discreteness analysis results, and the first data in the source database and the second data in the target database are sampled respectively according to the data sampling method, and the migration data verification is realized by comparing and verifying the data obtained by sampling. The corresponding data sampling method can be determined according to the discrete characteristics of the data in the data migration to obtain the sampling data for data verification, thereby improving the accuracy of data verification. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 An application environment diagram of a data verification method in an embodiment;

[0022] Figure 2 A schematic diagram of a data verification method in one embodiment;

[0023] Figure 3 A schematic diagram of a discreteness analysis process in one embodiment;

[0024] Figure 4 A schematic diagram of a process of data migration and verification processing in one embodiment;

[0025] Figure 5 A schematic diagram of a flow chart of a data verification method in another embodiment;

[0026] Figure 6 is a structural block diagram of a data verification device in one embodiment;

[0027] Figure 7 is an internal structure diagram of a computer device in one embodiment;

[0028] Figure 8FIG. 4 is a diagram showing the internal structure of a computer device in yet another embodiment. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0030] The data verification method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown in the figure, the terminal 102 communicates with the server 104 through the network, and the data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The server 104 can communicate with the source database and the target database through the network respectively to migrate the data in the source database to the target database.

[0031] The user can trigger the data transfer process between the source database and the target database through the terminal 102, such as sending a data transfer request to the server 104, so that the server 104 transfers the data in the source database to the target database according to the data transfer request. After the data in the source database is transferred to the target database, for example, the server 104 transfers the first data in the source database to the target database, and the target database includes the second data corresponding to the first data. The server 104 can perform field discreteness analysis on the first data to obtain discreteness analysis results, and the discreteness analysis results include the coefficient of variation for measuring the degree of data discreteness, the unique value ratio for measuring the degree of data sparsity, and the information entropy for measuring the degree of data complexity. The server 104 can determine the data sampling method based on the coefficient of variation, the unique value ratio and the information entropy, and sample the first data and the second data respectively according to the data sampling method to obtain the first sampled data and the second sampled data. The server 104 can compare and verify the first sampled data and the second sampled data, so as to obtain the migration data verification results of the first data and the second data.

[0032] In some embodiments, the data verification method can also be implemented separately by the terminal 102 or the server 104, such as the terminal 102 or the server 104 can directly perform data verification on the first data in the source database and the second data in the target database. Among them, the terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The head-mounted devices can be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. The server 104 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services.

[0033] In an exemplary embodiment, Figure 2 As shown, a data verification method is provided, which is applied to Figure 1 The server in the example is used to illustrate, including the following steps 202 to 210. Among them:

[0034] Step 202 , determining first data in a source database and second data in a target database, wherein the second data is obtained by migrating the first data from the source database to the target database.

[0035] Among them, data migration is the process of transferring data between different systems, platforms or databases. When data migration is performed between different databases, the source database is the database from which data is transferred, and the target database is the database to which data is transferred. Specifically, data migration is to migrate the data in the source database to the target database. The first data is the data transferred out of the source database, and the second data is the data transferred into the target database. During the data migration process, if the data is transferred accurately, the first data transferred out of the source database and the data transferred into the target database are the same.

[0036] Exemplarily, when performing data transfer processing between a source database and a target database, the server may determine first data transferred out of the source database and second data transferred into the target database. In some embodiments, the server may receive a data verification request sent by a terminal, and determine the first data from the source database and the second data from the target database according to the data verification request.

[0037] Step 204, performing field discreteness analysis on the first data to obtain discreteness analysis results, wherein the discreteness analysis results include a coefficient of variation for measuring the degree of data discreteness, a unique value ratio for measuring the degree of data sparsity, and information entropy for measuring the degree of data complexity.

[0038] The discreteness analysis result is used to characterize the data discreteness of the first data, and is obtained by performing field discreteness analysis on the data of each field in the first data. The discreteness analysis result includes the coefficient of variation, the ratio of unique values, and the information entropy, and the discreteness characteristics of the first data can be comprehensively reflected through the coefficient of variation, the ratio of unique values, and the information entropy. The coefficient of variation is used to measure the degree of discreteness of the first data, the ratio of unique values ​​is used to measure the sparsity of the first data, and the information entropy is used to measure the complexity of the first data.

[0039] Exemplarily, the server may perform field discreteness analysis on the first data, such as performing discreteness analysis, sparsity analysis, and complexity analysis on the first data, thereby obtaining the coefficient of variation, the ratio of unique values, and information entropy, respectively, and obtaining discreteness analysis results based on the coefficient of variation, the ratio of unique values, and the information entropy. In some embodiments, the first data may include different types of data, and the discreteness analysis results corresponding to different types of data may include different measurement parameters, such as for numerical data, the corresponding discreteness analysis results may include the coefficient of variation and information entropy, but not the ratio of unique values.

[0040] Step 206, determining the data sampling method based on the coefficient of variation, the unique value ratio and the information entropy.

[0041] Among them, the data sampling method is a method for sampling the first data and the second data, such as at least one of systematic sampling or random sampling. Optionally, the server can comprehensively determine the data sampling method based on the coefficient of variation, the ratio of unique values ​​and the information entropy. For example, the server can construct a discrete feature based on the coefficient of variation, the ratio of unique values ​​and the information entropy, and determine the corresponding data sampling method through the discrete feature. In some embodiments, the server can pre-set corresponding weights for the coefficient of variation, the ratio of unique values ​​and the information entropy, respectively, and the server can perform weighted fusion based on the coefficient of variation, the ratio of unique values, the information entropy and the respective weights to construct a discrete feature of the first data.

[0042] Step 208: sampling the first data according to the data sampling method to obtain first sampled data, and sampling the second data according to the data sampling method to obtain second sampled data.

[0043] Exemplarily, the server may sample the first data and the second data respectively according to the data sampling method to obtain the first sampled data and the second sampled data respectively. In some embodiments, the server may determine the amount of sampled data, and sample the first data and the second data respectively according to the sampled data amount and the data sampling method to obtain the first sampled data and the second sampled data.

[0044] Step 210 , comparing and verifying the first sampling data and the second sampling data to obtain migration data verification results of the first data and the second data.

[0045] Among them, the migration data verification result is used to characterize the consistency between the first data in the source database and the second data in the target database. Optionally, the server can compare and verify the first sampling data and the second sampling data, such as the server can compare the data belonging to the same field in the first sampling data and the second sampling data one by one to determine whether the data belonging to the same field in the first sampling data and the second sampling data are consistent, thereby determining whether the data in the data migration process is consistent, and obtaining the migration data verification result. In some embodiments, after the server compares and verifies the first sampling data and the second sampling data, it can determine the data difference between the first sampling data and the second sampling data, and the server can generate a corresponding data verification report based on the data difference. The data verification report can record the data with differences between the first sampling data and the second sampling data, and the server can obtain the migration data verification result based on the data verification report.

[0046] In the above data verification method, for the first data in the source database of the data migration and the second data in the target database, a field discreteness analysis is performed on the first data to obtain discreteness analysis results including coefficient of variation, unique value ratio and information entropy, and a data sampling method is determined based on the coefficient of variation, unique value ratio and information entropy, and first sampling data and second sampling data are respectively collected from the first data and the second data according to the data sampling method, and the migration data verification results of the first data and the second data are obtained by comparing and verifying the first sampling data and the second sampling data. The data sampling method is determined by the coefficient of variation, unique value ratio and information entropy in the discreteness analysis results, and the first data in the source database and the second data in the target database are sampled respectively according to the data sampling method, and the migration data verification is realized by comparing and verifying the data obtained by sampling, and the corresponding data sampling method can be determined according to the discrete characteristics of the data in the data migration to obtain the sampling data for data verification, thereby improving the accuracy of data verification.

[0047] In an exemplary embodiment, Figure 3As shown, the discreteness analysis process, i.e., performing field discreteness analysis on the first data to obtain discreteness analysis results, includes steps 302 to 310. Among them:

[0048] Step 302: determining analysis sample data from the first data, where the analysis sample data includes numerical field data and character field data.

[0049] The analysis sample data is data determined from the first data for field discreteness analysis, and the data volume of the analysis sample data can be determined according to the data volume of the first data. Numerical field data is numerical data, such as age 30, salary 2000, assessment score 100 points, etc.; character field data is character data, such as ID number, name, ethnicity, nationality, resume description, etc.

[0050] Optionally, the server may sample the first data to obtain analysis sample data for field discreteness analysis from the first data. The analysis sample data may include numerical field data and character field data. Different types of data may be processed using different field discreteness analysis methods.

[0051] Step 304: Obtain the coefficient of variation of the first data according to the numerical statistical parameters of the numerical field data.

[0052] Exemplarily, for the numerical field data, the server may perform numerical statistics, such as statistically calculating parameters such as the mean value, standard deviation, and variance, to obtain corresponding numerical statistical parameters, and the server may construct the coefficient of variation of the first data based on the numerical statistical parameters. For example, the server may obtain the coefficient of variation of the first data based on the ratio between the standard deviation and the mean of the numerical field data.

[0053] Step 306, obtaining a unique value ratio of the first data according to a unique value statistical parameter of the character field data.

[0054] Exemplarily, for character field data, the server may count the number of unique values, where a unique value is data that is unique to the character field data, such as data with different specific data content in the character field data. The server may obtain the unique value ratio of the first data based on the unique value statistical parameter, such as the unique value statistical parameter may include the number of unique values, and the server may obtain the unique value ratio of the first data based on the ratio between the number of unique values ​​and the data volume of the character field data.

[0055] Step 308: Obtain information entropy of the first data by analyzing the value probability of each field data in the sample data.

[0056] The value probability is used to characterize the probability of a certain value appearing in the field data. Optionally, the server can calculate the corresponding value probability for each field data in the analysis sample data, and obtain the information entropy of the first data based on the value probability calculation.

[0057] Step 310, obtaining a discreteness analysis result based on the coefficient of variation, the unique value ratio and the information entropy.

[0058] Optionally, the server may combine the coefficient of variation, the unique value ratio and the information entropy to obtain the discreteness analysis result. For example, the server may combine the coefficient of variation, the unique value ratio and the information entropy to obtain the discreteness analysis result of the first data.

[0059] In this embodiment, the server determines the numerical field data and the character field data from the first data, obtains the coefficient of variation based on the numerical statistical parameters of the numerical field data, obtains the unique value ratio based on the unique value statistical parameters of the character field data, and obtains the information entropy based on the value probability of each field data in the analysis sample data, so that field discreteness analysis can be performed according to the types of various analysis sample data, and the accuracy of the field discreteness analysis can be ensured, which is conducive to improving the accuracy of data verification based on the discreteness analysis results.

[0060] In an exemplary embodiment, determining analysis sample data from first data includes: determining the data volume of the first data; determining the sample data volume based on the data volume of the first data; and determining analysis sample data from the first data according to the sample data volume.

[0061] Among them, the data volume of the first data may include the number of data rows in the first data, and the sample data volume may be the number of data rows in the first data that need to be subjected to field discreteness analysis. Optionally, the server may determine the data volume of the first data, determine the sample data volume based on the data volume of the first data, and sample the first data according to the sample data volume to obtain analysis sample data. In some embodiments, the server may obtain analysis sample data from the first data based on the sample data volume according to different sampling methods. For the analysis sample data obtained by each sampling method, the server may perform field discreteness analysis respectively to obtain corresponding discreteness analysis results. The server may combine the discreteness analysis results of each sampling method to obtain the discreteness analysis results of the first data. For example, the server may perform weighted fusion on the discreteness analysis results of each sampling method to obtain the discreteness analysis results of the first data.

[0062] In this embodiment, the server determines the sample data volume based on the data volume of the first data, and determines the analysis sample data from the first data according to the sample data volume, so that the data volume of the analysis sample data can be determined based on the data scale of the first data, thereby ensuring the accuracy of the field discreteness analysis based on the analysis sample data, which is conducive to improving the accuracy of data verification based on the discreteness analysis results.

[0063] In an exemplary embodiment, based on the coefficient of variation, the ratio of unique values ​​and the information entropy, a data sampling method is determined, including: determining a coefficient of variation threshold, a unique value ratio threshold and an information entropy threshold; obtaining discrete characteristics of the first data according to a first comparison result between the coefficient of variation and the coefficient of variation threshold, a second comparison result between the unique value ratio and the unique value ratio threshold, and a third comparison result between the information entropy and the information entropy threshold; and determining the data sampling method based on the discrete characteristics.

[0064] Among them, the coefficient of variation threshold, the unique value ratio threshold and the information entropy threshold can be configured according to actual needs, and are respectively used to compare and determine the coefficient of variation, the unique value ratio and the information entropy to construct corresponding discrete features. Optionally, the server can obtain the preset coefficient of variation threshold, the unique value ratio threshold and the information entropy threshold, and compare the coefficient of variation with the coefficient of variation threshold to obtain a first comparison result. The server can compare the unique value ratio with the unique value ratio threshold to obtain a second comparison result. The server can compare the information entropy with the information entropy threshold to obtain a third comparison result. The server can combine the first comparison result, the second comparison result and the third comparison result to obtain the discrete feature of the first data. In some embodiments, the server can perform weighted fusion on the first comparison result, the second comparison result and the third comparison result to obtain the discrete feature of the first data, and the discrete feature is used to characterize the discrete degree of the first data. The server can determine the corresponding data sampling method based on the discrete feature of the first data. For example, the discrete feature of the first data can include high discreteness and low discreteness. For high discreteness, the data sampling method can be determined to be a systematic sampling method; for low discreteness, the data sampling method can be determined to be a random sampling method.

[0065] In this embodiment, the server can compare the coefficient of variation, the ratio of unique values ​​and the information entropy according to their respective thresholds, obtain the discrete characteristics of the first data based on the respective comparison results, and determine the corresponding data sampling method based on the discrete characteristics, so that the corresponding data sampling method can be accurately determined based on the discrete characteristics of the first data, which can ensure the targetedness of the sampled data and improve the accuracy of data verification.

[0066] In an exemplary embodiment, the first sampling data and the second sampling data are compared and verified to obtain migration data verification results of the first data and the second data, including: dividing the first sampling data into at least one first data block according to a target block method; dividing the second sampling data into at least one second data block according to a target block method; comparing and verifying the first data block and the corresponding second data block to obtain a data block verification result; and obtaining the migration data verification result of the first data and the second data based on the data block verification result.

[0067] The target block mode is a division mode for dividing the sampled data into blocks. For example, the target block mode may include the number of block divisions, so that the sampled data can be divided according to the number of block divisions to obtain corresponding data blocks. The first data block is a data block obtained by dividing the first sampled data, and the second data block is a data block obtained by dividing the second sampled data. The data block verification result is a verification result obtained by comparing and verifying the divided first data block and the second data block.

[0068] Exemplarily, the server may determine a target block mode, and divide the first sampled data and the second sampled data respectively according to the target block mode, and obtain the first data block and the second data block respectively. The server may compare and verify the data blocks obtained by the division, and specifically the server may compare and verify the first data block and the corresponding second data block to obtain a data block verification result. In some embodiments, for each first data block, the server may determine the corresponding second data block having the same field, and the server may compare and verify the data of each field of the first data block and the corresponding second data block respectively to determine whether the data of each field is consistent, thereby obtaining the corresponding data block verification result. The server may traverse each first data block and the second data block, thereby obtaining a data block verification result of each data block. The server may combine the data block verification results of each data block to obtain the migration data verification results of the first data and the second data.

[0069] In this embodiment, the server can divide the first sampling data and the second sampling data into the first data block and the second data block respectively according to the target block method, and compare and verify the first data block and the corresponding second data block, and determine the migration data verification results of the first data and the second data based on the obtained data block verification results, so that comparison and verification can be performed based on data blocks, which is conducive to improving the processing efficiency of migration data verification.

[0070] In an exemplary embodiment, a first data block and a corresponding second data block are compared and verified to obtain a data block verification result, including: for each first data block, determining a second data block corresponding to the first data block; determining a first hash value for the first data block, and determining a second hash value for the second data block corresponding to the first data block; and obtaining a data block verification result based on a comparison result of the first hash value and the second hash value.

[0071] Among them, the hash value, also known as the hash value, hash value or message digest, is the mapping result obtained by mapping the input data through a hash algorithm. The hash value can specifically be a numerical value obtained through logical operations based on the file content. Different files can generate different hash values.

[0072] Optionally, the server may traverse each data block and perform comparison verification respectively. Specifically, the server may determine a first hash value of a first data block and a second hash value of a second data block corresponding to the first data block. The server may compare the first hash value with the second hash value and determine the corresponding data block verification result according to the comparison result.

[0073] In this embodiment, the server can calculate the hash value of the data block and compare it based on the hash value to determine the data block verification result corresponding to the data block, so that the data block can be compared based on the hash value, which is conducive to improving the processing efficiency of data verification.

[0074] The present application also provides an application scenario, and the application scenario applies the above data verification method. Specifically, the application of the data verification method in the application scenario is as follows:

[0075] This application scenario belongs to the field of data migration and data comparison technology in IT (Information Technology) and software development, and particularly involves comparison methods for massive data sampling and analysis. In the context of the rapid development of cloud computing and big data technology, enterprises often need to perform data migration operations, including migrating data from local data centers to the cloud, or migrating data between different areas in the cloud. This migration usually involves a large amount of data, and the data volume can reach millions or even hundreds of millions of records. After the migration is completed, it is crucial to verify the correctness and integrity of the data. When dealing with data verification after massive data migration, traditional methods have the following main problems: First, the verification efficiency is low. Traditional full data comparison and manual sampling methods are difficult to complete the data comparison task within a reasonable time, which affects the progress of business launch; second, the existing random sampling method is difficult to fully cover the diversity of data, especially when the data distribution is significantly discrete, it is easy to miss inconsistencies, affecting the accuracy of data migration; finally, the full comparison of massive data requires a lot of computing resources and storage space, which is costly and unrealistic to implement in an environment with limited resources. Therefore, how to efficiently and accurately verify the consistency of data after data migration has become an urgent problem to be solved.

[0076] Based on this, the data verification method provided in this application aims to solve the problems of low efficiency and insufficient accuracy of massive data verification after data migration, and involves a method of automated sampling and content comparison of massive data based on field discreteness analysis. Specifically, by analyzing the discreteness of the data table fields, information entropy, coefficient of variation, and unique value ratio are used for evaluation to determine whether to perform systematic sampling or random sampling. The sampled data is divided into blocks and hash values ​​are calculated using multiple threads to achieve fast and efficient data comparison. The data verification method provided in this application can significantly improve the efficiency and accuracy of data verification in a massive data environment, and is suitable for cloud computing, big data, and other scenarios that require large-scale data migration, such as e-commerce, financial services, and social media, etc., which solves the shortcomings of traditional methods in massive data processing.

[0077] Specifically, when migrating data from the source database to the target database, there are five key steps: data extraction, filtering, conversion, loading, and verification. Sampling and comparison processing is an integral part of the overall process, and is the core link in post-data migration verification. Through advanced algorithms and technical means, sampling and comparison processing can automatically verify the migrated data efficiently and accurately, ensuring the consistency and integrity of the data, and significantly improving the reliability of the entire data migration process. Figure 4As shown, when migrating data from the source database to the target database, the data to be migrated can be copied and decoupled to achieve data extraction, and the extracted data can be filtered and reorganized and mapped before being uploaded to the target database. The control of data migration can be implemented by the data migration tool control component (Data Transmission Service, DTS). After data migration, migration verification can be performed on the migrated data, specifically by comparing the migrated data with the test data in the target database, including sampling and comparison processing, such as object integrity verification, data integrity verification, and attribute accuracy verification.

[0078] Furthermore, if Figure 5 As shown, for the data table in the migration data, the server can determine the number of analysis sample rows according to the total data volume of the table, and perform field discreteness analysis on the obtained analysis sample data, and the analysis sample data specifically includes numerical and character data. The discreteness analysis results obtained by the field discreteness analysis specifically include the coefficient of variation, information entropy, and unique value ratio. The server determines the discrete characteristics based on the discreteness analysis results to determine whether the corresponding data sampling method is systematic sampling or random sampling. The server blocks the sampled data obtained by sampling, and performs multi-threaded hash calculation based on the obtained blocks. The server can compare the calculated hash values ​​to generate a data verification report, thereby realizing data verification processing in data migration.

[0079] Specifically, when determining the number of sample rows for analysis based on the overall data volume of the table, the server may determine the number of sample rows to be extracted during the preliminary analysis based on the overall number of rows in the data table. Row number selection strategies may include:

[0080] (1) The data volume is less than 10,000 rows: extract 500 rows of data;

[0081] (2) The data volume is between 10,000 and 1 million rows: extract 1,000 rows of data;

[0082] (3) If the data volume is greater than 1 million rows: extract 5,000 rows of data;

[0083] (4) If the data volume is greater than 10 million rows: Extract 10,000 rows of data.

[0084] For data tables of different sizes, different row number selections can effectively reflect the overall distribution characteristics of the data.

[0085] Furthermore, when performing field discreteness analysis, the server can uniformly divide the field types into character and numeric types, and use corresponding statistical methods for analysis. For numeric fields, the coefficient of variation is calculated to measure the relative discreteness of the data. The coefficient of variation helps determine the discreteness and fluctuation range of the data by evaluating the ratio of the standard deviation to the mean of the data. For character fields, the unique value ratio is calculated to measure the diversity of different values ​​in the data set and evaluate the sparsity of the data. Information entropy analysis is a method to evaluate data complexity by quantifying uncertainty. High information entropy indicates high diversity and high discreteness of data distribution. It is applicable to numeric and text data and can help analyze data distribution and discreteness. Through the calculation of information entropy, we can fully understand the distribution characteristics and discreteness of the data, thus providing a basis for subsequent sampling and data comparison.

[0086] Specifically, the information entropy calculation formula can be as follows:

[0087]

[0088] in, is the information entropy, is value The probability of occurrence. The higher the information entropy, the more discrete the data.

[0089] The coefficient of variation (CV) calculation formula can be as follows:

[0090]

[0091] in, is the standard deviation, is the mean, and the coefficient of variation is used to measure the relative dispersion of numerical data.

[0092] The Unique Value Ratio (UVR) calculation formula can be as follows:

[0093] UVR=

[0094] Among them, the unique value ratio is used to measure the ratio of the number of different values ​​in the data set to the total number of records, which reflects the diversity and discreteness of the data.

[0095] When judging the discrete characteristics of a table, the server can judge the overall discrete characteristics of the data table based on the results of the field discreteness analysis. Among them, threshold setting: setting the thresholds for the coefficient of variation, information entropy, and unique value ratio. For example, the coefficient of variation threshold: 0.5, the information entropy threshold: 2.0, and the unique value ratio threshold: 0.3. If any two indicators of the coefficient of variation, information entropy, and unique value ratio of a field exceed the threshold, the table is judged to have high discreteness and systematic sampling is used. If all indicators are below the threshold, the table is judged to have low discreteness and random sampling is used.

[0096] When selecting a sampling method, assume that the user chooses to sample 20% of the data, and the total number of rows and discrete characteristics of the data tables on the source and target sides are known. Among them, system sampling is a regular sampling method that ensures uniform distribution of samples by selecting data at fixed intervals. First, it is necessary to determine the total number of rows N in the data table, and then calculate the sampling interval k according to the sampling ratio set by the user, that is, to extract a row of samples every k rows. Random sampling is an unbiased sampling method that ensures the representativeness of the sample by randomly selecting data points. According to the sampling ratio set by the user, calculate the number of rows n that need to be sampled. Use a random number generator to generate n random row numbers between 1 and N, and select the row data corresponding to these row numbers.

[0097] In data block processing, the server can block the sampled data according to certain rules so that hash values ​​can be calculated in parallel later. Blocking basis can include dividing the data into blocks of 1,000 rows according to the primary key or unique identifier of the data table. In multi-threaded hash calculation processing, the server can use multi-threading technology to calculate the hash value of each data block in parallel to speed up the calculation. In the process of comparing hash values ​​and generating data verification reports, the server can compare the hash values ​​of the corresponding blocks of the source database and the target database, and compare the hash values ​​of the source database and the target database block by block to find inconsistent blocks. Summarize the comparison results and generate a detailed report containing data differences and consistency to provide a basis for subsequent data correction and verification.

[0098] In a specific application, two MySQL database tables (source table and target table) containing 10 million records were compared. The data structure of the two tables is the same. The table name is customer_transactions, which is used to record customer transaction information. The table contains the following 10 fields: transaction_id (transaction ID), customer_id (customer ID), transaction_date (transaction date), transaction_amount (transaction amount), product_id (product ID), region (region), payment_method (payment method), transaction_status (transaction status), loyalty_points (loyalty points), category (product category).

[0099] In order to compare the data content, we first randomly selected 10,000 rows of data from each table to calculate the information entropy, unique value ratio and coefficient of variation of the fields. Through analysis, we found that the following 8 fields have high discreteness: transaction_id, customer_id, product_id, region, payment_method, transaction_status, loyalty_points and category. The coefficient of variation and information entropy of these fields are higher than the preset threshold, so the table is judged to be a high discrete data table, and the systematic sampling method is applicable for data comparison.

[0100] During the sampling process, a sampling ratio of 10% was set, and 1 row of data was extracted from every 10 rows of 10 million records, and finally 1 million records were obtained for comparison. Next, the sampled data was divided into a data block for every 1000 rows of data, totaling 1000 data blocks, and the hash value of each data block was calculated in parallel using multi-threading technology. The hash comparison of each data block was completed in 0.02 seconds, and the total time was about 20 seconds. By comparing the hash values ​​of the source table and the target table block by block, it was found that 98% of the data block hash values ​​matched, indicating that the data was basically consistent. For 2% of the unmatched data blocks, further analysis found that the differences were mainly concentrated in the loyalty_points field, and calculation errors occurred in these data blocks and needed to be repaired. The entire verification process took a total of about 5 minutes, which verified the efficiency and accuracy of the data verification method provided by this application in processing massive data migration verification, and provided effective protection for data consistency and integrity.

[0101] The data verification method provided by this application, by combining the analysis of information entropy, coefficient of variation and unique value ratio, can accurately determine the discreteness of the data table, automatically select the appropriate sampling method, greatly improve the efficiency and accuracy of data comparison, and is particularly suitable for the verification of massive data. In addition, the data verification method provided by this application can cope with data tables of different sizes, and through the flexible selection of systematic sampling and random sampling, it ensures the reliability and universality of data verification, and effectively improves the verification quality after data migration.

[0102] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0103] Based on the same inventive concept, the embodiment of the present application also provides a data verification device for implementing the data verification method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more data verification device embodiments provided below can refer to the limitations on the data verification method above, and will not be repeated here.

[0104] In an exemplary embodiment, Figure 6 As shown, a data verification device 600 is provided, comprising: a data determination module 602, a discreteness analysis module 604, a sampling mode determination module 606, a data sampling module 608 and a data comparison module 610, wherein:

[0105] A data determination module 602 is used to determine first data in a source database and second data in a target database, where the second data is obtained by migrating the first data from the source database to the target database;

[0106] A discreteness analysis module 604 is used to perform field discreteness analysis on the first data to obtain discreteness analysis results, wherein the discreteness analysis results include a coefficient of variation for measuring the degree of data discreteness, a unique value ratio for measuring the degree of data sparsity, and an information entropy for measuring the degree of data complexity;

[0107] A sampling mode determination module 606 is used to determine a data sampling mode based on the coefficient of variation, the unique value ratio and the information entropy;

[0108] The data sampling module 608 is used to sample the first data according to the data sampling method to obtain the first sampled data, and sample the second data according to the data sampling method to obtain the second sampled data;

[0109] The data comparison module 610 is used to compare and verify the first sampling data and the second sampling data to obtain migration data verification results of the first data and the second data.

[0110] In one embodiment, the discreteness analysis module 604 is also used to determine analysis sample data from the first data, the analysis sample data including numerical field data and character field data; obtain the coefficient of variation of the first data based on the numerical statistical parameters of the numerical field data; obtain the unique value ratio of the first data based on the unique value statistical parameters of the character field data; obtain the information entropy of the first data based on the value probability of each field data in the analysis sample data; and obtain the discreteness analysis result based on the coefficient of variation, the unique value ratio and the information entropy.

[0111] In one embodiment, the discreteness analysis module 604 is further used to determine the data volume of the first data; determine the sample data volume based on the data volume of the first data; and determine the analysis sample data from the first data according to the sample data volume.

[0112] In one embodiment, the sampling method determination module 606 is also used to determine the coefficient of variation threshold, the unique value ratio threshold and the information entropy threshold; based on the first comparison result between the coefficient of variation and the coefficient of variation threshold, the second comparison result between the unique value ratio and the unique value ratio threshold, and the third comparison result between the information entropy and the information entropy threshold, the discrete characteristics of the first data are obtained; and the data sampling method is determined based on the discrete characteristics.

[0113] In one embodiment, the data comparison module 610 is also used to divide the first sampled data into at least one first data block according to the target block method; divide the second sampled data into at least one second data block according to the target block method; compare and verify the first data block and the corresponding second data block to obtain a data block verification result; and obtain a migration data verification result of the first data and the second data based on the data block verification result.

[0114] In one embodiment, the data comparison module 610 is also used to determine, for each first data block, a second data block corresponding to the first data block; determine a first hash value for the first data block, and determine a second hash value for the second data block corresponding to the first data block; and obtain a data block verification result based on the comparison result of the first hash value and the second hash value.

[0115] Each module in the above data verification device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module above.

[0116] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 7 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data involved in data verification. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data verification method is implemented.

[0117] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 8As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, a data verification method is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0118] Those skilled in the art will understand that Figure 7 or Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0119] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0120] In one embodiment, a computer-readable storage medium is provided, storing a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0121] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0122] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0123] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0124] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0125] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A data verification method, characterized in that: The method comprises: Determine first data in a source database and second data in a target database, wherein the second data is obtained by migrating the first data from the source database to the target database; Performing a field discreteness analysis on the first data to obtain a discreteness analysis result, wherein the discreteness analysis result includes a coefficient of variation for measuring the degree of data discreteness, a unique value ratio for measuring the degree of data sparsity, and an information entropy for measuring the degree of data complexity; Determining a data sampling method based on the coefficient of variation, the unique value ratio and the information entropy; Sampling the first data according to the data sampling method to obtain first sampled data, and sampling the second data according to the data sampling method to obtain second sampled data; The first sampling data and the second sampling data are compared and verified to obtain migration data verification results of the first data and the second data.

2. The method according to claim 1, characterized in that: The performing field discreteness analysis on the first data to obtain discreteness analysis results includes: Determine analysis sample data from the first data, wherein the analysis sample data includes numerical field data and character field data; Obtaining a coefficient of variation of the first data according to a numerical statistical parameter of the numerical field data; Obtaining a unique value ratio of the first data according to a unique value statistical parameter of the character field data; Obtaining information entropy of the first data according to the value probability of each field data in the analysis sample data; Based on the coefficient of variation, the unique value ratio and the information entropy, a discreteness analysis result is obtained.

3. The method according to claim 2, characterized in that The step of determining analysis sample data from the first data comprises: determining the data volume of the first data; Determine the sample data volume based on the data volume of the first data; According to the sample data amount, analysis sample data is determined from the first data.

4. The method according to claim 1, characterized in that: The determining of the data sampling method based on the coefficient of variation, the unique value ratio and the information entropy includes: Determine the coefficient of variation threshold, unique value ratio threshold, and information entropy threshold; Obtaining a discrete feature of the first data according to a first comparison result between the coefficient of variation and the coefficient of variation threshold, a second comparison result between the unique value ratio and the unique value ratio threshold, and a third comparison result between the information entropy and the information entropy threshold; A data sampling method is determined based on the discrete characteristics.

5. The method according to any one of claims 1 to 4, characterized in that: The comparing and verifying the first sampling data and the second sampling data to obtain the migration data verification results of the first data and the second data includes: Dividing the first sampled data into at least one first data block according to a target block manner; Dividing the second sampled data into at least one second data block according to the target block mode; Compare and verify the first data block and the corresponding second data block to obtain a data block verification result; According to the data block verification results, migration data verification results of the first data and the second data are obtained.

6. The method according to claim 5, characterized in that The comparing and verifying the first data block and the corresponding second data block to obtain a data block verification result includes: For each first data block, determine a second data block corresponding to the first data block; Determine a first Hash value for the first data block, and determine a second Hash value for a second data block corresponding to the first data block; A data block verification result is obtained based on the comparison result of the first Hash value and the second Hash value.

7. A data verification device, characterized in that: The device comprises: A data determination module, used to determine first data in a source database and second data in a target database, wherein the second data is obtained by migrating the first data from the source database to the target database; A discreteness analysis module, used for performing field discreteness analysis on the first data to obtain discreteness analysis results, wherein the discreteness analysis results include a coefficient of variation for measuring the degree of data discreteness, a unique value ratio for measuring the degree of data sparsity, and an information entropy for measuring the degree of data complexity; A sampling mode determination module, used for determining a data sampling mode based on the coefficient of variation, the unique value ratio and the information entropy; A data sampling module, configured to obtain first sampled data by sampling the first data according to the data sampling method, and obtain second sampled data by sampling the second data according to the data sampling method; The data comparison module is used to compare and verify the first sampling data and the second sampling data to obtain migration data verification results of the first data and the second data.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images