Data difference comparison method, device, computer equipment and computer program product

Abstract

The present application relates to a data difference comparison method, device, computer equipment, storage medium and computer program product. The method comprises: obtaining source data and target data; performing data statistics on each stream data of the source data and the target data respectively, obtaining a first multidimensional data histogram of the source data and a second multidimensional data histogram of the target data, wherein the first coordinates of the first multidimensional data histogram and the second multidimensional data histogram are hash values, and the second coordinates are the number of occurrences of the hash values ​​calculated for each stream data of the source data and the target data respectively; comparing the consistency of the first multidimensional data histogram and the second multidimensional data histogram, and using the consistency comparison result of the first multidimensional data histogram and the second multidimensional data histogram as the consistency comparison result of the source data and the target data. The use of this method greatly reduces the workload of data difference comparison, thereby reducing the comparison time of data difference comparison.

Description

Technical Field

[0001] The present application relates to the technical field of big data analysis, and in particular to a data difference comparison method, apparatus, computer equipment, storage medium, and computer program product. Background Art

[0002] In daily Internet data management, it is often necessary to perform differential comparison on the transaction data generated by multiple different information systems. The main purpose of differential comparison is to determine whether the data of two parties are completely consistent. The accuracy of the comparison results is particularly important.

[0003] In traditional technology, the commonly used data difference comparison method is the traversal method, that is, by determining the primary key in each party's transaction data, comparing all the information corresponding to the primary key, and determining whether the primary key information is completely consistent, thereby judging whether there are differences in each party's transaction data. Using this method to compare data requires traversing its own data item by item based on the benchmark data.

[0004] With the development of the internet and mobile internet, the dramatic increase in transaction data has become a hallmark of the big data era. Using traditional methods to compare transaction data differences becomes increasingly cumbersome as the data volume increases. Each additional piece of data requires significant comparison, significantly increasing the workload and doubling the time required. Summary of the Invention

[0005] Based on this, it is necessary to provide a data difference comparison method, device, computer equipment, computer-readable storage medium and computer program product to address the above technical problems, which can make the data comparison process simpler and faster.

[0006] In a first aspect, the present application provides a data difference comparison method. The method comprises:

[0007] Get source data and target data;

[0008] Performing data statistics on each piece of stream data of the source data to obtain a first multidimensional data histogram of the source data, wherein a first coordinate of the first multidimensional data histogram is a hash value, and a second coordinate is a number of occurrences of the hash value calculated for each piece of stream data of the source data;

[0009] Performing data statistics on each piece of stream data of the target data to obtain a second multidimensional data histogram of the target data, wherein a first coordinate of the second multidimensional data histogram is a hash value, and a second coordinate is a number of occurrences of the hash value calculated for each piece of stream data of the target data;

[0010] The consistency of the first multidimensional data histogram and the second multidimensional data histogram is compared, and a consistency comparison result of the first multidimensional data histogram and the second multidimensional data histogram is used as a consistency comparison result of the source data and the target data.

[0011] In one embodiment, performing data statistics on each piece of flow data of the source data to obtain a first multidimensional data histogram of the source data includes: obtaining a hash value of each piece of flow data in the source data; performing statistics on the hash value of each piece of flow data in the source data to generate the first multidimensional data histogram;

[0012] The performing data statistics on each piece of flow data of the target data to obtain a second multidimensional data histogram of the target data includes: obtaining a hash value of each piece of flow data in the target data; and performing statistics on the hash value of each piece of flow data in the target data to generate the second multidimensional data histogram.

[0013] In one embodiment, obtaining the hash value of each piece of flow data in the source data includes: performing hash calculations on each piece of flow data in the source data to obtain the hash value of each piece of flow data in the source data;

[0014] The acquiring of the hash value of each piece of stream data in the target data includes: performing hash calculations on each piece of stream data of the target data to obtain the hash value of each piece of stream data of the target data.

[0015] In one embodiment, performing hash calculations on each piece of stream data of the source data to obtain a hash value of each piece of stream data of the source data includes: extracting data content of a specified field in each piece of stream data of the source data; performing hash calculations on the data content of the specified field in each piece of stream data of the source data to obtain a hash value of each piece of stream data of the source data;

[0016] The hash calculation is performed on each piece of the target data to obtain the hash value of each piece of the target data, including: extracting the data content of the specified field in each piece of the target data; and hash calculation is performed on the data content of the specified field in each piece of the target data to obtain the hash value of each piece of the target data.

[0017] In one embodiment, the step of performing hash calculations on each piece of stream data of the source data to obtain a hash value of each piece of stream data of the source data further includes: segmenting each piece of stream data of the source data to obtain data blocks, wherein each data block contains different stream data; and performing hash calculations on each piece of stream data contained in each data block to obtain a hash value of each piece of stream data of the source data;

[0018] The method of performing hash calculations on each piece of stream data of the target data to obtain hash values ​​of each piece of stream data of the target data also includes: performing data segmentation on each piece of stream data of the target data to obtain data blocks, wherein each data block contains different stream data; and performing hash calculations on the stream data contained in each data block to obtain hash values ​​of each piece of stream data of the target data.

[0019] In one embodiment, obtaining the hash value of each piece of stream data in the source data includes: searching for the hash value of each piece of stream data in the source data, wherein the stream data records the hash value of each piece of stream data in the source data, or stores a correspondence between each piece of stream data in the source data and the hash value of the stream data;

[0020] The obtaining of the hash value of each piece of stream data in the target data includes: searching for the hash value of each piece of stream data in the target data, wherein the stream data records the hash value of each piece of stream data of the target data, or stores the correspondence between each piece of stream data of the target data and the hash value of the stream data.

[0021] In one embodiment, the second coordinate of the first multidimensional data histogram includes the number of occurrences of hash values ​​corresponding to the at least two hash functions calculated for each piece of stream data of the source data;

[0022] The second coordinate of the second multidimensional data histogram includes the number of occurrences of hash values ​​corresponding to the at least two hash functions calculated for each piece of stream data of the target data;

[0023] The first multidimensional data histogram and the second multidimensional data histogram also include a third coordinate. The third coordinate of the first multidimensional data histogram is the hash function type corresponding to the hash value of each piece of pipeline data of the source data; the third coordinate of the second multidimensional data histogram is the hash function type corresponding to the hash value of each piece of pipeline data of the target data.

[0024] In one embodiment, the method further comprises:

[0025] If the consistency comparison result is inconsistent, determining a difference histogram between the first multidimensional data histogram and the second multidimensional data histogram, and determining a difference hash value of the difference based on the difference histogram;

[0026] According to the hash value, the corresponding pipeline data is located in the source data.

[0027] In a second aspect, the present application further provides a data difference comparison device. The device comprises:

[0028] Data acquisition module, used to acquire source data and target data;

[0029] a source data statistical analysis module, configured to perform data statistics on each piece of stream data of the source data to obtain a first multidimensional data histogram of the source data, wherein a first coordinate of the first multidimensional data histogram is a hash value, and a second coordinate is a number of occurrences of the hash value calculated for each piece of stream data of the source data;

[0030] a target data statistical analysis module, configured to perform data statistics on each piece of stream data of the target data to obtain a second multidimensional data histogram of the target data, wherein a first coordinate of the second multidimensional data histogram is a hash value, and a second coordinate is a number of occurrences of the hash value calculated for each piece of stream data of the target data;

[0031] A comparison result acquisition module is used to compare the consistency of the first multidimensional data histogram and the second multidimensional data histogram, and use the consistency comparison result of the first multidimensional data histogram and the second multidimensional data histogram as the consistency comparison result of the source data and the target data.

[0032] In one embodiment, the source data statistical analysis module includes:

[0033] A source data hash value acquisition module is used to obtain the hash value of each piece of flow data in the source data;

[0034] A source data hash value statistics module, configured to collect statistics on the hash values ​​of each piece of pipeline data to generate the first multidimensional data histogram;

[0035] The target data statistical analysis module includes:

[0036] A target data hash value acquisition module is used to obtain the hash value of each piece of flow data in the target data;

[0037] The target data hash value statistics module is used to collect statistics on the hash value of each piece of pipeline data in the target data to generate the second multi-dimensional data histogram.

[0038] In one embodiment, the source data hash value acquisition module is used to perform hash calculations on each piece of flow data of the source data to obtain a hash value of each piece of flow data;

[0039] The target data hash value acquisition module is used to perform hash calculations on each piece of pipeline data of the target data to obtain a hash value of each piece of pipeline data.

[0040] In one embodiment, the source data hash value acquisition module is used to extract the data content of a specified field in each piece of stream data of the source data; perform hash calculation on the data content of the specified field in each piece of stream data to obtain the hash value of each piece of stream data of the source data;

[0041] The target data hash value acquisition module is used to extract the data content of the specified field in each piece of stream data of the target data; perform hash calculation on the data content of the specified field in each piece of stream data to obtain the hash value of each piece of stream data of the target data.

[0042] In one embodiment, the source data hash value acquisition module includes:

[0043] A source data segmentation module is used to segment each piece of pipeline data of the source data to obtain data blocks, where each data block contains different pipeline data;

[0044] A source data hash calculation module, configured to perform hash calculations on the stream data contained in each data block of the source data to obtain a hash value of each stream data of the source data;

[0045] The target data hash value acquisition module includes:

[0046] A target data segmentation module is used to segment each piece of pipeline data of the target data to obtain data blocks, where each data block contains different pipeline data;

[0047] The target data hash calculation module is used to perform hash calculations on the stream data contained in each data block of the target data to obtain the hash value of each stream data of the target data.

[0048] In one embodiment, the source data hash value acquisition module is used to search for the hash value of each piece of stream data in the source data, wherein the stream data records the hash value of each piece of stream data of the source data, or stores the corresponding relationship between each piece of stream data of the source data and the hash value of the stream data;

[0049] The target data hash value acquisition module is used to search for the hash value of each stream data in the target data. The stream data records the hash value of each stream data of the target data, or stores the correspondence between each stream data of the target data and the hash value of the stream data.

[0050] In one embodiment, the second coordinate of the first multidimensional data histogram includes the number of occurrences of hash values ​​corresponding to the at least two hash functions calculated for each piece of stream data of the source data;

[0051] The second coordinate of the second multidimensional data histogram includes the number of occurrences of hash values ​​corresponding to the at least two hash functions calculated for each piece of stream data of the target data;

[0052] The first multidimensional data histogram and the second multidimensional data histogram also include a third coordinate. The third coordinate of the first multidimensional data histogram is the hash function type corresponding to the hash value of each piece of pipeline data of the source data; the third coordinate of the second multidimensional data histogram is the hash function type corresponding to the hash value of each piece of pipeline data of the target data.

[0053] In one embodiment, the apparatus further comprises: a data locating module;

[0054] The data positioning module is used to determine the difference histogram between the first multidimensional data histogram and the second multidimensional data histogram if the consistency comparison result is inconsistent, and determine the difference hash value of the difference based on the difference histogram; and locate the corresponding pipeline data in the source data based on the hash value.

[0055] In a third aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0056] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the above method when executed by a processor.

[0057] In a fifth aspect, the present application further provides a computer program product, which includes a computer program that implements the steps of the above method when executed by a processor.

[0058] The above-mentioned data difference comparison method, device, computer equipment, storage medium and computer program product, when performing consistency comparison on a large amount of flow data, first obtains source data and target data, performs data statistics on the source data and target data respectively, obtains a first multidimensional data histogram of the source data and a second multidimensional data histogram of the target data, and then compares the consistency of the first multidimensional data histogram and the second multidimensional data histogram. The consistency comparison result of the multidimensional data histogram is the consistency comparison result of the source data and the target data. Figure 1 The consistency comparison results can intuitively show the consistency of all transaction data in the source data and the target data. There is no need to traverse all transaction data in the source data and the target data one by one, making the entire data comparison process more convenient and intuitive, greatly reducing the workload of data comparison and thus reducing the comparison time of data comparison. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 Schematic diagram of a flow chart of a data difference comparison method in one embodiment;

[0060] Figure 2 Schematic diagram of a flow chart of a data difference comparison method in another embodiment;

[0061] Figure 3 Schematic diagram of a flow chart of a data difference comparison method in another embodiment;

[0062] Figure 4 is a structural block diagram of a data difference comparison device in one embodiment;

[0063] Figure 5 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0065] The acquisition, storage, use, and processing of data in this application's technical solution comply with relevant national laws and regulations.

[0066] In one embodiment, Figure 1 As shown, a data difference comparison method is provided. This embodiment uses the method applied to a terminal as an example for illustration. It is understandable that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0067] Step 202: Acquire source data and target data.

[0068] Flow data refers to business flow data generated for a specific business process, which is used to record relevant data information during the business process. It is understood that relevant data information may include, but is not limited to, user identification, business type, business content, business initiation time, and transaction number. The device that generates this business flow data may be a device involved in the business process, such as a business server or terminal, and this application does not limit this.

[0069] Among them, the source data and the target data are business flow data sets generated for the same business process and stored on different systems respectively. It is understandable that the source data and the target data can be business flow data sets generated for the same business process on different systems respectively, or they can be business flow data sets copied from the downstream business system by the upstream business system, and the downstream system generates the business flow data sets for the business process. Specifically, when the business flow data set of one system is defined as the target data, the business flow data sets of other systems are the source data. It is understandable that the source data can be one or more, and this application does not limit this. The target data can only be one.

[0070] Specifically, the source data and target data to be compared are obtained.

[0071] Step 204 , performing data statistics on each stream data of the source data to obtain a first multidimensional data histogram of the source data, wherein the first coordinate of the first multidimensional data histogram is the hash value, and the second coordinate is the number of occurrences of the hash value calculated for each stream data of the source data.

[0072] Specifically, data statistics are performed on each stream data of the source data, and the hash values ​​calculated for each stream data of the source data are used as the first coordinate, and the number of occurrences of each hash value of the source data is used as the second coordinate to generate a corresponding first multidimensional data histogram.

[0073] Step 206 , performing data statistics on each stream data of the target data to obtain a second multidimensional data histogram of the target data, wherein the first coordinate of the second multidimensional data histogram is the hash value, and the second coordinate is the number of occurrences of the hash value calculated for each stream data of the target data.

[0074] Specifically, data statistics are performed on each stream data of the target data, and the hash values ​​calculated for each stream data of the target data are used as the first coordinate, and the number of occurrences of each hash value of the target data is used as the second coordinate to generate a corresponding second multidimensional data histogram.

[0075] Step 208 : comparing the consistency of the first multidimensional data histogram and the second multidimensional data histogram, and using the consistency comparison result of the first multidimensional data histogram and the second multidimensional data histogram as the consistency comparison result of the source data and the target data.

[0076] The consistency between the first multidimensional data histogram and the second multidimensional data histogram can be obtained by comparing the shapes of the data histograms, or by comparing the consistency of the coordinate values ​​in the generated data histograms, which is not limited in this application.

[0077] If the shapes of the multidimensional data histograms or the values ​​of each coordinate are completely consistent, it is considered that the consistency comparison result between the first multidimensional data histogram and the second multidimensional data histogram is no difference; if the shapes of the multidimensional data histograms or the values ​​of each coordinate are not completely consistent, it is considered that the consistency comparison result between the first multidimensional data histogram and the second multidimensional data histogram is different.

[0078] Specifically, the first multidimensional data histogram is compared with the second multidimensional data histogram to obtain a consistency comparison result between the first multidimensional data histogram and the second multidimensional data histogram, and the consistency comparison result is used as the consistency comparison result between the source data and the target data. If the consistency comparison result is no difference, it means that the data information of the source data and the target data are completely consistent and there is no difference data; if the consistency comparison result is a difference, it means that the data information of the source data and the target data are inconsistent and there is difference data.

[0079] The above data difference comparison method, when performing consistency comparison on a large amount of flow data, first obtains the source data and the target data, performs data statistics on the source data and the target data respectively, obtains the first multidimensional data histogram of the source data and the second multidimensional data histogram of the target data, and then compares the consistency of the first multidimensional data histogram and the second multidimensional data histogram. The consistency comparison result of the multidimensional data histogram is the consistency comparison result of the source data and the target data. Figure 1 The consistency comparison results can intuitively show the consistency of all transaction data in the source data and the target data. There is no need to traverse all transaction data in the source data and the target data one by one, making the entire data comparison process more convenient and intuitive, greatly reducing the workload of data comparison and thus reducing the comparison time of data comparison.

[0080] In one embodiment, performing data statistics on each piece of stream data of the source data to obtain a first multi-dimensional data histogram of the source data includes:

[0081] Get the hash value of each flow data in the source data;

[0082] Specifically, each piece of pipeline data in the source data corresponds to one or more hash values, and the hash values ​​of all the pipeline data in the source data are obtained.

[0083] The hash value of each piece of pipeline data in the source data is counted to generate a first multidimensional data histogram.

[0084] Specifically, the hash values ​​of each piece of flow data are counted according to their size, hash values ​​with the same value are grouped together, and the number of occurrences of each hash value is counted. The hash value is used as the first coordinate and the number of occurrences of the hash value is used as the second coordinate to generate a first multidimensional data histogram.

[0085] Performing data statistics on each piece of pipeline data of the target data to obtain a second multidimensional data histogram of the target data includes: obtaining a hash value of each piece of pipeline data in the target data; performing statistics on the hash value of each piece of pipeline data in the target data to generate a second multidimensional data histogram.

[0086] In this embodiment, the method for obtaining the second multidimensional data histogram of the target data is the same as the method for obtaining the first data histogram of the source data. The only difference is the data processing object, which will not be repeated here.

[0087] In the above embodiment, by statistically analyzing the hash values ​​of each flow data in the source data and the target data, a corresponding multidimensional data histogram is generated. The data difference comparison process that originally required comparing all the information is simplified to a data difference comparison process of comparing whether the multidimensional data histograms are consistent, making the entire process simpler and more intuitive, and reducing the comparison time required for data difference comparison.

[0088] In one embodiment, obtaining the hash value of each piece of flow data in the source data includes:

[0089] Perform hash calculations on each piece of source data to obtain a hash value for each piece of source data.

[0090] Hash, commonly translated as "hashing," transforms an input of arbitrary length (also called a pre-image) into a fixed-length output through a hashing algorithm. This output is the hash value. This conversion is a compression mapping; that is, the space of hash values ​​is typically much smaller than the space of the input. Different inputs may hash to the same output, making it impossible to uniquely identify the input value from the hash value. Simply put, it is a function that compresses a message of arbitrary length into a message digest of a fixed length. A hash algorithm converts data into a signature that is closely related to every byte of the source data. Another characteristic of hash algorithms is that they are difficult to reverse. A hash algorithm is a broad concept that can be considered a concept. Using a hash algorithm can improve storage space utilization, enhance data query efficiency, and provide digital signatures for secure data transmission.

[0091] Specifically, each piece of stream data in the source data is merged into a corresponding string, and a hash function is used to calculate each string to obtain a hash value corresponding to each piece of stream data. Each string contains all the information of the corresponding stream data.

[0092] Acquiring the hash value of each piece of stream data in the target data includes: performing hash calculations on each piece of stream data of the target data to obtain the hash value of each piece of stream data of the target data.

[0093] In this embodiment, the method of obtaining the hash value of each flow data in the target data is the same as the above method of obtaining the hash value of each flow data in the source data. The only difference is the data processing object, which will not be repeated here.

[0094] In this embodiment, by performing hash calculations on each piece of flow data in the source data and the target data, a hash value corresponding to each piece of flow data is obtained, all the information that originally needs to be compared in each piece of flow data is merged to obtain a corresponding character string, and then a hash calculation is performed on the character string, thereby achieving dimensionality reduction of the data that needs to be compared and greatly reducing the amount of data that needs to be compared.

[0095] In one embodiment, hash calculation is performed on each piece of source data to obtain a hash value of each piece of source data, including:

[0096] Extract the data content of the specified fields in each stream of source data;

[0097] The designated fields are the unique primary key and key comparison information that require the most comparison within each transaction data entry. For example, bank remittance transaction information includes information such as the unique serial number, transaction name, transaction type, transaction initiation time, and transaction amount. The unique primary key is the unique serial number, and the key comparison information is the transaction initiation time and amount. Therefore, the data extracted from the designated fields corresponds to the unique serial number, transaction initiation time, and amount. As you can see, the designated fields can be customized to meet different business needs.

[0098] Specifically, the data content corresponding to the specified field of each piece of pipeline data of the source data is obtained, and the data content is extracted.

[0099] Perform hash calculation on the data content of the specified field in each piece of stream data of the source data to obtain the hash value of each piece of stream data of the source data.

[0100] Specifically, the data contents of the specified fields in each piece of stream data of the extracted source data are merged into a character string, and a hash function is used to calculate each character string to obtain a hash value corresponding to each piece of stream data of the source data.

[0101] Performing hash calculations on each piece of target data's flow data to obtain hash values ​​for each piece of target data's flow data, including: extracting data content of a specified field in each piece of target data's flow data; performing hash calculations on data content of a specified field in each piece of target data's flow data to obtain hash values ​​for each piece of target data's flow data.

[0102] In this embodiment, hash calculation is performed on each piece of stream data of the target data respectively to obtain the hash value of each piece of stream data of the target data. The method is the same as the above-mentioned method of performing hash calculation on each piece of stream data of the source data respectively to obtain the hash value of each piece of stream data of the source data. The only difference is the data processing object, which will not be repeated here.

[0103] In this embodiment, the hash value of each piece of flow data is obtained by performing a hash calculation on the data content of the specified field of each piece of flow data in the source data and the target data, and the data to be compared is further reduced in dimension, thereby further reducing the amount of data to be compared and reducing the time required for data difference comparison.

[0104] In one embodiment, performing hash calculations on each piece of source data to obtain a hash value of each piece of source data further includes:

[0105] Perform data segmentation on each stream data of the source data to obtain data blocks, each data block contains different stream data;

[0106] Data segmentation refers to allocating each piece of pipeline data contained in the source data to each data block unit according to a preset segmentation rule to obtain corresponding data blocks. Each data block contains different pipeline data. It is understood that the preset segmentation rule can be to divide the pipeline data set in the source data into equal proportions or different proportions, and this application does not limit this.

[0107] Specifically, each piece of pipeline data in the source data is segmented according to a preset segmentation rule to obtain data blocks, each of which contains different pipeline data.

[0108] Hash calculations are performed on the stream data contained in each data block to obtain the hash value of each stream data of the source data.

[0109] Specifically, hash calculation is performed on each piece of pipeline data in each data block to obtain the hash value of each piece of pipeline data. The hash values ​​in each data block are aggregated to obtain the hash value of each piece of pipeline data in the source data.

[0110] Performing hash calculations on each piece of stream data of the target data to obtain hash values ​​of each piece of stream data of the target data, further comprising: performing data segmentation on each piece of stream data of the target data to obtain data blocks, wherein each data block contains different stream data; performing hash calculations on the stream data contained in each data block to obtain hash values ​​of each piece of stream data of the target data.

[0111] In this embodiment, hash calculation is performed on each stream data of the target data respectively to obtain the hash value of each stream data in the target data. The method is the same as the above-mentioned method of performing hash calculation on each stream data of the source data respectively to obtain the hash value of each stream data in the source data. The only difference is the data processing object, which will not be repeated here.

[0112] In this embodiment, when the source data or target data contains a large amount of pipeline data, the data is segmented into a small number of dispersed data blocks. The hash values ​​obtained after hashing the data blocks are aggregated to obtain the hash values ​​of all the pipeline data in the source data or target data. Using the method in this embodiment, the hash calculation object is changed from a single large-volume pipeline data to multiple small-volume pipeline data, which reduces the workload of the system and makes it easier and faster to obtain the hash values ​​of the required pipeline data.

[0113] In one embodiment, obtaining a hash value of each piece of flow data in the source data includes:

[0114] The hash value of each piece of stream data in the source data is searched, and the stream data records the hash value of each piece of stream data of the source data, or stores the corresponding relationship between each piece of stream data of the source data and the hash value of the stream data.

[0115] The corresponding relationship between the transaction data and the hash value of the transaction data means that the transaction data corresponding to the hash value can be queried through the hash value, or the corresponding hash value can be queried through the transaction data.

[0116] Specifically, each piece of stream data in the source data is hashed to obtain a corresponding hash value, and each hash value is recorded in the corresponding piece of stream data. Alternatively, the stream data stores a correspondence between each piece of stream data and its hash value. It is understood that the process of performing hash calculations on each piece of stream data in the source data can occur during the generation of each piece of stream data, or after all pieces of stream data in the source data have been generated, and this application does not limit this.

[0117] Obtaining the hash value of each stream data in the target data includes: searching for the hash value of each stream data in the target data, recording the hash value of each stream data of the target data in the stream data, or storing the correspondence between each stream data of the target data and the hash value of the stream data.

[0118] In this embodiment, the method of obtaining the hash value of each stream data in the target data is the same as the above method of obtaining the hash value of each stream data in the source data. The only difference is the data processing object, which will not be repeated here.

[0119] In this embodiment, a hash calculation is performed on the flow data in the source data or target data in advance to obtain a corresponding hash value, and the hash value or the corresponding relationship is stored in the flow data. Using the method of this embodiment, when it is necessary to compare data differences between the source data and the target data, the hash value corresponding to each flow data can be directly obtained without performing additional calculations, which greatly reduces the time required for the data difference comparison process.

[0120] In one embodiment, the second coordinate of the first multidimensional data histogram includes the number of occurrences of hash values ​​corresponding to each hash function calculated using at least two hash functions for each piece of stream data of the source data; the second coordinate of the second multidimensional data histogram includes the number of occurrences of hash values ​​corresponding to each hash function calculated using at least two hash functions for each piece of stream data of the target data.

[0121] Specifically, when performing hash value calculation on each pipeline data in the source function or the target function, at least two hash functions of different types are used, and the number of occurrences of the hash value corresponding to each hash function is used as the second coordinate of the multidimensional data histogram.

[0122] The first multidimensional data histogram and the second multidimensional data histogram also include a third coordinate. The third coordinate of the first multidimensional data histogram is the hash function type corresponding to the hash value of each piece of pipeline data generating the source data; the third coordinate of the second multidimensional data histogram is the hash function type corresponding to the hash value of each piece of pipeline data generating the target data.

[0123] Specifically, the hash function type corresponding to the hash value of each pipeline data in the source data and the target data is used as the third coordinate of the corresponding first multidimensional data histogram and second multidimensional data histogram.

[0124] In this embodiment, by using multiple types of hash functions to perform hash calculations on each piece of flow data, the hash conflict problem that occurs when using a single hash function can be avoided, that is, the calculated flow data is different but the hash value is the same, thereby causing data comparison errors.

[0125] In one embodiment, Figure 2 As shown, a data difference comparison method is provided, including:

[0126] Step 302: Acquire source data and target data.

[0127] Step 304 , performing data statistics on each stream data of the source data to obtain a first multidimensional data histogram of the source data, wherein the first coordinate of the first multidimensional data histogram is the hash value, and the second coordinate is the number of occurrences of the hash value calculated for each stream data of the source data.

[0128] Step 306 , performing data statistics on each stream data of the target data to obtain a second multidimensional data histogram of the target data, wherein the first coordinate of the second multidimensional data histogram is the hash value, and the second coordinate is the number of occurrences of the hash value calculated for each stream data of the target data.

[0129] Step 308 : Compare the consistency of the first multidimensional data histogram and the second multidimensional data histogram, and use the consistency comparison result of the first multidimensional data histogram and the second multidimensional data histogram as the consistency comparison result of the source data and the target data.

[0130] Step 309 : If the consistency comparison result is consistent, it is determined that the consistency comparison result between the source data and the target data is no difference.

[0131] Specifically, if the shapes of the multidimensional data histograms or the values ​​of each coordinate are completely consistent, it is considered that the consistency comparison result between the first multidimensional data histogram and the second multidimensional data histogram is no difference, that is, it is determined that the consistency comparison result between the source data and the target data is no difference.

[0132] Step 310 : If the consistency comparison result is inconsistent, determine a difference histogram between the first multidimensional data histogram and the second multidimensional data histogram, and determine a difference hash value of the difference based on the difference histogram.

[0133] The difference histogram is obtained by subtracting the first multidimensional data histogram from the second multidimensional data histogram.

[0134] Specifically, when the consistency comparison result is inconsistent, the first multidimensional data histogram is subtracted from the second multidimensional data histogram to obtain a corresponding difference histogram, which includes a difference hash value.

[0135] Step 312: locate the corresponding pipeline data in the source data according to the hash value.

[0136] When performing hash calculation on each piece of flow data in the source data to obtain a corresponding hash value, the hash value and hash function corresponding to each piece of flow data are recorded in each piece of flow data.

[0137] Specifically, a hash value with a difference is read in the difference histogram, and by querying the record of the hash value in the source data and the target data, the corresponding flow data that generates the hash value is located, thereby determining the flow data with a difference between the source data and the target data.

[0138] In this embodiment, a difference histogram is obtained by subtracting the first multidimensional data histogram from the second multidimensional data histogram. The hash values ​​of the differences are read from the difference histogram. Based on the hash values ​​recorded in the flow data, the flow data with differences can be directly found. Using the method of this embodiment, when it is known that the compared data has differences, the flow data with differences can be quickly located from a large amount of flow data, greatly accelerating the speed of finding the difference data.

[0139] In one embodiment, a data difference comparison method is provided, comprising the following steps:

[0140] This embodiment involves a data comparison process between source data and target data. The source data is the transaction data generated by the bank's downstream system during the inflow of funds, and the target data is the transaction data transmitted or copied from the downstream system by the bank's upstream system via the network.

[0141] Obtain source and target data, extract the data content of specified fields from each transaction record in the source and target data, and combine the data content of the specified fields in each transaction record into a string. Each string contains the unique primary key content and key comparison content corresponding to each transaction record.

[0142] At least two different types of hash functions are used to calculate each character string to obtain a corresponding hash value, and the calculated hash value and its corresponding hash function are stored in each corresponding piece of flow data information.

[0143] The hash values ​​obtained after hash calculation of each stream data in the source data and the target data are respectively counted to obtain first statistical hash data of the source data and second statistical hash data of the target data.

[0144] The first statistical hash data and the second statistical hash data are respectively segmented according to a preset rule to obtain corresponding data blocks.

[0145] According to each data block, the corresponding sub-multidimensional data histogram is generated respectively;

[0146] The first coordinate of the multidimensional data histogram is the hash value, and the second coordinate is the number of times the hash value appears in each hash function. It will be appreciated that in this embodiment, the first coordinate is the horizontal coordinate, and the second coordinate is the dimensional coordinate of the multidimensional data histogram. The dimensionality is determined by the number of hash function types. Therefore, in this embodiment, the number of second coordinates is at least two.

[0147] The sub-data histograms are added together to obtain a first multidimensional data histogram corresponding to the source data and a second multidimensional data histogram corresponding to the target data.

[0148] Subtracting the first multidimensional data histogram from the second multidimensional data histogram to obtain a corresponding difference histogram; and determining difference hash values ​​for differences in the difference histogram and the type of hash function that generates the difference hash values. The determination of the difference hash values ​​for differences in the difference histogram and the type of hash function that generates the difference hash values ​​is based on reading coordinates in the difference histogram. If non-zero data exists in the coordinate data of the difference histogram, it indicates that a difference hash value exists. The difference hash values ​​for differences in the difference histogram and the type of hash function that generates the difference hash values ​​are determined based on the coordinates.

[0149] According to the difference hash value and the type of hash function corresponding to the difference hash value, the hash value information stored in the source data and the target data is queried to locate the flow data with differences between the source data and the target data.

[0150] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0151] Based on the same inventive concept, the present application also provides a data difference comparison device for implementing the aforementioned data difference comparison method. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations in one or more data difference comparison device embodiments provided below can be found in the above-mentioned limitations on the data difference comparison method and will not be further elaborated here.

[0152] In one embodiment, Figure 4 As shown, a data difference comparison device is provided, comprising: a data acquisition module 401, a source data statistical analysis module 402, a target data statistical analysis module 403 and a comparison result acquisition module 404, wherein:

[0153] The data acquisition module 401 is used to acquire source data and target data.

[0154] Flow data refers to business flow data generated for a specific business process, which is used to record relevant data information during the business process. It is understood that relevant data information may include, but is not limited to, user identification, business type, business content, business initiation time, and transaction number. The device that generates this business flow data may be a device involved in the business process, such as a business server or terminal, and this application does not limit this.

[0155] Among them, the source data and the target data are business flow data sets generated for the same business process and stored on different systems respectively. It is understandable that the source data and the target data can be business flow data sets generated for the same business process on different systems respectively, or they can be business flow data sets copied from the downstream business system by the upstream business system, and the downstream system generates the business flow data sets for the business process. Specifically, when the business flow data set of one system is defined as the target data, the business flow data sets of other systems are the source data. It is understandable that the source data can be one or more, and this application does not limit this. The target data can only be one.

[0156] Specifically, the source data and target data to be compared are obtained.

[0157] The source data statistical analysis module 402 is used to perform data statistics on each piece of source data flow data to obtain a first multidimensional data histogram of the source data. The first coordinate of the first multidimensional data histogram is the hash value, and the second coordinate is the number of occurrences of the hash value calculated for each piece of source data flow data.

[0158] Specifically, data statistics are performed on each stream data of the source data, and the hash values ​​calculated for each stream data of the source data are used as the first coordinate, and the number of occurrences of each hash value of the source data is used as the second coordinate to generate a corresponding first multidimensional data histogram.

[0159] The target data statistical analysis module 403 is used to perform data statistics on each stream data of the target data to obtain a second multidimensional data histogram of the target data. The first coordinate of the second multidimensional data histogram is the hash value, and the second coordinate is the number of occurrences of the hash value calculated for each stream data of the target data.

[0160] Specifically, data statistics are performed on each stream data of the target data, and the hash values ​​calculated for each stream data of the target data are used as the first coordinate, and the number of occurrences of each hash value of the target data is used as the second coordinate to generate a corresponding second multidimensional data histogram.

[0161] The comparison result acquisition module 404 is configured to compare the consistency of the first multidimensional data histogram and the second multidimensional data histogram, and use the consistency comparison result of the first multidimensional data histogram and the second multidimensional data histogram as the consistency comparison result of the source data and the target data.

[0162] The consistency between the first multidimensional data histogram and the second multidimensional data histogram can be obtained by comparing the shapes of the data histograms, or by comparing the consistency of the coordinate values ​​in the generated data histograms, which is not limited in this application.

[0163] If the shapes of the multidimensional data histograms or the values ​​of each coordinate are completely consistent, it is considered that the consistency comparison result between the first multidimensional data histogram and the second multidimensional data histogram is no difference; if the shapes of the multidimensional data histograms or the values ​​of each coordinate are not completely consistent, it is considered that the consistency comparison result between the first multidimensional data histogram and the second multidimensional data histogram is different.

[0164] Specifically, the first multidimensional data histogram is compared with the second multidimensional data histogram to obtain a consistency comparison result between the first multidimensional data histogram and the second multidimensional data histogram, and the consistency comparison result is used as the consistency comparison result between the source data and the target data. If the consistency comparison result is no difference, it means that the data information of the source data and the target data are completely consistent and there is no difference data; if the consistency comparison result is a difference, it means that the data information of the source data and the target data are inconsistent and there is difference data.

[0165] When the data difference comparison device is used to compare the consistency of a large amount of flow data, the source data and the target data are first obtained, and after performing data statistics on the source data and the target data respectively, a first multidimensional data histogram of the source data and a second multidimensional data histogram of the target data are obtained. Then, the consistency of the first multidimensional data histogram and the second multidimensional data histogram are compared, and the consistency comparison result of the multidimensional data histogram is the consistency comparison result of the source data and the target data. Figure 1 The consistency comparison results can intuitively show the consistency of all transaction data in the source data and the target data. There is no need to traverse all transaction data in the source data and the target data one by one, making the entire data comparison process more convenient and intuitive, greatly reducing the workload of data comparison and thus reducing the comparison time of data comparison.

[0166] In one embodiment, the source data statistical analysis module includes:

[0167] The source data hash value acquisition module is used to obtain the hash value of each flow data in the source data;

[0168] A source data hash value statistics module is used to count the hash values ​​of each piece of pipeline data and generate a first multidimensional data histogram;

[0169] Target data statistical analysis module, including:

[0170] The target data hash value acquisition module is used to obtain the hash value of each piece of flow data in the target data;

[0171] The target data hash value statistics module is used to count the hash values ​​of each piece of pipeline data in the target data to generate a second multidimensional data histogram.

[0172] In one embodiment, the source data hash value acquisition module is used to perform hash calculations on each piece of source data flow to obtain a hash value of each piece of flow data;

[0173] The target data hash value acquisition module is used to perform hash calculations on each piece of target data to obtain the hash value of each piece of data.

[0174] In one embodiment,

[0175] The source data hash value acquisition module is used to extract the data content of the specified field in each piece of stream data of the source data; perform hash calculation on the data content of the specified field in each piece of stream data to obtain the hash value of each piece of stream data of the source data;

[0176] The target data hash value acquisition module is used to extract the data content of the specified field in each piece of stream data of the target data; perform hash calculation on the data content of the specified field in each piece of stream data to obtain the hash value of each piece of stream data of the target data.

[0177] In one embodiment, the source data hash value acquisition module includes:

[0178] The source data segmentation module is used to segment each stream data of the source data into data blocks, where each data block contains different stream data;

[0179] The source data hash calculation module is used to perform hash calculations on the stream data contained in each data block of the source data to obtain the hash value of each stream data of the source data;

[0180] Target data hash value acquisition module, including:

[0181] The target data segmentation module is used to segment each stream data of the target data into data blocks, where each data block contains different stream data;

[0182] The target data hash calculation module is used to perform hash calculations on the stream data contained in each data block of the target data to obtain the hash value of each stream data of the target data.

[0183] In one embodiment, the source data hash value acquisition module is used to search for the hash value of each piece of stream data in the source data, the stream data records the hash value of each piece of stream data of the source data, or stores the corresponding relationship between each piece of stream data of the source data and the hash value of the stream data;

[0184] The target data hash value acquisition module is used to search for the hash value in each stream data in the target data. The stream data records the hash value of each stream data of the target data, or stores the corresponding relationship between each stream data of the target data and the hash value of the stream data.

[0185] In one embodiment, the second coordinate of the first multidimensional data histogram includes the number of occurrences of hash values ​​corresponding to the hash functions obtained by respectively calculating each piece of stream data of the source data using at least two hash functions;

[0186] The second coordinate of the second multidimensional data histogram includes the number of occurrences of hash values ​​corresponding to the hash functions respectively calculated on each piece of stream data of the target data using at least two hash functions;

[0187] The first multidimensional data histogram and the second multidimensional data histogram also include a third coordinate. The third coordinate of the first multidimensional data histogram is the hash function type corresponding to the hash value of each piece of pipeline data generating the source data; the third coordinate of the second multidimensional data histogram is the hash function type corresponding to the hash value of each piece of pipeline data generating the target data.

[0188] In one embodiment, the data difference comparison device further includes: a data positioning module;

[0189] The data positioning module is used to determine the difference histogram between the first multidimensional data histogram and the second multidimensional data histogram if the consistency comparison result is inconsistent, and determine the difference hash value of the difference based on the difference histogram; and locate the corresponding flow data in the source data based on the hash value.

[0190] Each module in the above-mentioned data difference comparison device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0191] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 5As shown. The computer device includes a processor, a memory and a network interface connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store source data and target data, as well as other data generated during the data difference comparison process. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a data difference comparison method is implemented.

[0192] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0193] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps of the data difference comparison method of each of the above embodiments when executing the computer program.

[0194] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the data difference comparison method of each embodiment described above are implemented.

[0195] In one embodiment, a computer program product is provided, including a computer program, which, when executed by a processor, implements the steps of the data difference comparison method of each of the above embodiments.

[0196] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0197] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in the various embodiments provided herein may be, but are not limited to, a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, and the like.

[0198] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0199] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A data difference comparison method, characterized in that: The method comprises: Get source data and target data; Obtaining a hash value of each piece of pipeline data in the source data; performing statistics on the hash values ​​according to the size of the hash values ​​of each piece of pipeline data in the source data, placing hash values ​​with the same value together, and counting the number of occurrences of each hash value to generate a first multidimensional data histogram, wherein a first coordinate of the first multidimensional data histogram is the hash value, and a second coordinate includes the number of occurrences of hash values ​​corresponding to each hash function calculated using at least two hash functions for each piece of pipeline data in the source data; Obtaining a hash value of each piece of pipeline data in the target data; performing statistics on the hash values ​​according to the hash values ​​of each piece of pipeline data in the target data, placing hash values ​​with the same value together, and counting the number of occurrences of each hash value to generate a second multidimensional data histogram, wherein a first coordinate of the second multidimensional data histogram is a hash value, and a second coordinate includes the number of occurrences of hash values ​​corresponding to each hash function, which are calculated using at least two hash functions for each piece of pipeline data in the target data; The first multidimensional data histogram and the second multidimensional data histogram further include a third coordinate, wherein the third coordinate of the first multidimensional data histogram is a hash function type corresponding to a hash value of each stream data of the source data; the third coordinate of the second multidimensional data histogram is a hash function type corresponding to a hash value of each stream data of the target data; The consistency of the first multidimensional data histogram and the second multidimensional data histogram is compared, and a consistency comparison result of the first multidimensional data histogram and the second multidimensional data histogram is used as a consistency comparison result of the source data and the target data.

2. The method according to claim 1, wherein: The obtaining of the hash value of each piece of flow data in the source data comprises: performing hash calculations on each piece of flow data in the source data to obtain the hash value of each piece of flow data in the source data; The acquiring of the hash value of each piece of stream data in the target data includes: performing hash calculations on each piece of stream data of the target data to obtain the hash value of each piece of stream data of the target data.

3. The method according to claim 2, wherein: The performing hash calculation on each piece of stream data of the source data to obtain a hash value of each piece of stream data of the source data includes: extracting data content of a specified field in each piece of stream data of the source data; performing hash calculation on the data content of the specified field in each piece of stream data of the source data to obtain a hash value of each piece of stream data of the source data; The hash calculation is performed on each piece of the target data to obtain the hash value of each piece of the target data, including: extracting the data content of the specified field in each piece of the target data; and hash calculation is performed on the data content of the specified field in each piece of the target data to obtain the hash value of each piece of the target data.

4. The method according to claim 2, wherein: The step of performing hash calculations on each piece of stream data of the source data to obtain a hash value of each piece of stream data of the source data further includes: segmenting each piece of stream data of the source data to obtain data blocks, wherein each data block contains different stream data; and performing hash calculations on each piece of stream data contained in each data block to obtain a hash value of each piece of stream data of the source data; The method of performing hash calculations on each piece of stream data of the target data to obtain hash values ​​of each piece of stream data of the target data also includes: performing data segmentation on each piece of stream data of the target data to obtain data blocks, wherein each data block contains different stream data; and performing hash calculations on the stream data contained in each data block to obtain hash values ​​of each piece of stream data of the target data.

5. The method according to claim 1, wherein: The obtaining of the hash value of each piece of stream data in the source data comprises: searching for the hash value of each piece of stream data in the source data, wherein the stream data records the hash value of each piece of stream data in the source data, or stores a correspondence between each piece of stream data in the source data and the hash value of the stream data; The obtaining of the hash value of each piece of stream data in the target data includes: searching for the hash value of each piece of stream data in the target data, wherein the stream data records the hash value of each piece of stream data of the target data, or stores the correspondence between each piece of stream data of the target data and the hash value of the stream data.

6. The method according to claim 5, characterized in that The method further comprises: If the consistency comparison result is inconsistent, determining a difference histogram between the first multidimensional data histogram and the second multidimensional data histogram, and determining a difference hash value of the difference based on the difference histogram; According to the hash value, the corresponding pipeline data is located in the source data.

7. A data difference comparison device, characterized in that: The device comprises: Data acquisition module, used to acquire source data and target data; a source data statistical analysis module, configured to obtain a hash value of each piece of pipeline data in the source data; perform statistics on the hash value of each piece of pipeline data to generate a first multidimensional data histogram, wherein the first coordinate of the first multidimensional data histogram is the hash value, and the second coordinate is the number of occurrences of the hash value corresponding to each hash function calculated using at least two hash functions for each piece of pipeline data of the source data; a target data statistical analysis module, configured to obtain a hash value of each piece of pipeline data in the target data; perform statistics on the hash value of each piece of pipeline data in the target data to generate a second multidimensional data histogram, wherein the first coordinate of the second multidimensional data histogram is the hash value, and the second coordinate includes the number of occurrences of the hash value corresponding to each hash function calculated using at least two hash functions for each piece of pipeline data of the target data; the first multidimensional data histogram and the second multidimensional data histogram further include a third coordinate, wherein the third coordinate of the first multidimensional data histogram is the type of hash function corresponding to the hash value of each piece of pipeline data of the source data; and the third coordinate of the second multidimensional data histogram is the type of hash function corresponding to the hash value of each piece of pipeline data of the target data; A comparison result acquisition module is used to compare the consistency of the first multidimensional data histogram and the second multidimensional data histogram, and use the consistency comparison result of the first multidimensional data histogram and the second multidimensional data histogram as the consistency comparison result of the source data and the target data.

8. The device according to claim 7, characterized in that: The source data hash value acquisition module is used to perform hash calculations on each piece of stream data of the source data to obtain a hash value of each piece of stream data; The target data hash value acquisition module is used to perform hash calculations on each piece of pipeline data of the target data to obtain a hash value of each piece of pipeline data.

9. The device according to claim 8, characterized in that: The source data hash value acquisition module is used to extract the data content of the specified field in each piece of stream data of the source data; perform hash calculation on the data content of the specified field in each piece of stream data to obtain the hash value of each piece of stream data of the source data; The target data hash value acquisition module is used to extract the data content of the specified field in each piece of stream data of the target data; A hash calculation is performed on the data content of the specified field in each piece of pipeline data to obtain the hash value of each piece of pipeline data of the target data.

10. The device according to claim 8, characterized in that: The source data hash value acquisition module includes: A source data segmentation module is used to segment each piece of pipeline data of the source data to obtain data blocks, where each data block contains different pipeline data; A source data hash calculation module, configured to perform hash calculations on the stream data contained in each data block of the source data to obtain a hash value of each stream data of the source data; The target data hash value acquisition module includes: A target data segmentation module is used to segment each piece of pipeline data of the target data to obtain data blocks, where each data block contains different pipeline data; The target data hash calculation module is used to perform hash calculations on the stream data contained in each data block of the target data to obtain the hash value of each stream data of the target data.

11. The device according to claim 7, characterized in that: The source data hash value acquisition module is used to search for the hash value of each piece of stream data in the source data, wherein the stream data records the hash value of each piece of stream data of the source data, or stores the corresponding relationship between each piece of stream data of the source data and the hash value of the stream data; The target data hash value acquisition module is used to search for the hash value of each stream data in the target data. The stream data records the hash value of each stream data of the target data, or stores the correspondence between each stream data of the target data and the hash value of the stream data.

12. The device according to claim 11, characterized in that The device further comprises: a data positioning module; The data positioning module is used to determine the difference histogram between the first multidimensional data histogram and the second multidimensional data histogram if the consistency comparison result is inconsistent, determine the difference hash value of the difference based on the difference histogram; and locate the corresponding pipeline data in the source data based on the hash value.

13. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images