Data comparison method and device, computer equipment and storage medium

Abstract

The invention relates to a data comparison method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the steps of obtaining a to-be-compared source data set and a to-be-compared target data set, and determining candidate reference fields according to the source data set and the target data set; respectively counting data volume distribution conditions corresponding to each candidate reference field in the source data set and the target data set, and selecting candidate reference fields conforming to uniform distribution conditions of the source data set and the target data set as target reference fields according to the data volume distribution conditions; performing fragmentation processing on data in the source data set and the target data set according to the target reference field to obtain a source comparison data set and a target comparison data set; and comparing the source comparison data set with the target comparison data set to obtain a data comparison result. By adopting the method, the overhigh calculation load of the specific fragment in the data comparison process can be avoided.

Description

Technical Field

[0001] The present application relates to the field of cloud computing technology, and in particular to a data comparison method, apparatus, computer equipment, storage medium and computer program product. Background Art

[0002] At present, in related schemes, when comparing large data sets, the method of sorting by unique key and then performing range sharding is usually adopted. After the two data sets to be compared are sharded, they are distributed to different computing nodes for comparison. The comparison is performed according to the sorting order of the unique key, so as to obtain the comparison results of each group of shards, and then the difference information is statistically calculated.

[0003] However, this range-based sharding solution is prone to concentrated distribution problems in scenarios where data is unevenly distributed, that is, the amount of data in some shards is too large, while the amount of data in other shards is small, causing the computing load of specific shards to be too high. Summary of the invention

[0004] Based on this, it is necessary to provide a data comparison method, device, computer equipment, computer-readable storage medium and computer program product to address the above technical problems, which can avoid excessive computing load on specific slices during the data comparison process.

[0005] In a first aspect, the present application provides a data comparison method, comprising:

[0006] Acquire a source data set and a target data set to be compared, and determine a candidate reference field according to the source data set and the target data set;

[0007] Counting the data volume distribution of each candidate reference field in the source data set and the target data set respectively, and selecting the candidate reference field that meets the uniform distribution conditions of both the source data set and the target data set as the target reference field according to the data volume distribution;

[0008] Slice the data in the source data set and the target data set according to the target reference field to obtain a source comparison data set and a target comparison data set;

[0009] The source comparison data set and the target comparison data set are compared to obtain a data comparison result.

[0010] In one embodiment, the respectively counting the data volume distribution corresponding to each candidate reference field in the source data set and the target data set, and selecting the candidate reference field that meets the uniform distribution conditions of both the source data set and the target data set as the target reference field according to the data volume distribution, includes:

[0011] Determine the field range corresponding to each candidate reference field, and for each candidate reference field, respectively count the data volume corresponding to each field range in the source data set and the target data set to obtain the data volume distribution of the candidate reference field;

[0012] According to the data volume distribution, selecting the candidate reference field that meets the uniform distribution condition for the source data set as the reference field to be verified;

[0013] According to the data volume distribution, the reference field to be verified that meets the uniform distribution condition for the target data set is selected as the target reference field.

[0014] In one embodiment, the slicing of the data in the source data set and the target data set according to the target reference field to obtain the source comparison data set and the target comparison data set includes:

[0015] Extracting data from the source data set and the target data set respectively according to the target reference field;

[0016] Divide the data in the same field range into the same comparison data slice;

[0017] The comparison data slices are collected for the source data set and the target data set respectively to obtain a source comparison data set and a target comparison data set.

[0018] In one of the embodiments, the selecting, according to the data volume distribution, the candidate reference field that meets the uniform distribution conditions of the source data set and the target data set as the target reference field further includes:

[0019] In the case that the candidate reference field that meets the uniform distribution condition does not exist in the source data set, or the reference field to be verified that meets the uniform distribution condition does not exist in the target data set, a hash function is performed on the data corresponding to each of the candidate reference fields to obtain hash shard data;

[0020] A hash reference field is selected according to the data volume distribution of the hash shard data, and the hash reference field is used as the target reference field.

[0021] In one embodiment, the step of slicing the data in the source data set and the target data set according to the target reference field to obtain a source comparison data set and a target comparison data set further includes:

[0022] In the case where the target reference field is the hash reference field, converting the hash fragment data in the source data set and the target data set into integers to obtain rounded data;

[0023] The number of hash shards is obtained, and according to the number of hash shards and the hash reference field, the rounded data is sharded to obtain a source comparison data set and a target comparison data set.

[0024] In one embodiment, determining the candidate reference field according to the source data set and the target data set includes:

[0025] In the case that both the source data set and the target data set have unique keys, all fields included in the unique keys are used as candidate reference fields;

[0026] In the case that the source data set or the target data set does not have a unique key, the comparison fields in the source data set and the target data set are used as candidate reference fields.

[0027] In a second aspect, the present application also provides a data comparison device, comprising:

[0028] A data acquisition module, used to acquire a source data set and a target data set to be compared, and determine a candidate reference field according to the source data set and the target data set;

[0029] A data statistics module, used to count the data volume distribution corresponding to each candidate reference field in the source data set and the target data set, and select the candidate reference field that meets the uniform distribution conditions of the source data set and the target data set as the target reference field according to the data volume distribution;

[0030] A data slicing module, used to slice the data in the source data set and the target data set according to the target reference field to obtain a source comparison data set and a target comparison data set;

[0031] The data comparison module is used to compare the source comparison data set with the target comparison data set to obtain a data comparison result.

[0032] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0033] Acquire a source data set and a target data set to be compared, and determine a candidate reference field according to the source data set and the target data set;

[0034] Counting the data volume distribution of each candidate reference field in the source data set and the target data set respectively, and selecting the candidate reference field that meets the uniform distribution conditions of both the source data set and the target data set as the target reference field according to the data volume distribution;

[0035] Slice the data in the source data set and the target data set according to the target reference field to obtain a source comparison data set and a target comparison data set;

[0036] The source comparison data set and the target comparison data set are compared to obtain a data comparison result.

[0037] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0038] Acquire a source data set and a target data set to be compared, and determine a candidate reference field according to the source data set and the target data set;

[0039] Counting the data volume distribution of each candidate reference field in the source data set and the target data set respectively, and selecting the candidate reference field that meets the uniform distribution conditions of both the source data set and the target data set as the target reference field according to the data volume distribution;

[0040] Slice the data in the source data set and the target data set according to the target reference field to obtain a source comparison data set and a target comparison data set;

[0041] The source comparison data set and the target comparison data set are compared to obtain a data comparison result.

[0042] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0043] Acquire a source data set and a target data set to be compared, and determine a candidate reference field according to the source data set and the target data set;

[0044] Counting the data volume distribution of each candidate reference field in the source data set and the target data set respectively, and selecting the candidate reference field that meets the uniform distribution conditions of both the source data set and the target data set as the target reference field according to the data volume distribution;

[0045] Slice the data in the source data set and the target data set according to the target reference field to obtain a source comparison data set and a target comparison data set;

[0046] The source comparison data set and the target comparison data set are compared to obtain a data comparison result.

[0047] The above-mentioned data comparison method, device, computer equipment, storage medium and computer program product obtain the source data set and target data set to be compared, and determine the candidate reference fields according to the source data set and the target data set, respectively count the data volume distribution corresponding to each candidate reference field in the source data set and the target data set, and select the candidate reference field that meets the uniform distribution conditions of the source data set and the target data set as the target reference field according to the data volume distribution, so that the respective sharding ranges corresponding to the reference field used for data sharding are more uniform, and the data in the source data set and the target data set are respectively sharded according to the target reference field to obtain the source comparison data set and the target comparison data set, and the source comparison data set and the target comparison data set are compared to obtain the data comparison result. Therefore, in the scenario of uneven data distribution, it is possible to select a suitable target reference field, avoid the problem of concentrated distribution, avoid the excessive amount of data in some shards and the small amount of data in other shards, and thus avoid the excessive computational load of specific shards during the data comparison process. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0049] FIG1 is an application environment diagram of a data comparison method in one embodiment;

[0050] FIG2 is a schematic diagram of a flow chart of a data comparison method in one embodiment;

[0051] FIG3 is a schematic diagram of a flow chart of steps S204 to S206 of a data comparison method in one embodiment;

[0052] FIG4 is a structural block diagram of a data comparison device in one embodiment;

[0053] FIG5 is a diagram showing the internal structure of a computer device in one embodiment;

[0054] FIG. 6 is a diagram showing the internal structure of a computer device in another embodiment. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0056] The data comparison method provided in the embodiment of the present application can be applied in the application environment shown in Figure 1. Among them, the terminal 102 communicates with the server 104 through the network. The terminal 102 can be used to collect the source data set and the target data set, and send the source data set and the target data set to the server 104. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The data storage system can be used to store the source data set and the target data set. Among them, the terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server 104 can be implemented with an independent server or a server cluster consisting of multiple servers.

[0057] In an exemplary embodiment, as shown in FIG2 , a data comparison method is provided, and the method is applied to the server 104 in FIG1 as an example for explanation, including the following steps S202 to S208 . Among them:

[0058] Step S202: obtaining a source data set and a target data set to be compared, and determining a candidate reference field according to the source data set and the target data set.

[0059] The source data set and the target data set may both be data tables, and the candidate reference fields may be fields in the primary key or unique key of the data table.

[0060] Exemplarily, server 104 may obtain a data table input by a user collected by terminal 102, and determine the number of shards by counting the number of computing nodes, the number of computing resources, the magnitude of table row data, and table structure information, and then determine appropriate candidate benchmark fields for the two data tables based on the number of shards and the relative comparison.

[0061] Exemplarily, when both the source data set and the target data set have a unique key, the server 104 may use all fields included in the unique key as candidate reference fields; when neither the source data set nor the target data set has a unique key, the server 104 may use comparison fields in the source data set and the target data set as candidate reference fields.

[0062] Step S204 , statistically analyzing the data volume distribution of each candidate reference field in the source data set and the target data set, and selecting a candidate reference field that meets the uniform distribution conditions of both the source data set and the target data set as the target reference field according to the data volume distribution.

[0063] For example, the server 104 may first determine whether the user has input a preset reference field, and if there is a preset reference field, use the preset reference field as the target reference field.

[0064] Exemplarily, the server 104 may determine the field range corresponding to each candidate benchmark field, and for each candidate benchmark field, respectively count the data volume corresponding to each field range in the source data set and the target data set to obtain the data volume distribution of the candidate benchmark field, and based on the data volume distribution, select the candidate benchmark field that meets the uniform distribution condition for the source data set as the benchmark field to be verified, and based on the data volume distribution, select the benchmark field that meets the uniform distribution condition for the target data set as the target benchmark field.

[0065] Furthermore, when determining the data volume distribution of the candidate benchmark fields, the server 104 may sample the candidate benchmark field values ​​to perform a numerical histogram analysis, and perform a chi-square test to compare whether the difference between the actual distribution and the expected uniform distribution is within the expected value, thereby selecting a group of target benchmark fields corresponding to the values ​​whose histogram distribution meets the uniform distribution condition, and then performing range slicing according to the target benchmark field value range.

[0066] Step S206 , segmenting the data in the source data set and the target data set according to the target reference field to obtain a source comparison data set and a target comparison data set.

[0067] For example, to ensure that the target reference field data formats of the source data set and the target data set are consistent, the server 104 may standardize the reference field, and then the server 104 may select a corresponding number of rows for segmentation according to a fixed value of the target reference field.

[0068] Exemplarily, the server 104 may extract data from the source data set and the target data set respectively according to the target reference field; divide the data in the same field range into the same comparison data slice; collect the comparison data slices for the source data set and the target data set respectively to obtain the source comparison data set and the target comparison data set.

[0069] Step S208: comparing the source comparison data set and the target comparison data set to obtain a data comparison result.

[0070] Exemplarily, the server 104 can set a set of comparison tables in each computing node, and insert the target reference field values ​​of the source comparison data set and the target comparison data set after sharding into the corresponding comparison tables respectively, for standardized processing with consistent data formats, thereby facilitating heterogeneous databases to uniformly process the processing difference field types, so as to obtain the row data added, updated, and deleted by the source comparison data set shard relative to the target comparison data set shard, and send it back to the statistics node after collection. Next, the server 104 can send the row differences after sharding processing to the statistics node for collection. After all the shards are processed, the change information is generated through the row differences, and the row differences and change information are stored in the database for query and display.

[0071] In the above data comparison method, by obtaining the source data set and the target data set to be compared, and determining the candidate reference fields according to the source data set and the target data set, the data volume distribution corresponding to each candidate reference field in the source data set and the target data set is respectively counted, and the candidate reference field that meets the uniform distribution conditions of both the source data set and the target data set is selected as the target reference field according to the data volume distribution, so that the respective sharding ranges corresponding to the reference field used for data sharding are more uniform, and the data in the source data set and the target data set are respectively sharded according to the target reference field to obtain the source comparison data set and the target comparison data set, and the source comparison data set and the target comparison data set are compared to obtain the data comparison result. Therefore, in the scenario of uneven data distribution, it is possible to select a suitable target reference field, avoid the problem of concentrated distribution, avoid the excessive amount of data in some shards and the small amount of data in other shards, and thus avoid the excessive computational load of specific shards during the data comparison process.

[0072] In an exemplary embodiment, as shown in FIG3 , step S204 may include steps S302 to S304 . Among them:

[0073] Step S302, when there are no candidate reference fields that meet the uniform distribution condition in the source data set, or when there are no reference fields to be verified that meet the uniform distribution condition in the target data set, a hash function is performed on the data corresponding to each candidate reference field to obtain hash shard data.

[0074] For example, when there are no candidate benchmark fields that meet the uniform distribution condition in the source data set or the target data set, the server 104 can perform hash sharding on the source data set and the target data set, perform hash function calculation on the data corresponding to each candidate benchmark field, and then perform segmented statistics on the hash calculation results.

[0075] Step S304, selecting a hash reference field according to the data volume distribution of the hash shard data, and using the hash reference field as the target reference field.

[0076] Exemplarily, the server 104 may use different hash functions for calculation, select a hash sharding method that meets the uniform distribution condition, and obtain hash sharding data. Among them, hash sharding distributes data to different storage nodes, and can group data through hash functions so that data can be evenly distributed on multiple shards.

[0077] Accordingly, step S206 may include steps S306 to S308. Among them:

[0078] Step S306, when the target reference field is a hash reference field, the hash fragment data in the source data set and the target data set are converted into integers to obtain rounded processed data.

[0079] Exemplarily, when the target reference field adopts a hash reference field, the server 104 may convert the relevant hash fragment data in the source data set and the target data set into integers. Specifically, the hash reference field is usually a string consisting of letters and numbers, such as "a1b2c3". Next, the server 104 may convert the string into an integer, for example, by converting each character through ASCII encoding, and then combining these encoded values ​​together according to certain rules, for example, by adding, multiplying, or combining these values ​​by shifting, etc., to generate a final integer value.

[0080] For example, the string "a1b2" may be converted into ASCII values ​​[97, 49, 98, 50], and then the server 104 may combine these values ​​according to a certain algorithm to obtain a unique and stable integer.

[0081] For example, the server 104 can also directly use a hash function in a programming language, take the hash field as input, and convert it into an integer output. The integer obtained in this way is usually unique and evenly distributed, and is more suitable for sharding processing.

[0082] Step S308, obtaining the number of hash shards, and performing sharding processing on the rounded data according to the number of hash shards and the hash reference field, to obtain a source comparison data set and a target comparison data set.

[0083] For example, the server 104 can perform a modulo operation on each integer value and the number of shards according to the set number of hash shards, and the result after the modulo operation is the shard ID. In this way, each data row can get a specific shard ID. Finally, the server 104 can classify each data row into a corresponding shard according to the calculated shard ID, thereby completing the data distribution. This process ensures that each data row can be evenly distributed to different shards, which is conducive to efficient query and distributed storage of data.

[0084] In another exemplary embodiment, the server 104 obtains a source data set and a target data set to be compared, and determines candidate reference fields based on the source data set and the target data set. When a unique key exists in both the source data set and the target data set, all fields included in the unique key are used as candidate reference fields. When a unique key does not exist in the source data set or the target data set, the comparison fields in the source data set and the target data set are used as candidate reference fields.

[0085] Next, the server 104 determines the field range corresponding to each candidate reference field, and for each candidate reference field, respectively counts the data volume corresponding to each field range in the source data set and the target data set, obtains the data volume distribution of the candidate reference field, selects the candidate reference field that meets the uniform distribution condition for the source data set as the reference field to be verified according to the data volume distribution, and then selects the reference field that meets the uniform distribution condition for the target data set as the target reference field according to the data volume distribution. When there is no candidate reference field that meets the uniform distribution condition in the source data set, or there is no reference field that meets the uniform distribution condition in the target data set, the server 104 performs a hash function calculation on the data corresponding to each candidate reference field to obtain hash sharding data, and then selects a hash reference field according to the data volume distribution of the hash sharding data, and uses the hash reference field as the target reference field.

[0086] Subsequently, the server 104 extracts data from the source data set and the target data set respectively according to the target reference field, divides the data in the same field range into the same comparison data slice, collects the comparison data slices for the source data set and the target data set respectively, and obtains the source comparison data set and the target comparison data set. When the target reference field is a hash reference field, the server 104 converts the hash slice data in the source data set and the target data set into integers to obtain rounding processing data, then obtains the number of hash slices, and performs slice processing on the rounding processing data according to the number of hash slices and the hash reference field to obtain the source comparison data set and the target comparison data set.

[0087] Finally, the source comparison data set and the target comparison data set are compared to obtain the data comparison results.

[0088] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0089] Based on the same inventive concept, the embodiment of the present application also provides a data comparison device for implementing the data comparison method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more data comparison device embodiments provided below can refer to the limitations on the data comparison method above, and will not be repeated here.

[0090] In an exemplary embodiment, as shown in FIG4 , a data comparison device is provided, including: a data acquisition module 402 , a data statistics module 404 , a data segmentation module 406 and a data comparison module 408 , wherein:

[0091] The data acquisition module 402 is used to acquire a source data set and a target data set to be compared, and determine a candidate reference field according to the source data set and the target data set;

[0092] The data statistics module 404 is used to count the data volume distribution of each candidate reference field in the source data set and the target data set, and select the candidate reference field that meets the uniform distribution conditions of the source data set and the target data set as the target reference field according to the data volume distribution;

[0093] The data slicing module 406 is used to slice the data in the source data set and the target data set according to the target reference field to obtain a source comparison data set and a target comparison data set;

[0094] The data comparison module 408 is used to compare the source comparison data set and the target comparison data set to obtain a data comparison result.

[0095] In one of the embodiments, the data statistics module 404 is specifically used to: determine the field range corresponding to each candidate benchmark field, and for each candidate benchmark field, respectively count the data volume corresponding to each field range in the source data set and the target data set to obtain the data volume distribution of the candidate benchmark field; based on the data volume distribution, select the candidate benchmark field that meets the uniform distribution condition for the source data set as the benchmark field to be verified; based on the data volume distribution, select the benchmark field that meets the uniform distribution condition for the target data set as the target benchmark field.

[0096] In one embodiment, the data sharding module 406 includes:

[0097] A data extraction unit, used to extract data from a source data set and a target data set respectively according to a target reference field;

[0098] A data partitioning unit, used to partition data in the same field range into the same comparison data slice;

[0099] The first sharding unit is used to collect comparison data shards for the source data set and the target data set respectively to obtain a source comparison data set and a target comparison data set.

[0100] In one embodiment, the data sharding module 406 further includes:

[0101] A hash calculation unit is used to perform hash function calculation on the data corresponding to each candidate reference field to obtain hash shard data when there is no candidate reference field that meets the uniform distribution condition in the source data set, or when there is no reference field to be verified that meets the uniform distribution condition in the target data set; select a hash reference field according to the data volume distribution of the hash shard data, and use the hash reference field as the target reference field.

[0102] In one embodiment, the data sharding module 406 further includes:

[0103] A data rounding unit, used for converting the hash fragment data in the source data set and the target data set into integers to obtain rounded data when the target reference field is a hash reference field;

[0104] The second sharding unit is used to obtain the number of hash shards, and perform sharding processing on the rounded data according to the number of hash shards and the hash reference field to obtain a source comparison data set and a target comparison data set.

[0105] In one of the embodiments, the data acquisition module 402 is specifically used to: when both the source data set and the target data set have a unique key, use all fields included in the unique key as candidate reference fields; when neither the source data set nor the target data set has a unique key, use comparison fields in the source data set and the target data set as candidate reference fields.

[0106] Each module in the above data comparison device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0107] In an exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be shown in FIG5. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data such as a source data set and a target data set. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a data comparison method is implemented.

[0108] In an exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be shown in FIG6. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, the memory, and the input / output interface are connected via a system bus, and the communication interface, the display unit, and the input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner may be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a data comparison method is implemented. The display unit of the computer device is used to form a visually visible picture, which may be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0109] Those skilled in the art will understand that the structure shown in FIG. 6 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0110] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the following steps when executing the computer program: respectively counting the data volume distribution corresponding to each candidate reference field in the source data set and the target data set, and selecting a candidate reference field that meets the uniform distribution conditions of both the source data set and the target data set as a target reference field according to the data volume distribution; respectively slicing the data in the source data set and the target data set according to the target reference field to obtain a source comparison data set and a target comparison data set; and comparing the source comparison data set and the target comparison data set to obtain a data comparison result.

[0111] In one embodiment, when the processor executes the computer program, the following steps are also implemented: determining the field range corresponding to each candidate benchmark field, and for each candidate benchmark field, respectively counting the data volume corresponding to each field range in the source data set and the target data set to obtain the data volume distribution of the candidate benchmark field; based on the data volume distribution, selecting the candidate benchmark field that meets the uniform distribution condition for the source data set as the benchmark field to be verified; based on the data volume distribution, selecting the benchmark field that meets the uniform distribution condition for the target data set as the target benchmark field.

[0112] In one embodiment, when the processor executes the computer program, the following steps are also implemented: extracting data from the source data set and the target data set respectively according to the target reference field; dividing the data in the same field range into the same comparison data slice; collecting the comparison data slices for the source data set and the target data set respectively to obtain the source comparison data set and the target comparison data set.

[0113] In one embodiment, when the processor executes the computer program, the following steps are also implemented: when there are no candidate benchmark fields that meet the uniform distribution conditions in the source data set, or when there are no benchmark fields to be verified that meet the uniform distribution conditions in the target data set, a hash function is performed on the data corresponding to each candidate benchmark field to obtain hash shard data; a hash benchmark field is selected based on the data volume distribution of the hash shard data, and the hash benchmark field is used as the target benchmark field.

[0114] In one embodiment, when the processor executes the computer program, the following steps are also implemented: when the target reference field is a hash reference field, the hash shard data in the source data set and the target data set are converted into integers to obtain rounded processed data; the number of hash shards is obtained, and according to the number of hash shards and the hash reference field, the rounded processed data is sharded to obtain a source comparison data set and a target comparison data set.

[0115] In one embodiment, when the processor executes the computer program, the following steps are also implemented: when a unique key exists in both the source data set and the target data set, all fields contained in the unique key are used as candidate reference fields; when a unique key does not exist in the source data set or the target data set, comparison fields in the source data set and the target data set are used as candidate reference fields.

[0116] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented: respectively counting the data volume distribution corresponding to each candidate reference field in the source data set and the target data set, and selecting a candidate reference field that meets the uniform distribution conditions of both the source data set and the target data set as the target reference field according to the data volume distribution; respectively slicing the data in the source data set and the target data set according to the target reference field to obtain a source comparison data set and a target comparison data set; and comparing the source comparison data set and the target comparison data set to obtain a data comparison result.

[0117] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: determining the field range corresponding to each candidate benchmark field, and for each candidate benchmark field, respectively counting the data volume corresponding to each field range in the source data set and the target data set to obtain the data volume distribution of the candidate benchmark field; based on the data volume distribution, selecting the candidate benchmark field that meets the uniform distribution condition for the source data set as the benchmark field to be verified; based on the data volume distribution, selecting the benchmark field that meets the uniform distribution condition for the target data set as the target benchmark field.

[0118] In one embodiment, when the processor executes the computer program, the following steps are also implemented: extracting data from the source data set and the target data set respectively according to the target reference field; dividing the data in the same field range into the same comparison data slice; collecting the comparison data slices for the source data set and the target data set respectively to obtain the source comparison data set and the target comparison data set.

[0119] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: when there are no candidate benchmark fields that meet the uniform distribution conditions in the source data set, or when there are no benchmark fields to be verified that meet the uniform distribution conditions in the target data set, a hash function is performed on the data corresponding to each candidate benchmark field to obtain hash shard data; a hash benchmark field is selected based on the data volume distribution of the hash shard data, and the hash benchmark field is used as the target benchmark field.

[0120] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: when the target reference field is a hash reference field, the hash shard data in the source data set and the target data set are converted into integers to obtain rounded processed data; the number of hash shards is obtained, and the rounded processed data is sharded according to the number of hash shards and the hash reference field to obtain a source comparison data set and a target comparison data set.

[0121] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented: when a unique key exists in both the source data set and the target data set, all fields contained in the unique key are used as candidate reference fields; when a unique key does not exist in the source data set or the target data set, comparison fields in the source data set and the target data set are used as candidate reference fields.

[0122] In one embodiment, a computer program product is provided, including a computer program, which, when executed by a processor, implements the following steps: respectively counting the data volume distribution corresponding to each candidate reference field in the source data set and the target data set, and selecting a candidate reference field that meets the uniform distribution conditions of both the source data set and the target data set as a target reference field according to the data volume distribution; respectively slicing the data in the source data set and the target data set according to the target reference field to obtain a source comparison data set and a target comparison data set; and comparing the source comparison data set and the target comparison data set to obtain a data comparison result.

[0123] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: determining the field range corresponding to each candidate benchmark field, and for each candidate benchmark field, respectively counting the data volume corresponding to each field range in the source data set and the target data set to obtain the data volume distribution of the candidate benchmark field; based on the data volume distribution, selecting the candidate benchmark field that meets the uniform distribution condition for the source data set as the benchmark field to be verified; based on the data volume distribution, selecting the benchmark field that meets the uniform distribution condition for the target data set as the target benchmark field.

[0124] In one embodiment, when the processor executes the computer program, the following steps are also implemented: extracting data from the source data set and the target data set respectively according to the target reference field; dividing the data in the same field range into the same comparison data slice; collecting the comparison data slices for the source data set and the target data set respectively to obtain the source comparison data set and the target comparison data set.

[0125] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: when there are no candidate benchmark fields that meet the uniform distribution conditions in the source data set, or when there are no benchmark fields to be verified that meet the uniform distribution conditions in the target data set, a hash function is performed on the data corresponding to each candidate benchmark field to obtain hash shard data; a hash benchmark field is selected based on the data volume distribution of the hash shard data, and the hash benchmark field is used as the target benchmark field.

[0126] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: when the target reference field is a hash reference field, the hash shard data in the source data set and the target data set are converted into integers to obtain rounded processed data; the number of hash shards is obtained, and the rounded processed data is sharded according to the number of hash shards and the hash reference field to obtain a source comparison data set and a target comparison data set.

[0127] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented: when a unique key exists in both the source data set and the target data set, all fields contained in the unique key are used as candidate reference fields; when a unique key does not exist in the source data set or the target data set, comparison fields in the source data set and the target data set are used as candidate reference fields.

[0128] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0129] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0130] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0131] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A data comparison method, characterized in that: The method comprises: Acquire a source data set and a target data set to be compared, and determine a candidate reference field according to the source data set and the target data set; Counting the data volume distribution of each candidate reference field in the source data set and the target data set respectively, and selecting the candidate reference field that meets the uniform distribution conditions of both the source data set and the target data set as the target reference field according to the data volume distribution; Slice the data in the source data set and the target data set according to the target reference field to obtain a source comparison data set and a target comparison data set; The source comparison data set and the target comparison data set are compared to obtain a data comparison result.

2. The method according to claim 1, characterized in that: The respectively counting data volume distribution conditions corresponding to each candidate reference field in the source data set and the target data set, and selecting the candidate reference field that meets the uniform distribution conditions of both the source data set and the target data set as the target reference field according to the data volume distribution conditions, includes: Determine the field range corresponding to each candidate reference field, and for each candidate reference field, respectively count the data volume corresponding to each field range in the source data set and the target data set to obtain the data volume distribution of the candidate reference field; According to the data volume distribution, selecting the candidate reference field that meets the uniform distribution condition for the source data set as the reference field to be verified; According to the data volume distribution, the reference field to be verified that meets the uniform distribution condition for the target data set is selected as the target reference field.

3. The method according to claim 2, characterized in that The step of performing slicing processing on the data in the source data set and the target data set respectively according to the target reference field to obtain a source comparison data set and a target comparison data set includes: Extracting data from the source data set and the target data set respectively according to the target reference field; Divide the data in the same field range into the same comparison data slice; The comparison data slices are collected for the source data set and the target data set respectively to obtain a source comparison data set and a target comparison data set.

4. The method according to claim 2, characterized in that: The step of selecting the candidate reference field that satisfies the uniform distribution conditions of the source data set and the target data set as the target reference field according to the data volume distribution condition further includes: In the case that the candidate reference field that meets the uniform distribution condition does not exist in the source data set, or the reference field to be verified that meets the uniform distribution condition does not exist in the target data set, a hash function is performed on the data corresponding to each of the candidate reference fields to obtain hash shard data; A hash reference field is selected according to the data volume distribution of the hash shard data, and the hash reference field is used as the target reference field.

5. The method according to claim 4, characterized in that The slicing of the data in the source data set and the target data set according to the target reference field to obtain a source comparison data set and a target comparison data set also includes: In the case where the target reference field is the hash reference field, converting the hash fragment data in the source data set and the target data set into integers to obtain rounded data; The number of hash shards is obtained, and according to the number of hash shards and the hash reference field, the rounded data is sharded to obtain a source comparison data set and a target comparison data set.

6. The method according to any one of claims 1 to 4, characterized in that: The step of determining a candidate reference field according to the source data set and the target data set includes: In the case that both the source data set and the target data set have unique keys, all fields included in the unique keys are used as candidate reference fields; In the case that the source data set or the target data set does not have a unique key, the comparison fields in the source data set and the target data set are used as candidate reference fields.

7. A data comparison device, characterized in that: The device comprises: A data acquisition module, used to acquire a source data set and a target data set to be compared, and determine a candidate reference field according to the source data set and the target data set; A data statistics module, used to count the data volume distribution corresponding to each candidate reference field in the source data set and the target data set, and select the candidate reference field that meets the uniform distribution conditions of the source data set and the target data set as the target reference field according to the data volume distribution; A data slicing module, used to slice the data in the source data set and the target data set according to the target reference field to obtain a source comparison data set and a target comparison data set; The data comparison module is used to compare the source comparison data set with the target comparison data set to obtain a data comparison result.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images