Data comparison method, apparatus, device, and storage medium

Abstract

Embodiments of the present application provide a data comparison method, device and equipment and a storage medium, and belong to the technical field of information processing. The method comprises: obtaining a first digital hash set in a first server and a second digital hash set in a second server; comparing a plurality of first identifiers in the first digital hash set and a plurality of second identifiers in the second digital hash set to obtain a first comparison result; comparing a first digital hash value corresponding to the first identifier and a second digital hash value corresponding to the second identifier matched with the first identifier to obtain a second comparison result; and taking the first comparison result and the second comparison result as a comparison result of the first data table and the second data table. The embodiments of the present application improve the efficiency of data comparison in a big data scenario.

Description

Technical Field

[0001] This application relates to the field of information processing technology, and in particular to a data comparison method, apparatus, device and storage medium. Background Technology

[0002] In cross-system and cross-network data storage scenarios, it is often necessary to perform consistency comparisons of data between different servers, requiring that all corresponding field values ​​of matching data rows must be completely identical. For example, data table A on server A needs to be compared row by row with data table B on server B. Traditional methods require transmitting the entire data row to the same node before performing a full field comparison, resulting in high network bandwidth consumption and comparison time increasing linearly with the data volume, leading to low efficiency. In big data scenarios, the time overhead and resource consumption issues of full transmission and comparison modes are particularly prominent.

[0003] In existing solutions, some systems use cryptographic hash algorithms to generate data feature values ​​to reduce transmission volume. They use MD5 and SHA algorithms to generate a 128-bit hash value for the entire data entry for comparison. Compared to comparing all field values, this reduces data transmission volume and improves comparison efficiency. However, this method is less efficient for comparing large datasets. Summary of the Invention

[0004] The main objective of this application is to provide a data comparison method, apparatus, device, and storage medium, which aims to improve the efficiency of data comparison in big data scenarios.

[0005] To achieve the above objectives, a first aspect of this application proposes a data comparison method, the method comprising:

[0006] Obtain a first hash set from the first server and a second hash set from the second server. The first hash set is obtained by performing a hash operation on the first data table to be compared in the first server, and the second hash set is obtained by performing a hash operation on the second data table to be compared in the second server.

[0007] A first comparison result is obtained by comparing a plurality of first identifiers in the first digital hash set with a plurality of second identifiers in the second digital hash set. The first digital hash set includes a plurality of first identifiers and a first digital hash value corresponding to each first identifier. The second digital hash set includes a plurality of second identifiers and a second digital hash value corresponding to each second identifier.

[0008] A second comparison result is obtained by comparing the first digital hash value corresponding to the first identifier and the second digital hash value corresponding to the second identifier that matches the first identifier.

[0009] The first comparison result and the second comparison result are used as the comparison result between the first data table and the second data table.

[0010] In some embodiments, comparing a plurality of first identifiers in the first digital hash set and a plurality of second identifiers in the second digital hash set to obtain a first comparison result includes:

[0011] Store all first identifiers in the first digital hash set into the first set;

[0012] Store all the second identifiers in the second digital hash set into the second set;

[0013] Perform a subtraction set operation on the first set and the second set to obtain the same data and different data between the first set and the second set. The same data includes identifiers that exist in both the first digital hash set and the second digital hash set. The different data includes identifiers that exist in the first digital hash set but do not exist in the second digital hash set.

[0014] The identifiers in the difference data are stored in the comparison failure dataset as the first comparison result.

[0015] In some embodiments, the step of comparing the first digital hash value corresponding to the first identifier and the second digital hash value corresponding to the second identifier that matches the first identifier to obtain a second comparison result includes:

[0016] For each identifier in the same data, if the first digital hash value corresponding to the identifier is the same as the second digital hash value corresponding to the identifier, then the identifier is stored in the comparison dataset; and

[0017] If the first digital hash value corresponding to the identifier is different from the second digital hash value corresponding to the identifier, then the identifier is stored in the comparison failure dataset;

[0018] The comparison passed dataset and the comparison failed dataset obtained based on the same data are used as the second comparison result.

[0019] In some embodiments, after obtaining the first digital hash set in the first server and the second digital hash set in the second server, and before comparing the plurality of first identifiers in the first digital hash set and the plurality of second identifiers in the second digital hash set to obtain a first comparison result, the method further includes:

[0020] Calculate the data volume of the first and second data tables to be compared based on the first and second hash sets.

[0021] If the amount of data in the first data table and the second data table is less than the first preset threshold, then the step of comparing the multiple first identifiers in the first digital hash set and the multiple second identifiers in the second digital hash set is executed.

[0022] If the amount of data in the first data table and the second data table is greater than or equal to the first preset threshold and less than the second preset threshold, then the first digital hash set and the second digital hash set are inserted into the relational database, and the comparison results of the first data table and the second data table are obtained by building indexes and conditional filtering.

[0023] If the amount of data in the first data table and the second data table is greater than or equal to the second preset threshold, then the first digital hash set and the second digital hash set are inserted into the distributed large database, and the comparison results of the first data table and the second data table are obtained by building indexes and conditional filtering.

[0024] In some embodiments, if the data volume of the first data table and the second data table is greater than or equal to the first preset threshold and less than the second preset threshold, then the first digital hash set and the second digital hash set are inserted into a relational database, and a comparison result of the first data table and the second data table is obtained by constructing an index and conditional filtering, including:

[0025] If the amount of data in the first data table and the second data table is greater than or equal to the first preset threshold and less than the second preset threshold, then the first digital hash set and the second digital hash set are inserted into the relational database.

[0026] An index is constructed for the first identifier, the second identifier, the first digital hash value, and the second digital hash value;

[0027] The first data table and the second data table are associated through the first identifier and the second identifier, and the first digital hash value and the second digital hash value are filtered according to the conditions to obtain the second identifier corresponding to the second digital hash value that differs from the first digital hash value;

[0028] The second identifier corresponding to the second digital hash value that is the same as the first digital hash value is stored in the comparison passed dataset, and the second identifier corresponding to the second digital hash value that is different from the first digital hash value is stored in the comparison failed dataset, so as to obtain the comparison result of the first data table and the second data table.

[0029] In some embodiments, if the data volume of the first data table and the second data table is greater than or equal to the second preset threshold, then the first digital hash set and the second digital hash set are inserted into a distributed large database, and a comparison result of the first data table and the second data table is obtained by constructing an index and conditional filtering, including:

[0030] If the amount of data in the first data table and the second data table is greater than or equal to the second preset threshold, then the first digital hash set and the second digital hash set are inserted into the distributed big data database.

[0031] An index is constructed for the first identifier, the second identifier, the first digital hash value, and the second digital hash value;

[0032] The first data table and the second data table are associated through the first identifier and the second identifier, and the first digital hash value and the second digital hash value are filtered according to the conditions to obtain the second identifier corresponding to the second digital hash value that differs from the first digital hash value;

[0033] The second identifier corresponding to the second digital hash value that is the same as the first digital hash value is stored in the comparison passed dataset, and the second identifier corresponding to the second digital hash value that is different from the first digital hash value is stored in the comparison failed dataset, so as to obtain the comparison result of the first data table and the second data table.

[0034] In some embodiments, the first numerical hash set is obtained by the first server according to the following process:

[0035] Retrieve the first data table;

[0036] The collision probability is calculated based on the amount of data in the first data table to obtain the first data amount threshold.

[0037] Select a hash algorithm based on the first data volume threshold;

[0038] The first data table is processed according to the hash algorithm to obtain the first number hash set;

[0039] And / or,

[0040] The second digital hash set is obtained by the second server according to the following process:

[0041] Obtain the second data table;

[0042] The collision probability is calculated based on the amount of data in the second data table to obtain the second data amount threshold;

[0043] Select a hash algorithm based on the second data volume threshold;

[0044] The second data table is processed according to the hash algorithm to obtain the second digital hash set.

[0045] To achieve the above objectives, a second aspect of this application provides a data comparison apparatus, the apparatus comprising:

[0046] The acquisition module is used to acquire a first digital hash set in the first server and a second digital hash set in the second server. The first digital hash set is obtained by performing a hash operation on a first data table to be compared in the first server, and the second digital hash set is obtained by performing a hash operation on a second data table to be compared in the second server.

[0047] The first comparison module is used to compare a plurality of first identifiers in the first digital hash set with a plurality of second identifiers in the second digital hash set to obtain a first comparison result. The first digital hash set includes a plurality of first identifiers and a first digital hash value corresponding to each first identifier. The second digital hash set includes a plurality of second identifiers and a second digital hash value corresponding to each second identifier.

[0048] The second comparison module is used to compare the first digital hash value corresponding to the first identifier and the second digital hash value corresponding to the second identifier that matches the first identifier to obtain a second comparison result;

[0049] The result determination module is used to take the first comparison result and the second comparison result as the comparison result between the first data table and the second data table.

[0050] To achieve the above objectives, a third aspect of the present application provides a computer device, the computer device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method described in the first aspect.

[0051] To achieve the above objectives, a fourth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect.

[0052] The data comparison method, apparatus, device, and storage medium proposed in this application involve performing a hash operation on a first data table to be compared in a first server to obtain a first digital hash set, which contains multiple first identifiers and a first digital hash value corresponding to each first identifier; performing a hash operation on a second data table to be compared in a second server to obtain a second digital hash set, which contains multiple second identifiers and a second digital hash value corresponding to each second identifier; comparing the multiple first identifiers in the first digital hash set with the multiple second identifiers in the second digital hash set to obtain a first comparison result, which includes identifiers that exist in the first server but not in the second server; comparing the first digital hash value corresponding to the first identifier with the second digital hash value corresponding to the second identifier that matches the first identifier to obtain a second comparison result, which includes data with the same identifier but different hash values ​​in the first and second servers; and merging the first and second comparison results to obtain the final comparison result of the first and second data tables. This approach reduces data transmission volume through hash algorithms and efficiently obtains the differences between the two data tables by comparing identifiers and hash values. Attached Figure Description

[0053] Figure 1 This is a flowchart illustrating the data comparison method provided in an embodiment of this application;

[0054] Figure 2 This is a flowchart of the data comparison process provided in the embodiments of this application;

[0055] Figure 3 This is a flowchart of data conversion, transmission, comparison, and judgment provided in an embodiment of this application;

[0056] Figure 4 This is a schematic diagram of the data comparison device provided in the embodiments of this application;

[0057] Figure 5 This is a schematic diagram of the hardware structure of the computer device provided in the embodiments of this application. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0059] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0060] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0061] In data storage across various industries, data comparisons between two ends of a system or network are frequently required. This necessitates that after matching, the corresponding values ​​of all remaining fields must be consistent. For example, if table A (containing n fields) on server A needs to be compared with each record under each of the n fields in table B on server B, traditional methods require transmitting the entire table data to the same node and then comparing each field individually. This leads to inefficiencies such as the entire table data traversing the network, high network bandwidth consumption, and comparison time increasing linearly with the data volume. Especially in big data scenarios, this full-data-volume approach results in even longer overall processing times. Therefore, how to achieve more efficient comparisons of two tables across networks with lower bandwidth consumption has become an important research topic in the industry.

[0062] Currently, some existing solutions employ cryptographic hash algorithms to generate data feature values ​​to reduce transmission volume before data comparison. For example, MD5 and SHA algorithms are used to generate a 128-bit hash value for the entire data before comparison. Compared to comparing all fields, this method significantly reduces data transmission and improves comparison efficiency. However, such methods still have certain efficiency bottlenecks and comparison error risks when dealing with big data scenarios. In terms of efficiency, merging and transforming multiple data entries across multiple fields results in low hash calculation efficiency; there is also the risk of comparison errors due to data collisions, which occur when two different data entries produce the same value after MD5 hashing; furthermore, the efficiency of bit-by-bit comparison of long string hash values ​​still has room for improvement.

[0063] Based on this, embodiments of this application provide a data comparison method, apparatus, device, and storage medium, which aim to reduce data traffic, improve hash conversion speed, avoid collision risks, and improve comparison efficiency.

[0064] The data comparison method, apparatus, device, and storage medium provided in this application are specifically described through the following embodiments. First, the data comparison method in this application is described.

[0065] The data comparison method provided in this application relates to the field of information processing technology. The data comparison method provided in this application can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application implementing the data comparison method, but is not limited to the above forms.

[0066] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer computer devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0067] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data required for the proper functioning of these embodiments acquired.

[0068] Figure 1 This is an optional flowchart of the data comparison method provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps S100 to S400.

[0069] Step S100: Obtain a first hash set from the first server and a second hash set from the second server. The first hash set is obtained by performing a hash operation on the first data table to be compared in the first server, and the second hash set is obtained by performing a hash operation on the second data table to be compared in the second server. The data table in this embodiment may include, but is not limited to, data tables, database tables, Excel files, JSON format, or other semi-structured data; no limitation is made here.

[0070] In cross-server / cross-network data table comparisons, directly transmitting the entire data table across servers consumes significant network bandwidth. Hash operations can convert data into shorter hash values, thereby reducing data transmission volume. A hash value can be considered a "fingerprint" of data; even minor changes in the data content will result in a change in the hash value, thus it can be used to determine data consistency.

[0071] In this embodiment, a data comparison is performed using two tables, A and B, which are cross-server and cross-network. Table A (the first data table) is the data table to be compared on server A (the first server), and table B (the second data table) is the data table to be compared on server B (the second server). The comparison operation between tables A and B is performed on a comparison server, which can be either server A or B, or a third server. Efficient hash programs are distributed across their respective servers, performing calculations independently. These programs are responsible for reading data, performing hash calculations, and sending the results to the comparison server. The comparison program is installed on the comparison server and is responsible for receiving the numerical hash sets from the efficient hash programs, performing comparison calculations, and outputting the results. The flowchart of the comparison process is as follows: Figure 2 As shown.

[0072] Specifically, an efficient hashing program reads the data to be compared from its respective data table, performs a hash operation on the read data, and obtains a digital hash set. The digital hash set contains a unique identifier (id) for each data item and its corresponding digital hash value. Then, the digital hash set is serialized into binary format for easy network transmission; and the serialized data is further compressed using a compression algorithm to reduce the amount of data transmitted. Finally, the compressed data is sent to the comparison server.

[0073] The comparison program receives a set of numerical hashes from an efficient hashing program, decompresses the received data, and restores it to its original format. Then, the comparison program performs a comparison calculation on the two numerical hashes to identify any differences. Finally, the comparison results are output, for example, to generate a report or update a database.

[0074] In this embodiment, a hash operation is performed on the first data table to be compared in the first server to obtain a first digital hash set, which contains multiple first identifiers (ids) and a first digital hash value corresponding to each first identifier. A hash operation is performed on the second data table to be compared in the second server to obtain a second digital hash set, which contains multiple second identifiers (ids) and a second digital hash value corresponding to each second identifier. The first and second digital hash sets are transmitted to the same server and compared to determine the differences between the two data tables.

[0075] Specifically, the first numerical hash set is obtained by the first server according to the following process:

[0076] Retrieve the first data table;

[0077] The collision probability is calculated based on the amount of data in the first data table to obtain the first data amount threshold.

[0078] Select a hash algorithm based on the first data volume threshold;

[0079] The first data table is processed according to the hash algorithm to obtain the first number hash set;

[0080] And / or,

[0081] The second digital hash set is obtained by the second server according to the following process:

[0082] Obtain the second data table;

[0083] The collision probability is calculated based on the amount of data in the second data table to obtain the second data amount threshold;

[0084] Select a hash algorithm based on the second data volume threshold;

[0085] The second data table is processed according to the hash algorithm to obtain the second digital hash set.

[0086] Preferably, in order to improve the speed and efficiency of hash operations, this embodiment uses a high-performance unencrypted hash algorithm, such as xxHash. The xxHash algorithm features fast computation, high throughput, and efficient numeric hash generation, ensuring computational speed while also having a low collision risk.

[0087] Efficient hash programs can choose the `intdigest` method of `xxHash` to obtain a non-zero integer hash value for the data during the conversion process. `intdigest()` is a method in the `xxHash` library used to generate hash values ​​as unsigned integers. Other methods for converting data into hash values ​​can also be chosen, such as `hexdigest()` of `xxHash`, etc., without limitation here. The non-negative integer hash values ​​generated by `xxHash64` and `xxHash128` occupy 8 and 16 bytes of numeric hash space, respectively, while the string hash space for `MD5` and `sha256` is 32 and 64 bytes, respectively. String hashing requires converting each 4-bit binary number into a 1-byte ASCII character (hexadecimal character), doubling the storage space. Numeric hashes, on the other hand, are stored directly in a compact binary format without additional overhead. This reduction in storage usage further optimizes low-bandwidth transmission.

[0088] In actual production, when faced with offline scenarios involving massive amounts of data, the efficient hashing program provides optional compression parameters. The compression efficiency of binary integer storage for digital hash files is usually significantly higher than that of string hash files, further achieving optimization for low-traffic transmission.

[0089] When performing conversion calculations, the efficient hash program uses xxHash, as a non-cryptographic hash algorithm, which offers superior performance in terms of computation speed and throughput. Based on single-threaded performance, xxHash64 achieves 5.5–6.0 GB / s; xxHash128 achieves 4.8–5.3 GB / s; while MD5 achieves 0.4–0.6 GB / s and SHA-256 achieves 0.15–0.25 GB / s. It is evident that xxHash's computation speed is significantly faster than the latter two. This is because the xxHash series utilizes the AVX2 / SSE instruction set. xxHash64 requires only about 20 rounds of computation, while MD5 and SHA-256 require 64 rounds. Furthermore, MD5 / SHA are processed serially, relying on the results of the previous round, and cannot be parallelized. SHA-256, on the other hand, involves multiple bitwise operations and modulo additions in each round, resulting in high computational density. This achieves the goal of efficiently converting hash values, reducing the overall system time.

[0090] For example, an efficient hashing program reads data from a data table and, depending on the range of data volume, transforms it using a high-performance unencrypted xxHash algorithm to obtain a set of numerical hashes.

[0091] When reading data, the efficient hashing program pre-calculates the amount of data to be read and then selects between two implementations of the xxHash hashing algorithm family: xxHash64 and xxHash128. The efficient hashing program defaults to the strictest mode, xxHash128. It also allows users to customize the implementation algorithm based on collision tolerance.

[0092] In the data table, each row of data consists of a unique key `id` (typically a unique key in production; however, multiple unique keys can be used, and different names can be used. Here, we'll use `id` as an example) and other fields. Assuming the table has `x` rows (records) of data, then the table will have `x` `id` values, and all `id` values ​​in the table will be unique, serving as a unique identifier for each row of data.

[0093] Assume the data in table A or table B is as shown in Table 1:

[0094] Table 1

[0095] id Name Age Email ... Insert_date 1 Andy 22 xxx1@y1mail.com ... 2025-01-01 2 Coco 23 xxx2@y1mail.com ... 2025-01-01 ... ... ... ... ... ...

[0096] Suppose table A has n fields: unique key id, f2, f3, ..., fn. Each field can be of character or numeric type, and there are m rows of data. After applying the xxHash algorithm, table A yields a numeric hash set with two fields: unique key id and hashNum, where hashNum is a numeric hash value. After processing by an efficient hashing program, the hash result set structure of the data table is shown in Table 2.

[0097] Table 2

[0098] id hashNun 1 11602999007945094979 2 15372576229549661886 ... ...

[0099] Taking the xxHash64 algorithm as an example, calculate the data reduction ratio:

[0100] Assume n is the total number of fields in the original table A (including the unique key id), s is the average storage size (in bytes) of other fields (f2 to fn), k is the storage size of the id field (in bytes, usually 4 or 8 bytes), and m is the number of data rows.

[0101] The theoretical calculation of the original data volume is as follows:

[0102] Original size of each record: Original size of a single record = k + (n-1) × s;

[0103] Total data volume: Original total size = m × (k + (n-1) × s).

[0104] The theoretically calculated amount of data after processing is:

[0105] Size of the hash result for each record: Size of a single record after processing = k + 8 (xxHash64 outputs 8 bytes);

[0106] Total data volume: Total size after processing = m × (k + 8).

[0107] Finally, the theoretical formula for the data reduction ratio is as follows:

[0108]

[0109] If the id is 4 bytes (INT), the average size s of the other fields is 50 bytes, and the total number of fields n is 5 (including id). Substituting these values ​​into the formula above, we can obtain the reduction ratio as follows:

[0110]

[0111] Therefore, in practical scenarios, the more fields (larger n) and the larger the field values ​​(larger s), the higher the data reduction ratio. In subsequent network transmission, there's no need to transmit the entire table dataset; only the hash result set needs to be sent to the comparison end, thus achieving low-bandwidth transmission.

[0112] This embodiment uses an efficient hashing program to convert the data table into a numerical hash set, enabling low-traffic cross-server data table transmission and facilitating subsequent data comparison operations. Through the cooperation of the efficient hashing program and the comparison program, the cross-server data table comparison is automated and efficient.

[0113] Step S200: Compare the plurality of first identifiers in the first digital hash set with the plurality of second identifiers in the second digital hash set to obtain a first comparison result. The first digital hash set includes a plurality of first identifiers and a first digital hash value corresponding to each first identifier. The second digital hash set includes a plurality of second identifiers and a second digital hash value corresponding to each second identifier.

[0114] In this embodiment, the differences between the two data tables are first identified by comparing the identifiers in the two hash sets. For example, data that exists in the first server but not in the second server, and data that exists in the second server but not in the first server. Set operations (e.g., difference, union, intersection) can be used to compare the identifiers in the two hash sets to quickly identify the differences. Various methods can be used to compare the identifiers in the two hash sets, and no limitation is made here. Finally, the obtained difference data is stored in the comparison-failed dataset, and the same data is then processed for the next comparison step. The difference data stored in the comparison-failed dataset is used as the first comparison result.

[0115] This embodiment can quickly identify data differences by comparing the identifiers (id) in two digital hash sets, avoiding the need to transmit and compare the entire data table, thereby reducing the amount of data and computation and improving the efficiency of data comparison; it can also quickly locate records with inconsistent data in the two data tables based on the identifiers, which facilitates subsequent processing.

[0116] Step S300: Compare the first digital hash value corresponding to the first identifier and the second digital hash value corresponding to the second identifier that matches the first identifier to obtain a second comparison result.

[0117] In this embodiment, after storing data with differing identifiers (ids) into the comparison failure dataset, the IDs of each piece of data with the same identifiers are matched. The first identifier in the first hash set is matched one-to-one with the second identifier in the second hash set. The hash values ​​corresponding to the same identifiers are then compared. Since hash values ​​are numeric, they can be directly compared to determine if the two hash values ​​are equal. The identifiers corresponding to the hash values ​​with the same comparison result are stored in the comparison success dataset, while the identifiers corresponding to the hash values ​​with different comparison results are stored in the comparison failure dataset, thus obtaining the second comparison result.

[0118] This embodiment can determine whether data with the same identifier in two data tables are consistent by comparing numerical hash values. Even if the data content changes slightly, the hash value will change, thus revealing the data differences. Using integer hash values ​​for comparison has higher efficiency and lower computational complexity compared to string hash values, making data comparison tasks more efficient, accurate, and reliable.

[0119] Step S400: The first comparison result and the second comparison result are used as the comparison result between the first data table and the second data table.

[0120] In this embodiment, the first comparison result and the second comparison result are combined to obtain the final comparison result of the first data table and the second data table. The comparison result includes a comparison-passed dataset and a comparison-failed dataset. The comparison-passed dataset includes data in the first data table and the second data table with the same identifier (id) and numerical hash value. The comparison-failed dataset includes data in the first data table and the second data table with different identifiers (id) and data with the same identifier (id) but different numerical hash values.

[0121] This embodiment achieves efficient, secure, and low-traffic data comparison across server tables through hash operations and comparisons. Using an efficient hash algorithm (such as xxHash) to perform hash operations on the data table quickly yields a numerical hash set, improving the efficiency of data comparison. Using hash values ​​for data comparison avoids directly transmitting the entire data table, significantly reducing data transmission volume and network bandwidth usage. Different comparison methods can be selected based on the data size to avoid data collisions. Using integer hash values ​​for comparison offers higher efficiency and lower computational complexity compared to string hash values.

[0122] In some embodiments, step S200 may include, but is not limited to, steps S210 to S240:

[0123] Step S210: Store all the first identifiers in the first digital hash set into the first set;

[0124] Step S220: Store all the second identifiers in the second digital hash set into the second set;

[0125] Step S230: Perform a subtraction set operation on the first set and the second set to obtain the same data and different data between the first set and the second set. The same data includes identifiers that exist simultaneously in the first digital hash set and the second digital hash set. The different data includes identifiers that exist in the first digital hash set but do not exist in the second digital hash set.

[0126] Step S240: Store the identifier in the difference data into the comparison failure dataset as the first comparison result.

[0127] In this embodiment, the data in table A (the first data table) and table B (the second data table) are compared. Typically, one table is used as a baseline, and the differences between the other table and the baseline table are calculated. First, the data present in the baseline table but absent in the other table is calculated. This step only requires reading the IDs from both tables and performing set operations. In the comparison program, the set operation process is as follows: all IDs from table A (assuming A is the baseline) are read and stored in set a (the first set); all IDs from table B are read and stored in set b (the second set); using the common computer set operation, set a minus set b, the set of IDs not present in set b is calculated. This yields the first part of the difference data, i.e., data present in A but absent in B. Based on the set operation results, identical data (identifiers present in both hash sets) and difference data (identifiers present in the first hash set but absent in the second hash set) are obtained. The identifiers in the difference data are stored in the comparison failure dataset as the first comparison result.

[0128] This embodiment uses the difference set operation to quickly find the differences between two data tables, providing a foundation for further data processing operations.

[0129] In some embodiments, step S300 may include, but is not limited to, steps S310 to S320:

[0130] Step S310: For each identifier in the same data, if the first digital hash value corresponding to the identifier is the same as the second digital hash value corresponding to the identifier, then the identifier is stored in the comparison passed dataset; if the first digital hash value corresponding to the identifier is different from the second digital hash value corresponding to the identifier, then the identifier is stored in the comparison failed dataset.

[0131] Step S320: The comparison passed dataset and the comparison failed dataset obtained based on the same data are used as the second comparison result.

[0132] In this embodiment, after the id values ​​of tables A and B match, the values ​​of other fields in tables A and B are compared. A comparison is considered successful only if all id values ​​and numeric hash values ​​are identical. If x records in tables A and B have matching id values, then x hashNum values ​​can be obtained. Next, the hash values ​​of each record are compared to obtain the final comparison result.

[0133] By comparing two numeric hash values, we can determine whether data with the same identifier in two data tables are consistent. Since hash values ​​are numeric, they can be directly compared to determine if they are equal. We iterate through the common IDs (i.e., the same data set), comparing the numeric hash values ​​one by one. If the numeric hash values ​​match, the ID is stored in the comparison-passed dataset; if the numeric hash values ​​do not match, the ID is stored in the comparison-failed dataset, resulting in a second comparison result.

[0134] This embodiment uses hash values ​​for data comparison, avoiding the direct transmission of the entire data table, thus significantly reducing data transmission volume and network bandwidth usage; using integer hash values ​​for comparison has higher efficiency and lower computational complexity compared to string hash values; through hash operations and comparisons, the data comparison results can be obtained quickly.

[0135] In some embodiments, after obtaining the first digital hash set in the first server and the second digital hash set in the second server, and before comparing the plurality of first identifiers in the first digital hash set and the plurality of second identifiers in the second digital hash set to obtain a first comparison result, the steps may include, but are not limited to, the following:

[0136] Calculate the data volume of the first and second data tables to be compared based on the first and second hash sets.

[0137] If the amount of data in the first data table and the second data table is less than the first preset threshold, then the step of comparing the multiple first identifiers in the first digital hash set and the multiple second identifiers in the second digital hash set is executed.

[0138] If the amount of data in the first data table and the second data table is greater than or equal to the first preset threshold and less than the second preset threshold, then the first digital hash set and the second digital hash set are inserted into the relational database, and the comparison results of the first data table and the second data table are obtained by building indexes and conditional filtering.

[0139] If the amount of data in the first data table and the second data table is greater than or equal to the second preset threshold, then the first digital hash set and the second digital hash set are inserted into the distributed large database, and the comparison results of the first data table and the second data table are obtained by building indexes and conditional filtering.

[0140] Wherein, if the data volume of the first data table and the second data table is greater than or equal to the first preset threshold and less than the second preset threshold, then the first digital hash set and the second digital hash set are inserted into the relational database, and by constructing indexes and conditional filtering, the comparison results of the first data table and the second data table are obtained, including:

[0141] If the amount of data in the first data table and the second data table is greater than or equal to the first preset threshold and less than the second preset threshold, then the first digital hash set and the second digital hash set are inserted into the relational database.

[0142] An index is constructed for the first identifier, the second identifier, the first digital hash value, and the second digital hash value;

[0143] The first data table and the second data table are associated through the first identifier and the second identifier, and the first digital hash value and the second digital hash value are filtered according to the conditions to obtain the second identifier corresponding to the second digital hash value that differs from the first digital hash value;

[0144] The second identifier corresponding to the second digital hash value that is the same as the first digital hash value is stored in the comparison passed dataset, and the second identifier corresponding to the second digital hash value that is different from the first digital hash value is stored in the comparison failed dataset, so as to obtain the comparison result of the first data table and the second data table.

[0145] If the data volume of the first data table and the second data table is greater than or equal to the second preset threshold, then the first digital hash set and the second digital hash set are inserted into the distributed large database. By constructing indexes and conditional filtering, the comparison results of the first data table and the second data table are obtained, including:

[0146] If the amount of data in the first data table and the second data table is greater than or equal to the second preset threshold, then the first digital hash set and the second digital hash set are inserted into the distributed big data database.

[0147] An index is constructed for the first identifier, the second identifier, the first digital hash value, and the second digital hash value;

[0148] The first data table and the second data table are associated through the first identifier and the second identifier, and the first digital hash value and the second digital hash value are filtered according to the conditions to obtain the second identifier corresponding to the second digital hash value that differs from the first digital hash value;

[0149] The second identifier corresponding to the second digital hash value that is the same as the first digital hash value is stored in the comparison passed dataset, and the second identifier corresponding to the second digital hash value that is different from the first digital hash value is stored in the comparison failed dataset, so as to obtain the comparison result of the first data table and the second data table.

[0150] In this embodiment, the efficient hashing program pre-calculates the amount of data to be read when reading data, and then selects between two implementations of the xxHash hashing algorithm family: xxHash64 and xxHash128. The efficient hashing program defaults to the strictest mode, xxHash128; it also allows users to customize the selection of the efficient hashing program based on collision tolerance.

[0151] According to the collision probability formula, based on the birthday paradox, the hash collision probability P is approximately:

[0152]

[0153] Where n is the number of data entries; b is the number of bits in the hash value (xxHash64 is 64 bits, xxHash128 is 128 bits).

[0154] Simplifying the formula, when n << 2 b / 2 hour:

[0155]

[0156] Based on the data volume range formula, the maximum safe data volume n is obtained by solving for P ≤ threshold.max :

[0157]

[0158] xxHash64 is already applicable to most scenarios with data volumes less than 30 trillion records (P≤10). -9 The xxHash128 dataset contains over 30 trillion records, resulting in an extremely low tolerance for collision risks (due to stringent requirements such as cryptography and unique key generation). Furthermore, the theoretical collision probability of xxHash128 is P≈3.6×10⁻⁶. -21 The actual risk is extremely low, with no publicly disclosed vulnerabilities; although the theoretical collision probability of the MD5 algorithm is P≈3.6×10⁻⁶, the actual risk is very low. -21 However, it has known vulnerabilities, and actual collision attacks exist (collisions can be artificially generated). Therefore, this system will default to xxHash128. Under the premise of a safe collision rate, the code provides an optional switching parameter function (xxHash64 is optional), which avoids the risk of collision without affecting the overall efficient hash generation.

[0159] The data produced by the efficient hashing program is then compressed and transformed before being transmitted to the same node for comparison. For example... Figure 3 As shown, the compression and transmission procedure in this system is as follows: First, the result data is serialized into an efficient binary format. This step can use Parquet + Snappy serialization. Parquet is a columnar storage file format with a high compression ratio. When converting data to Parquet format, the compression algorithm is a required parameter; the code explicitly selects the Snappy algorithm (an open-source compression / decompression library from Google, designed for high-speed data compression). After the serialization format conversion is complete, a second compression is performed using the Gzip compression algorithm (GNU zip is a widely used file compression and decompression tool, and also refers to the compression algorithm format it uses). Finally, the data is securely transferred to the comparison server via SFTP.

[0160] In this embodiment, Parquet+Snappy achieves a compression ratio more than 70% higher than CSV / JSON. Secondary Gzip compression further reduces the size by 20%–30%. This further reduces the data volume before transmission, meeting the requirements for low-bandwidth data transfer.

[0161] After acquiring the bilateral data, the comparison program decompresses it. In the receiving program, the received binary stream (bytes type) via the transport protocol is processed to obtain data blocks containing gzip-compressed serialized data. Finally, gzip is used to decompress the binary data to re-obtain the hash result set. During this process, the data volume is statistically assessed. The comparison program selects different processing methods based on the size of the data.

[0162] When the total amount of data in tables A and B is small, such as less than 1 million records (i.e., the amount of data in the first and second data tables is less than the first preset threshold, which is 1 million), the data in table B can be fully loaded into a hash table in memory, and then table A can be traversed for fast lookup and comparison.

[0163] When the total data volume is greater than 1 million but less than 100 million records (i.e., the data volume of the first and second data tables is greater than or equal to the first preset threshold but less than the second preset threshold, where the first preset threshold is 1 million and the second preset threshold is 100 million), the data from both sides is inserted into a relational database (such as MySQL / Oracle). Next, the program builds a B+Tree index on the `id` and `hashNum` fields. A B+Tree index is an indexing mechanism based on a B+ tree (Balance+Tree) data structure. The comparison code is constructed using a JOIN + conditional filtering SQL method. The comparison code is an SQL query code. The specific logic of the SQL query code is: the two tables are joined by `id`, and if the `hashNum` values ​​are different, the `id` result set is output. The above steps utilize the capabilities of relational databases to facilitate efficient data comparison operations. Furthermore, because `hashNum` is a non-zero integer type, coupled with the index construction, query performance can be significantly improved, allowing the entire query to be completed within seconds.

[0164] When the total data volume exceeds 100 million records (i.e., the data volume of the first and second data tables exceeds a second preset threshold, which is 100 million), the system loads the data into a distributed large database (such as Hive). A distributed large database is a data cluster composed of multiple servers, which can utilize the powerful computing power and memory of the cluster to perform parallel computation and comparison logic. In contrast, a typical relational database is a single node with limited technical resources. The above processing logic also constructs the comparison code using SQL with JOIN and conditional filtering. The data layer leverages Hive's distributed capabilities, storage optimization, and execution engine adaptability to efficiently handle JOINs and filtering of hundreds of millions of records. Furthermore, the data characteristics of the numeric hash result set, with a unique key ID ensuring JOIN efficiency and integer hashNum accelerating comparisons, make it suitable for Hive's large-scale computing scenarios. As the cluster's capabilities improve, the overall comparison time will be significantly reduced.

[0165] In the above data comparison process, regardless of the step, the CPU must ultimately perform a calculation to determine whether a certain hashNum is equal to another hashNum. HashNum is a non-negative integer. During the comparison, the number of instruction cycles (x86 architecture) is 1 CPU cycle, so the time complexity of a single comparison is O(1).

[0166] The efficiency of string comparison is as follows: Assuming a string length of L, the comparison operation requires traversing each character. In the best case, if the first characters are different, only one comparison is needed. In the worst case, if all L characters are the same, L comparisons are required. Therefore, the time complexity of a single comparison is O((L+1) / 2). In contrast, integer comparison only requires one CPU instruction, with a constant time complexity of O(1). Therefore, hash values ​​of non-negative integers can achieve more efficient comparisons.

[0167] This embodiment utilizes the characteristics of the system result data before and after processing, increases serialization and secondary compression, and reduces the data traffic transmitted over the network; according to different data volume ranges, three different comparison programs are selected to efficiently utilize the characteristics of digital hashing and computing resources.

[0168] For example, this embodiment will be illustrated through a specific example: Assume that table A in the first server has (three fields, three rows of data):

[0169]

[0170] Assume that table B in the second server has three fields and two records:

[0171] id name age

[0172] 1 andy 99

[0173] 2 owen 20

[0174] Since tables A and B are located across servers and networks, this embodiment compares whether the data in these two tables is consistent.

[0175] First, data from table A is loaded. After xxhash hash calculation, the hashing process for table A is as follows:

[0176]

[0177] Table A yields a hash dataset of numeric types:

[0178] id hashCode 1 14980114767480752425 2 6519001415365911238 3 3289438205855191353

[0182] Similarly, the hashing process for table B is as follows:

[0183] ID technology logic

[0184] 1 xxhash('andy|99')

[0185] 2 xxhash('owen|20')

[0186] Table B yields a hash dataset of numeric types:

[0187] id hashCode 1 5595172783991016177 2 6519001415365911238

[0190] After obtaining the hash datasets of the two tables, the hash datasets of the two tables are compared.

[0191] In table A, when id = 1, hashCode = 14980114767480752425;

[0192] In table B, when id=1, hashCode=5595172783991016177;

[0193] At this point, there is a discrepancy in hashCode between table A and table B, and this data is used as a marker of inconsistency.

[0194] Similarly, to compare other rows of data, you only need to match the id and then compare the hashCode to get the result.

[0195] In actual production, regardless of the number of fields in tables A and B, they can be reduced to two fields using this method. Furthermore, because this embodiment uses a more efficient hash algorithm, the conversion of tables A and B into a hash dataset is faster, and the resulting numeric data occupies less memory. During serialization and compression, numeric data is processed faster than strings, resulting in smaller compressed data volumes and thus more efficient data transmission. In the later comparison stage, numeric comparisons are more efficient than string comparisons.

[0196] This application embodiment obtains a digital hash set by performing hash operations on two data tables, reducing the amount of data transmission. It compares the identifiers in the two digital hash sets and the hash values ​​corresponding to each identifier to determine if the data is consistent, identifying identical and differing data, and outputting the comparison results. This application embodiment employs a more efficient hash algorithm, avoiding encryption-based and inefficient hash algorithms, aiming to convert data into hash values ​​more quickly and reduce the amount of comparison data. The hash algorithm explicitly generates unsigned positive integers, achieving less data and improving subsequent comparison efficiency. It avoids algorithms with potential for cracking, making data comparison more secure. It utilizes the characteristics of the system result data before and after processing, adding serialization and secondary compression to reduce network transmission data traffic. It selects three different comparison procedures based on different data volume ranges, efficiently utilizing the characteristics of digital hashing and computing resources. Through algorithm optimization and data volume selection, it provides a data comparison method for similar data fingerprints that is efficient in hash generation, avoids collision risks, greatly reduces data traffic, and is cross-system and cross-network.

[0197] Please see Figure 4 This application embodiment also provides a data comparison device 500, which can implement the above-described data comparison method. The data comparison device 500 includes:

[0198] The acquisition module 10 is used to acquire a first hash set from a first server and a second hash set from a second server. The first hash set is obtained by performing a hash operation on a first data table to be compared in the first server, and the second hash set is obtained by performing a hash operation on a second data table to be compared in the second server. The first hash set is obtained by the first server through the following process: acquiring the first data table; calculating the collision probability based on the data volume of the first data table to obtain a first data volume threshold; selecting a hash algorithm based on the first data volume threshold; processing the first data table according to the hash algorithm to obtain the first hash set; and / or, the second hash set is obtained by the second server through the following process: acquiring the second data table; calculating the collision probability based on the data volume of the second data table to obtain a second data volume threshold; selecting a hash algorithm based on the second data volume threshold; processing the second data table according to the hash algorithm to obtain the second hash set.

[0199] The first comparison module 20 is used to compare a plurality of first identifiers in the first digital hash set with a plurality of second identifiers in the second digital hash set to obtain a first comparison result. The first digital hash set includes a plurality of first identifiers and a first digital hash value corresponding to each first identifier. The second digital hash set includes a plurality of second identifiers and a second digital hash value corresponding to each second identifier.

[0200] The second comparison module 30 is used to compare the first digital hash value corresponding to the first identifier and the second digital hash value corresponding to the second identifier that matches the first identifier to obtain a second comparison result;

[0201] The result determination module 40 is used to take the first comparison result and the second comparison result as the comparison result between the first data table and the second data table.

[0202] In some implementations, the first comparison module 20 may include:

[0203] The first set unit is used to store all the first identifiers in the first digital hash set into the first set;

[0204] The second set unit is used to store all the second identifiers in the second digital hash set into the second set;

[0205] The difference unit is used to perform a difference set operation on the first set and the second set to obtain the same data and the difference data between the first set and the second set. The same data includes identifiers that exist in both the first digital hash set and the second digital hash set. The difference data includes identifiers that exist in the first digital hash set but do not exist in the second digital hash set.

[0206] The first result determination unit is used to store the identifier in the difference data into the comparison failure dataset as the first comparison result.

[0207] In some implementations, the second comparison module 30 may include:

[0208] The comparison unit is used to, for each identifier in the same data, if the first digital hash value corresponding to the identifier is the same as the second digital hash value corresponding to the identifier, then store the identifier in the comparison passed dataset; if the first digital hash value corresponding to the identifier is different from the second digital hash value corresponding to the identifier, then store the identifier in the comparison failed dataset.

[0209] The second result determination unit is used to take the comparison pass dataset and the comparison fail dataset obtained based on the same data as the second comparison result.

[0210] In some embodiments, the device further includes:

[0211] The calculation module is used to calculate the data volume of the first data table and the second data table to be compared based on the first digital hash set and the second digital hash set;

[0212] The first judgment module is used to perform the step of comparing multiple first identifiers in the first digital hash set and multiple second identifiers in the second digital hash set if the amount of data in the first data table and the second data table is less than a first preset threshold.

[0213] The second judgment module is used to insert the first digital hash set and the second digital hash set into a relational database if the amount of data in the first data table and the second data table is greater than or equal to the first preset threshold and less than the second preset threshold, and obtain the comparison result of the first data table and the second data table by constructing an index and condition filtering.

[0214] The third judgment module is used to insert the first digital hash set and the second digital hash set into the distributed big data database if the amount of data in the first data table and the second data table is greater than or equal to the second preset threshold, and obtain the comparison result of the first data table and the second data table by constructing indexes and condition filtering.

[0215] In some implementations, the second determination module may include:

[0216] The first judgment unit is used to insert the first digital hash set and the second digital hash set into the relational database if the amount of data in the first data table and the second data table is greater than or equal to the first preset threshold and less than the second preset threshold.

[0217] The first index unit is used to construct an index for the first identifier, the second identifier, the first digital hash value, and the second digital hash value;

[0218] The first filtering unit is used to associate the first data table and the second data table through the first identifier and the second identifier, and to perform conditional filtering based on the first digital hash value and the second digital hash value to obtain the second identifier corresponding to the second digital hash value that differs from the first digital hash value.

[0219] The third result determination unit is used to store the second identifier corresponding to the second digital hash value that is the same as the first digital hash value into the comparison passed dataset, and to store the second identifier corresponding to the second digital hash value that is different from the first digital hash value into the comparison failed dataset, so as to obtain the comparison result of the first data table and the second data table.

[0220] The third judgment module may include:

[0221] The second judgment unit is used to insert the first digital hash set and the second digital hash set into the distributed big data database if the amount of data in the first data table and the second data table is greater than or equal to the second preset threshold.

[0222] The second index unit is used to construct an index for the first identifier, the second identifier, the first digital hash value, and the second digital hash value;

[0223] The second filtering unit is used to associate the first data table and the second data table through the first identifier and the second identifier, and to perform conditional filtering based on the first digital hash value and the second digital hash value to obtain the second identifier corresponding to the second digital hash value that differs from the first digital hash value.

[0224] The fourth result determination unit is used to store the second identifier corresponding to the second digital hash value that is the same as the first digital hash value into the comparison passed dataset, and to store the second identifier corresponding to the second digital hash value that is different from the first digital hash value into the comparison failed dataset, so as to obtain the comparison result of the first data table and the second data table.

[0225] The specific implementation of this data comparison device is basically the same as the specific embodiment of the data comparison method described above, and will not be repeated here.

[0226] This application also provides a computer device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described data comparison method. This computer device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0227] Please see Figure 5 , Figure 5 The hardware structure of a computer device according to another embodiment is illustrated. The computer device includes:

[0228] The processor 501 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0229] The memory 502 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 502 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 502 and is called and executed by the processor 501 using the data comparison method of the embodiments of this application.

[0230] The input / output interface 503 is used to implement information input and output;

[0231] The communication interface 504 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0232] Bus 505 transmits information between various components of the device (e.g., processor 501, memory 502, input / output interface 503, and communication interface 504);

[0233] The processor 501, memory 502, input / output interface 503, and communication interface 504 are connected to each other within the device via bus 505.

[0234] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described data comparison method.

[0235] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0236] The data comparison method, data comparison apparatus, device, and storage medium provided in this application embodiment perform a hash operation on a first data table to be compared in a first server to obtain a first digital hash set, which contains multiple first identifiers and a first digital hash value corresponding to each first identifier; perform a hash operation on a second data table to be compared in a second server to obtain a second digital hash set, which contains multiple second identifiers and a second digital hash value corresponding to each second identifier; compare the multiple first identifiers in the first digital hash set with the multiple second identifiers in the second digital hash set to obtain a first comparison result, which contains identifiers that exist in the first server but not in the second server; compare the first digital hash value corresponding to the first identifier with the second digital hash value corresponding to the second identifier that matches the first identifier to obtain a second comparison result, which contains data with the same identifier but different hash values ​​in the first and second servers; and merge the first comparison result and the second comparison result to obtain a final comparison result of the first and second data tables. The hash algorithm reduces the amount of data transmission, and by comparing identifiers and hash values, it efficiently obtains the differences between the two data tables.

[0237] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0238] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0239] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0240] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0241] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0242] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0243] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0244] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0245] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0246] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0247] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A data comparison method, characterized in that, The method includes: Obtain a first hash set from the first server and a second hash set from the second server. The first hash set is obtained by performing a hash operation on the first data table to be compared in the first server, and the second hash set is obtained by performing a hash operation on the second data table to be compared in the second server. A first comparison result is obtained by comparing a plurality of first identifiers in the first digital hash set with a plurality of second identifiers in the second digital hash set. The first digital hash set includes a plurality of first identifiers and a first digital hash value corresponding to each first identifier. The second digital hash set includes a plurality of second identifiers and a second digital hash value corresponding to each second identifier. A second comparison result is obtained by comparing the first digital hash value corresponding to the first identifier and the second digital hash value corresponding to the second identifier that matches the first identifier. The first comparison result and the second comparison result are used as the comparison result between the first data table and the second data table; After obtaining the first digital hash set in the first server and the second digital hash set in the second server, and before comparing the multiple first identifiers in the first digital hash set and the multiple second identifiers in the second digital hash set to obtain a first comparison result, the method further includes: Calculate the data volume of the first and second data tables to be compared based on the first and second hash sets. If the amount of data in the first data table and the second data table is less than the first preset threshold, then the step of comparing the multiple first identifiers in the first digital hash set and the multiple second identifiers in the second digital hash set is executed. If the amount of data in the first data table and the second data table is greater than or equal to the first preset threshold and less than the second preset threshold, then the first digital hash set and the second digital hash set are inserted into the relational database, and the comparison results of the first data table and the second data table are obtained by building indexes and conditional filtering. If the amount of data in the first data table and the second data table is greater than or equal to the second preset threshold, then the first digital hash set and the second digital hash set are inserted into the distributed large database, and the comparison results of the first data table and the second data table are obtained by building indexes and conditional filtering.

2. The method according to claim 1, characterized in that, The step of comparing multiple first identifiers in the first digital hash set and multiple second identifiers in the second digital hash set to obtain a first comparison result includes: Store all first identifiers in the first digital hash set into the first set; Store all the second identifiers in the second digital hash set into the second set; Perform a subtraction set operation on the first set and the second set to obtain the same data and different data between the first set and the second set. The same data includes identifiers that exist in both the first digital hash set and the second digital hash set. The different data includes identifiers that exist in the first digital hash set but do not exist in the second digital hash set. The identifiers in the difference data are stored in the comparison failure dataset as the first comparison result.

3. The method according to claim 2, characterized in that, The step of comparing the first hash value corresponding to the first identifier and the second hash value corresponding to the second identifier that matches the first identifier to obtain a second comparison result includes: For each identifier in the same data, if the first digital hash value corresponding to the identifier is the same as the second digital hash value corresponding to the identifier, then the identifier is stored in the comparison dataset; and If the first digital hash value corresponding to the identifier is different from the second digital hash value corresponding to the identifier, then the identifier is stored in the comparison failure dataset; The comparison passed dataset and the comparison failed dataset obtained based on the same data are used as the second comparison result.

4. The method according to claim 1, characterized in that, If the data volume of the first data table and the second data table is greater than or equal to the first preset threshold and less than the second preset threshold, then the first digital hash set and the second digital hash set are inserted into the relational database. By constructing indexes and conditional filtering, the comparison results of the first data table and the second data table are obtained, including: If the amount of data in the first data table and the second data table is greater than or equal to the first preset threshold and less than the second preset threshold, then the first digital hash set and the second digital hash set are inserted into the relational database. An index is constructed for the first identifier, the second identifier, the first digital hash value, and the second digital hash value; The first data table and the second data table are associated through the first identifier and the second identifier, and the first digital hash value and the second digital hash value are filtered according to the conditions to obtain the second identifier corresponding to the second digital hash value that differs from the first digital hash value; The second identifier corresponding to the second digital hash value that is the same as the first digital hash value is stored in the comparison passed dataset, and the second identifier corresponding to the second digital hash value that is different from the first digital hash value is stored in the comparison failed dataset, so as to obtain the comparison result of the first data table and the second data table.

5. The method according to claim 1, characterized in that, If the data volume of the first data table and the second data table is greater than or equal to the second preset threshold, then the first digital hash set and the second digital hash set are inserted into the distributed large database. By constructing indexes and conditional filtering, the comparison results of the first data table and the second data table are obtained, including: If the amount of data in the first data table and the second data table is greater than or equal to the second preset threshold, then the first digital hash set and the second digital hash set are inserted into the distributed big data database. An index is constructed for the first identifier, the second identifier, the first digital hash value, and the second digital hash value; The first data table and the second data table are associated through the first identifier and the second identifier, and the first digital hash value and the second digital hash value are filtered according to the conditions to obtain the second identifier corresponding to the second digital hash value that differs from the first digital hash value; The second identifier corresponding to the second digital hash value that is the same as the first digital hash value is stored in the comparison passed dataset, and the second identifier corresponding to the second digital hash value that is different from the first digital hash value is stored in the comparison failed dataset, so as to obtain the comparison result of the first data table and the second data table.

6. The method according to claim 1, characterized in that, The first numerical hash set is obtained by the first server according to the following process: Retrieve the first data table; The collision probability is calculated based on the amount of data in the first data table to obtain the first data amount threshold. Select a hash algorithm based on the first data volume threshold; The first data table is processed according to the hash algorithm to obtain the first number hash set; And / or, The second digital hash set is obtained by the second server according to the following process: Obtain the second data table; The collision probability is calculated based on the amount of data in the second data table to obtain the second data amount threshold; Select a hash algorithm based on the second data volume threshold; The second data table is processed according to the hash algorithm to obtain the second digital hash set.

7. A data comparison device, characterized in that, The device includes: The acquisition module is used to acquire a first digital hash set in a first server and a second digital hash set in a second server. The first digital hash set is obtained by performing a hash operation on a first data table to be compared in the first server, and the second digital hash set is obtained by performing a hash operation on a second data table to be compared in the second server. The calculation module is used to calculate the data volume of the first data table and the second data table to be compared based on the first digital hash set and the second digital hash set; The first judgment module is used to perform the step of comparing multiple first identifiers in the first digital hash set and multiple second identifiers in the second digital hash set if the amount of data in the first data table and the second data table is less than a first preset threshold. The second judgment module is used to insert the first digital hash set and the second digital hash set into a relational database if the amount of data in the first data table and the second data table is greater than or equal to the first preset threshold and less than the second preset threshold, and obtain the comparison result of the first data table and the second data table by constructing an index and condition filtering. The third judgment module is used to insert the first digital hash set and the second digital hash set into the distributed big data database if the amount of data in the first data table and the second data table is greater than or equal to the second preset threshold, and obtain the comparison result of the first data table and the second data table by constructing indexes and condition filtering. The first comparison module is used to compare a plurality of first identifiers in the first digital hash set with a plurality of second identifiers in the second digital hash set to obtain a first comparison result. The first digital hash set includes a plurality of first identifiers and a first digital hash value corresponding to each first identifier. The second digital hash set includes a plurality of second identifiers and a second digital hash value corresponding to each second identifier. The second comparison module is used to compare the first digital hash value corresponding to the first identifier and the second digital hash value corresponding to the second identifier that matches the first identifier to obtain a second comparison result; The result determination module is used to take the first comparison result and the second comparison result as the comparison result between the first data table and the second data table.

8. A computer device, characterized in that, The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the data comparison method according to any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the data comparison method according to any one of claims 1 to 6.

Images