Data comparison methods, devices, equipment, storage media and products

By generating a tree diagram locally to compare datasets, the problem of database overload leading to crashes was solved, improving data comparison efficiency and system stability.

CN116226287BActive Publication Date: 2026-11-10INDUSTRIAL AND COMMERCIAL BANK OF CHINA
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202310287546.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-03-22
Publication Date
2026-11-10
Estimated Expiration
2043-03-22

AI Technical Summary

Technical Problem

When comparing data on a database, frequent access operations can overload the database, potentially causing it to crash and affecting the efficiency of data comparison.

Method used

Obtain the deduplicated dataset to be compared, store it in a tree diagram for comparison, generate the target dataset, use pointers to compare the data to be compared, generate comparison results including missing data, and generate the target dataset based on the comparison results.

Benefits of technology

By optimizing data comparison operations locally, the speed of data comparison is improved, the burden on servers and databases is reduced, and the robustness of the system is enhanced.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116226287B_ABST
    Figure CN116226287B_ABST
Patent Text Reader

Abstract

This application provides a data comparison method, apparatus, device, storage medium, and product, applied in the field of big data technology. The method includes: obtaining a deduplicated dataset to be compared; the deduplicated dataset includes unique data to be compared; storing the data to be compared in the deduplicated dataset into corresponding treemaps to generate a treemap to be compared; using pointers to compare the data to be compared in the treemap to obtain the comparison results; generating a target dataset based on the missing data in the comparison results and the deduplicated dataset to be compared; the target dataset includes the missing data and the data to be compared. Obtaining the deduplicated dataset to be compared and performing data comparison operations locally can optimize and improve data comparison speed and reduce server and database load by utilizing local resources, thus improving system robustness.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of big data technology, and in particular to a data comparison method, apparatus, device, storage medium and product. Background Technology

[0002] In traditional banking, accounting is a necessary part of financial work, requiring comparison of accounting data in the database and the acquisition of complete accounting data.

[0003] Currently, due to potential data gaps, data comparison is typically performed on a database to obtain complete accounting data. However, as the volume of data increases, direct data comparison and frequent access operations on the database place a significant burden on it, potentially causing database inaccessibility or even database crashes, and also impacting the efficiency of data comparison. Summary of the Invention

[0004] This application provides a data comparison method, apparatus, device, storage medium, and product to solve the problem that data comparison on a database may lead to others being unable to access the database or even causing the database to crash, and may also affect the efficiency of data comparison.

[0005] In a first aspect, this application provides a data comparison method, comprising: obtaining a deduplicated dataset to be compared; wherein the deduplicated dataset includes non-repeating data to be compared;

[0006] The data to be compared in the deduplicated dataset are stored into the corresponding treemap to generate the treemap to be compared.

[0007] A pointer is used to compare the data to be compared in the tree diagram to obtain the comparison result of the data to be compared; the comparison result of the data to be compared includes missing data.

[0008] A target dataset is generated based on the missing data in the comparison results of the data to be compared and the deduplicated dataset to be compared; the target dataset includes the missing data and the data to be compared.

[0009] Optionally, obtaining the deduplicated dataset to be compared includes:

[0010] Obtain the original dataset to be compared and determine the hash modulo value; use the hash modulo value to determine the number of storage units to generate the corresponding number of storage units; take the remainder of the hash value of the data to be compared in the original dataset with the hash modulo value, and store the data to be compared into the corresponding storage unit according to the remainder result; use multi-threaded concurrent traversal to traverse the data to be compared in each storage unit and put the data to be compared into the hash set corresponding to the storage unit; obtain the deduplicated dataset to be compared from the hash set.

[0011] Optionally, the data comparison method provided in the first aspect also includes:

[0012] If a certain piece of data to be compared cannot be stored in the hash set, then the data to be compared is stored in the duplicate storage unit.

[0013] Optionally, obtaining the original dataset to be compared and determining the hash modulo value includes:

[0014] Extract the original dataset to be compared from the database to be compared; determine the hash modulo value based on the data volume of the original dataset.

[0015] Optionally, the step of storing the data to be compared in the deduplicated dataset into corresponding treemaps to generate treemaps for comparison includes:

[0016] Store the data to be compared in the deduplicated dataset into multiple hashmaps; generate the same number of tree diagrams as the hashmaps, store the data to be compared in the hashmaps into the corresponding tree diagrams, and generate the tree diagrams to be compared.

[0017] Optionally, there may be multiple hash maps, and the step of storing the data to be compared in the hash maps into the corresponding tree diagrams to generate the tree diagrams to be compared includes:

[0018] By using multi-threaded concurrency, the data to be compared in the hash graph is stored into the corresponding tree graph, generating the tree graph to be compared.

[0019] Optionally, the data to be compared in the deduplicated dataset can be stored in multiple hashmaps, including:

[0020] Generate key-value pairs of data to be compared and determine the number of hash maps; store the key-value pairs of data to be compared into the corresponding hash maps.

[0021] Optionally, the comparison results of the data to be compared include inconsistent data, and the step of using pointers to compare the data to be compared in the tree diagram to obtain the comparison results includes:

[0022] Pointers are used to compare the data to be compared in the tree diagrams to be compared; if it is determined that the keys in the key-value pairs of the data to be compared pointed to by the pointers of the tree diagrams to be compared are the same but the values ​​are different, then the key-value pairs of the data to be compared pointed to by the pointers are determined to be inconsistent data, and the inconsistent data is stored in the inconsistent data storage unit; the pointers of the tree diagrams to be compared are moved down to compare the key-value pairs of the data to be compared pointed to by the pointers after the movement.

[0023] Optionally, the comparison result of the data to be compared includes missing data, and the step of using pointers to compare the data to be compared in the tree diagram to obtain the comparison result of the data to be compared further includes:

[0024] If it is determined that the keys in the key-value pairs to be compared are different when the pointers of the tree diagrams to be compared point to each other, then the size of the keys pointed to by the pointers is compared; the key-value pairs to be compared with smaller keys are identified as missing data and the missing data is stored in the missing data storage unit; the pointers pointing to smaller keys are moved down to compare the key-value pairs to be compared after the pointers are moved down.

[0025] Secondly, this application provides a data comparison device, comprising:

[0026] The acquisition module is used to acquire the deduplicated dataset to be compared; the deduplicated dataset includes non-repeating data to be compared.

[0027] The sorting module is used to store the data to be compared in the deduplicated dataset into the corresponding treemap, and generate the treemap to be compared.

[0028] The comparison module is used to compare the data to be compared in the tree diagram using pointers, and to obtain the comparison result of the data to be compared; the comparison result of the data to be compared includes missing data.

[0029] The generation module is used to generate a target dataset based on the missing data in the comparison results of the data to be compared and the deduplicated dataset to be compared; the target dataset includes the missing data and the data to be compared.

[0030] Thirdly, this application provides an electronic device, including: a processor, and a memory and a transceiver communicatively connected to the processor;

[0031] The memory stores computer-executed instructions; the transceiver is used for sending and receiving data.

[0032] The processor executes computer execution instructions stored in the memory to implement the data comparison method described in any of the above aspects.

[0033] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the data comparison method described in any of the preceding aspects.

[0034] Fifthly, this application provides a computer program product, including computer execution instructions, which, when executed by a processor, implement the data comparison method described in any of the preceding aspects.

[0035] This application provides a data comparison method, apparatus, device, storage medium, and product that obtains a deduplicated dataset to be compared. The deduplicated dataset includes unique data to be compared. The data to be compared in the deduplicated dataset is stored in corresponding treemaps to generate a treemap for comparison. Pointers are used to compare the data to be compared in the treemap to obtain the comparison results. A target dataset is generated based on the missing data in the comparison results and the deduplicated dataset. Obtaining the deduplicated dataset and performing the data comparison operation locally can optimize and improve the data comparison speed using local resources, reduce the burden on servers and databases, and further enhance the robustness of the system. Attached Figure Description

[0036] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0037] Figure 1 This is a schematic diagram of an application scenario provided by an embodiment of this application;

[0038] Figure 2 A flowchart illustrating the data comparison method provided in this application embodiment;

[0039] Figure 3 A flowchart illustrating another data comparison method provided in this application embodiment;

[0040] Figure 4 This is a schematic diagram of the structure of the data comparison device provided in the embodiments of this application;

[0041] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0042] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0043] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0044] The terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. In the following descriptions of embodiments, "a plurality of" means two or more, unless otherwise explicitly defined.

[0045] First, the prior art involved in this invention will be described and analyzed in detail.

[0046] In traditional banking, accounting is an essential part of financial work. As online transactions increase and the demand for improved accounting efficiency grows, performing accounting directly on the database and frequent access operations can place a huge burden on the database, potentially causing others to be unable to access it or even causing the database to crash.

[0047] The inventors discovered in their research that data can be loaded locally, and operations performed on the local data can then be synchronized to the database, thereby reducing the database load and ensuring system robustness. Therefore, this application provides a data comparison method, comprising: after obtaining a deduplicated dataset to be compared, storing the data to be compared in the deduplicated dataset into corresponding treemaps to generate a treemap to be compared; using pointers to compare the data to be compared in the treemap to obtain the comparison results; and generating a target dataset based on the missing data in the comparison results and the deduplicated dataset to be compared.

[0048] Figure 1 This is a schematic diagram of an application scenario provided in an embodiment of this application, such as... Figure 1 As shown, the system includes a database to be compared 1 and an electronic device 2. The electronic device 2 can obtain the original dataset A and the original dataset B to be compared from the database to be compared 1, and perform deduplication operations on the original dataset A and the original dataset B respectively. After deduplication of the original datasets, the electronic device 2 can execute the data comparison method provided in the embodiments of this application to compare the data to be compared, obtain the comparison results of the data to be compared, and generate a target dataset based on the missing data in the comparison results of the data to be compared and the deduplicated dataset to be compared.

[0049] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with relevant laws, regulations and standards, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0050] It should be noted that the data comparison method, apparatus, equipment, storage medium and product of this application can be used in the field of big data technology, or in any field other than big data technology. The application field of the data comparison method, apparatus, equipment, storage medium and product of this application is not limited.

[0051] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0052] Figure 2 This is a flowchart illustrating a data comparison method provided in an embodiment of this application. This embodiment addresses the problem that data comparison on a database may lead to unauthorized access to the database or even database crashes, and may also affect the efficiency of data comparison. The method in this embodiment is applied to a data comparison device, which can be located in an electronic device. The electronic device can be a digital computer of various forms, such as a laptop computer, desktop computer, workbench, personal digital assistant, server, blade server, mainframe computer, and other suitable computers.

[0053] like Figure 2 As shown, the specific steps of this method are as follows:

[0054] Step S101: Obtain the deduplicated dataset to be compared. The deduplicated dataset includes unique data to be compared. The data to be compared can be records from the dataset to be compared.

[0055] In this embodiment, there are multiple deduplicated datasets to be compared. The data to be compared is compared among the deduplicated datasets.

[0056] Step S102: Store the data to be compared in the deduplicated dataset into the corresponding treemap to generate the treemap to be compared.

[0057] For example, the data to be compared in the first deduplicated dataset is stored in the corresponding first tree diagram, and the data to be compared in the second deduplicated dataset is stored in the corresponding second tree diagram. There can be multiple tree diagrams or one tree diagram, and this application does not limit this.

[0058] Step S103: Use pointers to compare the data to be compared in the tree diagram to obtain the comparison results.

[0059] Specifically, the original dataset to be compared is deduplicated beforehand. After obtaining the deduplicated dataset to be compared, it is stored locally, and the comparison operation of the data to be compared in the deduplicated dataset is performed locally.

[0060] For example, in the financial field, when reconciling accounts, the datasets to be compared can be a host dataset and a UnionPay dataset. It is necessary to compare each piece of data to be compared in the host dataset and the UnionPay dataset and obtain the comparison results of the data to be compared.

[0061] It should be noted that a treemap is a key-value pair hash table ordered by key, based on the natural order of the keys. After storing the data to be compared from the deduplicated dataset into their respective treemaps, the treemaps then store the data to be compared in order.

[0062] Since the data to be compared is stored sequentially in the tree diagrams, it is convenient to compare the data in the tree diagrams to obtain the comparison results.

[0063] Step S104: Generate the target dataset based on the missing data in the comparison results of the data to be compared and the deduplicated dataset to be compared.

[0064] The comparison results of the data to be compared include missing data. The target dataset includes both missing data and the data to be compared. It should be understood that the deduplicated datasets to be compared may all contain missing data, which needs to be supplemented based on another deduplicated dataset to be compared in order to obtain the target dataset.

[0065] In this application embodiment, there is no limitation on the method of generating the target dataset based on the missing data in the comparison results of the data to be compared and the deduplicated dataset to be compared.

[0066] For example, missing data can be added to the deduplicated dataset, and deduplication can be performed on the deduplicated dataset after adding the missing data to obtain the target dataset.

[0067] The data comparison method provided in this application involves obtaining a deduplicated dataset to be compared; the deduplicated dataset includes unique data to be compared; storing the data to be compared in the deduplicated dataset into corresponding treemaps to generate treemaps for comparison; using pointers to compare the data to be compared in the treemaps to obtain the comparison results; and generating a target dataset based on the missing data in the comparison results and the deduplicated dataset. Obtaining the deduplicated dataset and performing the data comparison operation locally can optimize and improve the data comparison speed using local resources, reduce the burden on servers and databases, and further enhance the robustness of the system.

[0068] This application embodiment also provides a dataset deduplication method prior to step S101, which may include:

[0069] Step S201: Obtain the original dataset to be compared and determine the hash modulo value.

[0070] Step S202: Determine the number of storage units by the hash modulo value to generate the corresponding number of storage units.

[0071] Step S203: Take the remainder of the hash value of the data to be compared in the original dataset by taking the modulo value of the hash value, and store the data to be compared into the corresponding storage unit according to the remainder result.

[0072] Step S204: By using multi-threaded concurrent traversal of the data to be compared in each storage unit, the data to be compared is placed into the hash set corresponding to the storage unit.

[0073] Step S205: If it is impossible to store a certain data to be compared into a hash set, then store the data to be compared into a duplicate storage unit.

[0074] Step S206: Obtain the deduplicated dataset to be compared from the hash set.

[0075] In this embodiment, the original dataset to be compared is deduplicated by using a hash modulo algorithm and the characteristics of hash sets.

[0076] The database to be compared is the database that stores the original datasets to be compared. The original datasets to be compared can be datasets from a single database or datasets from different databases. The storage unit can be a file.

[0077] Specifically, one method for deduplicating datasets is as follows: After obtaining the original datasets to be compared, a set hash modulo value n can be obtained, or the hash modulo value n can be generated based on the data volume of the original datasets to be compared. After determining the hash modulo value n, n storage units are generated, and a thread pool is established. Based on the remainder of the hash modulo value, the data to be compared is stored in the corresponding storage unit. Through multi-threaded concurrency, the data in each of the n storage units is placed into n corresponding hash sets. The uniqueness of elements in the hash set is used to generate the deduplicated dataset to be compared, and duplicate data to be compared is retrieved.

[0078] A hash set guarantees the uniqueness of elements by: obtaining the hash value of the data to be compared; determining the storage location of the data in the hash set based on the hash value; if the storage location already contains the data to be compared, comparing it with the data to be stored; if they are different, storing the data to be stored in that location; if they are the same, not storing the data to be stored in the hash set, but storing it in a duplicate storage unit. Duplicate data to be compared can be retrieved from the duplicate storage unit, allowing users to find and identify duplicate data.

[0079] Optionally, the original dataset to be compared can be extracted from the database to be compared, and the hash modulo value can be determined based on the data volume of the original dataset.

[0080] For example, the mapping relationship between each data volume and the hash modulo value can be stored in advance. After obtaining the original dataset to be compared, the data volume of the original dataset to be compared is determined, and the hash modulo value to be used is determined according to the mapping relationship between each data volume and the hash modulo value.

[0081] In this embodiment, the original dataset to be compared can be extracted from the database to be compared, and the hash modulo value can be determined according to the data volume of the original dataset. This can reduce the steps of manual operation and increase the efficiency of data comparison.

[0082] Based on the above embodiments, this embodiment further involves a refinement of the implementation of step S102, which involves storing the data to be compared in the deduplicated dataset into the corresponding treemap to generate the treemap to be compared. Specifically, it includes the following steps:

[0083] Step S301: Store the data to be compared in the deduplicated dataset into multiple hashmaps.

[0084] Optionally, key-value pairs of data to be compared can be generated and the number of hash maps can be determined; the key-value pairs of data to be compared can be stored in the corresponding hash maps.

[0085] It should be understood that a hash map, also known as a hash map, is a collection used to store key-value pairs.

[0086] In this embodiment, the number of hash maps can be determined based on the data volume of the deduplicated dataset. The method of storing the key-value pairs of data to be compared into the corresponding hash map is similar to the method of storing the data to be compared into the corresponding storage unit; the corresponding hash map can be determined based on the result of taking the modulo of the hash value, which will not be elaborated further in this embodiment.

[0087] In this embodiment, the field that uniquely identifies a piece of data can be determined as the key in a key-value pair, and the remaining fields in a piece of data can be determined as the values ​​in the key-value pairs. For example, the last digit of the serial number can be determined as the key in the key-value pair, and the accounting information can be determined as the value in the key-value pair.

[0088] In this embodiment of the application, after storing the data to be compared in key-value pair form into the corresponding hash map, the data to be compared in key-value pair form in the hash map can be stored into the corresponding tree diagram. It can be sorted according to the key, which facilitates data comparison and makes it easier to distinguish the types of data that cannot be matched.

[0089] Step S302: Generate the same number of tree diagrams as the hash diagram, store the data to be compared in the hash diagram into the corresponding tree diagram, and generate the tree diagram to be compared.

[0090] The data comparison method provided in this application stores the data to be compared in the deduplicated dataset into multiple hashmaps, which facilitates the classification of the data to be compared in the deduplicated dataset to determine the corresponding hashmap and treemap. The same number of treemaps as the hashmaps are generated, and the data to be compared in the hashmaps is stored in the corresponding treemaps. The structural characteristics of the treemap can be used to sort the data to be compared by key size, which facilitates data comparison and improves the efficiency of data comparison.

[0091] Optionally, the data to be compared in the hash graph can be stored into the corresponding tree graph through multi-threaded concurrency to generate the tree graph to be compared.

[0092] In this embodiment, the number of hash graphs and tree graphs can be multiple. Multiple threads can be used concurrently to store the data to be compared from multiple hash graphs into their corresponding tree graphs, generating multiple tree graphs to be compared. This improves the efficiency of generating multiple tree graphs to be compared.

[0093] Based on the above embodiments, this embodiment further involves a refinement of the implementation of step S103, which uses pointers to compare the data to be compared in the tree diagram to obtain the comparison result of the data to be compared. Specifically, it includes the following steps:

[0094] Step S401: Check whether the key-value pairs of data to be compared are the same, as pointed to by the pointers of each tree diagram.

[0095] In this embodiment of the application, if the keys and / or values ​​of the data to be compared in the form of key-value pairs pointed to by the pointers are different, it is determined that the data to be compared are different.

[0096] For example, it can be first determined whether the keys in the key-value pairs to be compared are the same; if the keys are different, steps S404 to S406 are executed to store the different data to be compared and move the pointer down; if the keys are determined to be the same, it can be determined whether the values ​​in the key-value pairs to be compared are the same; if the values ​​are determined to be different, steps S402 and S403 are executed to store the different data to be compared and move the pointer down; if the values ​​are determined to be the same, step S403 is executed to move the pointer down.

[0097] Step S402: If it is determined that the values ​​in the key-value pairs of the comparison data pointed to by the pointers of the tree diagrams to be compared are different but the keys are the same, then the key-value pairs of the comparison data pointed to by the pointers are determined to be inconsistent data, and the inconsistent data is stored in the inconsistent data storage unit.

[0098] Step S403: Move the pointers of each tree diagram to be compared down to compare the key-value pairs of data that the pointers point to after the downshift.

[0099] In this embodiment, data to be compared that have different values ​​but the same key are identified as inconsistent data and stored in the inconsistent data storage unit. This allows users to know that the type of data that cannot be matched is inconsistent data. Users can determine the exact data within the inconsistent data and record it in the target dataset.

[0100] Step S404: If it is determined that the keys in the key-value pairs to be compared are different when the pointers of the tree diagrams to be compared point to each other, then compare the size of the keys pointed to by the pointers.

[0101] Step S405: Determine the key-value pair data to be compared as missing data and store the missing data in the missing data storage unit.

[0102] Step S406: Move the pointer pointing to the smaller key down to compare the key-value pairs of the data to be compared after the pointers have moved down.

[0103] In this embodiment of the application, the data to be compared with different keys are identified as missing data and stored in the missing data storage unit, so that users can know that the type of the data that cannot be matched is missing data.

[0104] The data comparison method provided in the above embodiments will be illustrated below with a specific example. Figure 3 A flowchart of another data comparison method provided in the embodiments of this application is shown below. Figure 3 As shown, it includes the following steps:

[0105] Step 1: Extract the host dataset and UnionPay dataset from their respective comparison databases to your local machine. The extracted host dataset and UnionPay dataset are the original datasets to be compared.

[0106] Step 2: Iterate through each piece of data in the host dataset and the UnionPay dataset respectively, perform a hash operation on each piece of data, take the modulo n of the hash value, and generate n small files. The small files serve as storage units.

[0107] Step 3: Create a new hash set for each small file.

[0108] Step 4: Iterate through each piece of data in the small file and put it into the corresponding hash set according to the remainder result.

[0109] Step 5: If it is determined that the data in Step 4 is duplicated with existing data, then this data is added to the duplicate file; otherwise, it is added to the hash set.

[0110] Step 6: Generate the host data deduplication file and the UnionPay data deduplication file based on the operations in Steps 4 and 5.

[0111] Step 7: Generate key-value pairs of data to be compared using the last digit of the serial number and the account information as key-value pairs, and store them in a hash graph. The data to be compared includes the serial number and the account information.

[0112] Step 8: Sort the key-value pairs of data to be compared using the structural characteristics of the tree diagram.

[0113] Step 9: Set two pseudo-pointers to traverse and compare the ascending tree diagram. Record transactions with inconsistent serial numbers to the corresponding missing file, and record transactions with consistent serial numbers but inconsistent other transaction data to the inconsistent file.

[0114] Step 10: Obtain data from missing and inconsistent files.

[0115] Step 11: Generate a reconciliation error data file based on the data in the missing and inconsistent files.

[0116] Figure 4 This is a schematic diagram of the data comparison device provided in an embodiment of this application. The data comparison device provided in this embodiment can execute the processing flow provided in the data comparison method embodiment. Figure 4 As shown, the data comparison device 50 includes: an acquisition module 501, a sorting module 502, a comparison module 503, and a generation module 504.

[0117] Specifically, the acquisition module 501 is used to acquire the deduplicated dataset to be compared; the deduplicated dataset includes non-repeating data to be compared.

[0118] The sorting module 502 is used to store the data to be compared in the deduplicated dataset into the corresponding treemap and generate the treemap to be compared.

[0119] The comparison module 503 is used to compare the data to be compared in the tree diagram using pointers, and obtain the comparison result of the data to be compared; the comparison result of the data to be compared includes missing data.

[0120] Generation module 504 is used to generate a target dataset based on the missing data in the comparison results of the data to be compared and the deduplicated dataset to be compared; the target dataset includes the missing data and the data to be compared.

[0121] The apparatus provided in this application embodiment can be specifically used to execute the method embodiment provided in Embodiment 1 above, and the specific functions will not be repeated here.

[0122] Optionally, the data comparison device 50 further includes: a deduplication module; the deduplication module is used for:

[0123] Obtain the original dataset to be compared and determine the hash modulo value; use the hash modulo value to determine the number of storage units to generate the corresponding number of storage units; take the remainder of the hash value of the data to be compared in the original dataset with the hash modulo value, and store the data to be compared into the corresponding storage unit according to the remainder result; use multi-threaded concurrent traversal to traverse the data to be compared in each storage unit and put the data to be compared into the hash set corresponding to the storage unit; obtain the deduplicated dataset to be compared from the hash set.

[0124] Optionally, the deduplication module is also used to: if a certain data to be compared cannot be stored in the hash set, then store the data to be compared in the duplicate storage unit.

[0125] Optionally, the deduplication module is specifically used to: extract the original dataset to be compared from the database to be compared; and determine the hash modulo value based on the data volume of the original dataset.

[0126] Optionally, the sorting module 502 is specifically used to: store the data to be compared in the deduplicated dataset into multiple hashmaps; generate the same number of tree diagrams as the hashmaps, store the data to be compared in the hashmaps into the corresponding tree diagrams, and generate the tree diagrams to be compared.

[0127] Optionally, there are multiple hash maps, and the sorting module 502 includes a tree map storage unit; the tree map storage unit is used to: store the data to be compared in the hash map into the corresponding tree map through multi-threaded concurrency, and generate the tree map to be compared.

[0128] Optionally, the sorting module 502 includes: a hash map storage unit; the hash map storage unit is used to: generate key-value pair data to be compared and determine the number of hash maps; and store the key-value pair data to be compared into the corresponding hash map.

[0129] Optionally, the comparison module 503 is specifically used to: compare the data to be compared in the tree diagrams to be compared using pointers; if it is determined that the keys in the key-value pairs of the data to be compared pointed to by the pointers of each tree diagram are the same but the values ​​are different, then the key-value pairs of the data to be compared pointed to by the pointers are determined to be inconsistent data, and the inconsistent data is stored in the inconsistent data storage unit; and the pointers of each tree diagram to be compared are moved down to compare the key-value pairs of the data to be compared pointed to by each pointer after the down movement.

[0130] Optionally, the comparison module 503 is further configured to: if it is determined that the keys in the key-value pairs to be compared pointed to by the pointers of each tree diagram to be compared are different, then compare the size of the keys pointed to by the pointers; determine the key-value pairs to be compared with smaller keys as missing data, and store the missing data in the missing data storage unit; move the pointers pointing to smaller keys down, so as to compare the key-value pairs to be compared pointed to by each pointer after the down-moving.

[0131] The apparatus provided in this application embodiment can be specifically used to execute the above method embodiments, and its specific functions will not be described in detail here.

[0132] Figure 5 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application, such as... Figure 5 As shown, this application also provides an electronic device 60, including: a processor 601, and a memory 602 and a transceiver 603 communicatively connected to the processor 601. The memory 602 stores computer-executable instructions; the transceiver 603 is used for sending and receiving data; the processor 601 executes the computer-executable instructions stored in the memory 602 to implement the method provided in any embodiment of this application.

[0133] Specifically, the program may include program code, which includes computer-executable instructions. Memory 602 may include high-speed RAM, and may also include non-volatile memory, such as at least one disk storage device. The computer-executable instructions are stored in memory 602 and configured to be executed by processor 601 to implement the method provided in any embodiment of this application. Related descriptions can be understood by referring to the relevant descriptions and effects corresponding to the steps in the accompanying drawings, and will not be elaborated upon here.

[0134] In this embodiment, the memory 602 and the processor 601 are connected via a bus. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 5 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0135] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method provided in any embodiment of this application.

[0136] This application also provides a computer program product, including computer execution instructions, which, when executed by a processor, implement the method provided in any embodiment of this application.

[0137] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0138] The modules described as separate components may or may not be physically separate. Similarly, the components shown as modules may or may not be physical modules; they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0139] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated module can be implemented in hardware or in a combination of hardware and software functional modules.

[0140] The program code used to implement the methods of this application may be written in any combination of one or more programming languages. This program code may be provided to the processor or controller of a general-purpose computer, special-purpose computer, or other programmable full-path fusion device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0141] In the context of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0142] Furthermore, although the operations are described in a specific order, this should be understood as requiring that such operations be performed in the specific order shown or in sequential order, or requiring that all illustrated operations be performed to achieve the desired result. In certain environments, multitasking and parallel processing may be advantageous. Similarly, although several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this application. Certain features described in the context of individual embodiments may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented individually or in any suitable sub-combination in multiple implementations.

[0143] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0144] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A data comparison method applied to electronic devices, characterized in that, include: Obtain the deduplicated dataset to be compared; the deduplicated dataset includes unique data to be compared. The data to be compared in the deduplicated dataset are stored into the corresponding treemaps to generate the treemaps to be compared; wherein the data to be compared are stored in the treemaps to be compared in order. A pointer is used to compare the data to be compared in the tree diagram to obtain the comparison result of the data to be compared; the comparison result of the data to be compared includes missing data. A target dataset is generated based on the missing data in the comparison results of the data to be compared and the deduplicated dataset to be compared; the target dataset includes the missing data and the data to be compared. Pointers are used to compare the data to be compared in the tree diagrams to obtain the comparison results, including: Determine whether the keys in the key-value pairs being compared, as pointed to by the pointers of each tree diagram, are the same; If the keys are different, compare the size of the keys pointed to by the pointers; The key-value pairs with smaller keys are identified as missing data, and the missing data is stored in the missing data storage unit. Move the pointer that points to the smaller key down to compare the key-value pairs that the pointers point to after the shift. If the keys are determined to be the same, then determine whether the values ​​in the key-value pairs being compared are the same. If the values ​​are determined to be different, the key-value pair to be compared pointed to by the pointer is determined to be inconsistent data, and the inconsistent data is stored in the inconsistent data storage unit. Move the pointers of each tree diagram to be compared down to compare the key-value pairs of data that the pointers point to after the downshift. If the values ​​are determined to be the same, the pointers of each tree diagram to be compared are moved down to compare the key-value pairs of data to be compared after the pointers are moved down. The process of obtaining the deduplicated dataset to be compared includes: Obtain the original dataset to be compared and determine the hash modulo value; The hash modulo value is used to determine the number of storage units to generate the corresponding number of storage units; The hash value of the data to be compared in the original dataset is moduloed by the hash value, and the data to be compared is stored in the corresponding storage unit based on the remainder result. The data to be compared is traversed concurrently in each storage unit by multiple threads, and placed into the hash set corresponding to the storage unit. The hash set ensures the uniqueness of elements by: obtaining the hash value of the data to be compared; determining the storage location of the data to be compared in the hash set based on the hash value; if the storage location is determined to already contain the data to be compared, then the data to be compared already stored in the storage location is compared with the data to be stored; if they are different, then the data to be compared is stored in the storage location; if they are the same, then the data to be compared is not stored in the hash set, but in the duplicate storage unit. Retrieve the deduplicated dataset to be compared from the hash set; If it is impossible to store a certain data to be compared into the hash set, then the data to be compared is stored into the duplicate storage unit; The number of tree diagrams is multiple. The step of storing the data to be compared in the deduplicated dataset into the corresponding tree diagram (treemap) to generate the tree diagram to be compared includes: The data to be compared in the deduplicated dataset is stored in multiple hashmaps; Generate the same number of tree diagrams as the hash diagram, store the data to be compared in the hash diagram into the corresponding tree diagram, and generate the tree diagram to be compared.

2. The method according to claim 1, characterized in that, The step of obtaining the original dataset to be compared and determining the hash modulo value includes: Extract the original dataset to be compared from the database to be compared; The hash modulo value is determined based on the data volume of the original dataset.

3. The method according to claim 1, characterized in that, The step of storing the data to be compared in the hash graph into the corresponding tree graph to generate the tree graph to be compared includes: By using multi-threaded concurrency, the data to be compared in the hash graph is stored into the corresponding tree graph, generating the tree graph to be compared.

4. The method according to claim 1, characterized in that, The step of storing the data to be compared in the deduplicated dataset into multiple hashmaps includes: Generate key-value pair data to be compared and determine the number of hash maps; Store the key-value pairs of data to be compared into the corresponding hash map.

5. A data comparison device, characterized in that, include: The acquisition module is used to acquire the deduplicated dataset to be compared; the deduplicated dataset includes non-repeating data to be compared. The sorting module is used to store the data to be compared in the deduplicated dataset into the corresponding treemap, and generate the treemap to be compared; wherein the data to be compared is stored in the treemap in order. The comparison module is used to compare the data to be compared in the tree diagram using pointers, and to obtain the comparison result of the data to be compared; the comparison result of the data to be compared includes missing data. The generation module is used to generate a target dataset based on the missing data in the comparison results of the data to be compared and the deduplicated dataset to be compared; the target dataset includes the missing data and the data to be compared. The comparison results of the data to be compared include inconsistent data. The comparison module is specifically used to determine whether the keys in the key-value pairs of the data to be compared, as pointed to by the pointers of each tree diagram, are the same. If the keys are different, compare the size of the keys pointed to by the pointers; The key-value pairs with smaller keys are identified as missing data, and the missing data is stored in the missing data storage unit. Move the pointer that points to the smaller key down to compare the key-value pairs that the pointers point to after the shift. If the keys are determined to be the same, then determine whether the values ​​in the key-value pairs being compared are the same. If the values ​​are determined to be different, the key-value pair to be compared pointed to by the pointer is determined to be inconsistent data, and the inconsistent data is stored in the inconsistent data storage unit. Move the pointers of each tree diagram to be compared down to compare the key-value pairs of data that the pointers point to after the downshift. If the values ​​are determined to be the same, the pointers of each tree diagram to be compared are moved down to compare the key-value pairs of data to be compared after the pointers are moved down. The deduplication module is used to obtain the original dataset to be compared and determine the hash modulo value; The hash modulo value is used to determine the number of storage units to generate the corresponding number of storage units; The hash value of the data to be compared in the original dataset is moduloed by the hash value, and the data to be compared is stored in the corresponding storage unit based on the remainder result. The data to be compared is traversed concurrently in each storage unit by multiple threads, and placed into the hash set corresponding to the storage unit. The hash set ensures the uniqueness of elements by: obtaining the hash value of the data to be compared; determining the storage location of the data to be compared in the hash set based on the hash value; if the storage location is determined to already contain the data to be compared, then the data to be compared already stored in the storage location is compared with the data to be stored; if they are different, then the data to be compared is stored in the storage location; if they are the same, then the data to be compared is not stored in the hash set, but in the duplicate storage unit. Retrieve the deduplicated dataset to be compared from the hash set; If it is impossible to store a certain data to be compared into the hash set, then the data to be compared is stored into the duplicate storage unit; The sorting module is specifically used to store the data to be compared in the deduplicated dataset into multiple hashmaps; Generate the same number of tree diagrams as the hash diagram, store the data to be compared in the hash diagram into the corresponding tree diagram, and generate the tree diagram to be compared.

6. An electronic device, characterized in that, include: A processor, and a memory and a transceiver communicatively connected to the processor; The memory stores computer-executed instructions; the transceiver is used for sending and receiving data. The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-4.

8. A computer program product comprising computer-executable instructions, characterized in that, When the computer execution instructions are executed by the processor, they implement the method as described in any one of claims 1-4.

Citation Information

Patent Citations

  • Data comparison method and device, medium and electronic equipment

    CN115510062A

  • Big data deduplication method and device, equipment and medium

    CN115631273A