Method, device and equipment for determining duplicated data

By using bit operations of merged computing nodes and bitmap arrays in the financial system, combined with the secondary dictionary check of local reverse dictionary, the problem of inefficient determination of duplicate data caused by manual inspection is solved, and efficient repetition in large data centers is achieved Data processing.

CN119025827APending Publication Date: 2024-11-26CHINA UNIONPAY
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411133960.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-16
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

In the prior art, manual line-by-line inspection of data results in inefficient and time-consuming determination of duplicate data in large data sets.

Method used

Based on the merged computing node, multi-machine duplication sets are obtained through bit operation of bitmap array and quadratic repetition of local reverse dictionary, and merged with a stand-alone duplication set to obtain a global duplication set, improving the efficiency and accuracy of duplicate data determination.

Benefits of technology

It realizes efficient determination of duplicate data in large data sets, reduces the waste of computing resources and storage resources, and improves processing efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119025827A_ABST
    Figure CN119025827A_ABST
Patent Text Reader

Abstract

The embodiment of the invention provides a method, a device and equipment for determining duplicated data, and relates to the field of big data. The method comprises the following steps: on the basis of merging computing nodes, obtaining bitmap arrays and single machine repetition sets of M computing nodes; performing pairwise bit operation processing on the bitmap array of the M computational nodes to obtain M repetition values; acquiring a local reverse dictionary corresponding to the computational node; according to each local reverse dictionary, performing secondary duplicate checking processing on the duplicate values to obtain a multi-machine duplicate set corresponding to the duplicate values; and performing merging processing on each multi-machine repetition set and each single-machine repetition set to obtain a global repetition set. According to the method, the data repetition searching efficiency is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of big data, and in particular to a method, device and equipment for determining duplicate data. Background Art

[0002] There is a large amount of duplicate data in the financial system. In order to reduce the waste of storage and computing resources caused by duplicate data, it is necessary to identify the duplicate data.

[0003] In the prior art, data is manually checked line by line, duplicate data is marked, and then deleted.

[0004] However, in the above method, manual processing is used for data sets with large data volumes, which is time-consuming and labor-intensive, thereby reducing the efficiency of determining duplicate data. Summary of the invention

[0005] The embodiments of the present application provide a method, apparatus and device for determining duplicate data, so as to solve the technical problem that the efficiency of determining duplicate data is reduced due to manual processing.

[0006] In a first aspect, an embodiment of the present application provides a method for determining duplicate data, including:

[0007] Based on the merged computing nodes, a bitmap array and a single-machine repetition set of M computing nodes are obtained; wherein the bitmap array represents a mapping processing record of each string representing business data in the computing node; the single-machine repetition set includes repeated strings; and M is an integer greater than or equal to 1;

[0008] Perform two-bit operations on the bitmap arrays of the M computing nodes to obtain M repeated values; wherein the repeated values ​​represent repeated mapping values ​​in every two computing nodes; and the mapping values ​​represent unique identifiers of business data;

[0009] Obtaining a local reverse dictionary corresponding to the computing node; and performing a secondary duplicate check on the duplicate value according to each local reverse dictionary to obtain a multi-machine duplicate set corresponding to the duplicate value; wherein the local reverse dictionary includes a reverse mapping relationship in which each mapping value is reversely mapped to a string; and the multi-machine duplicate set includes strings repeated in every two computing nodes;

[0010] Each of the multi-machine repetition sets and each of the single-machine repetition sets are merged to obtain a global repetition set; wherein the global repetition set includes repeated character strings in all computing nodes.

[0011] In a possible implementation, based on merging computing nodes, obtaining a bitmap array and a single-machine repetition set of M computing nodes includes:

[0012] Determine a target data source; and distribute a plurality of first data sets in the target data source to the M computing nodes; wherein the first data set includes at least one of the character strings;

[0013] Based on the computing node, each character string in the received first data set is checked for duplicates to obtain a single-machine duplicate set and a bitmap array corresponding to the computing node;

[0014] Based on the merged computing node, a bitmap array and a single-machine repetition set of the M computing nodes are obtained.

[0015] In a possible implementation, based on the computing node, each character string in the received first data set is checked for duplicates to obtain a single-machine duplicate set and a bitmap array corresponding to the computing node, including:

[0016] Based on the computing node, mapping processing is performed on each string in the first data set to obtain a mapping set, a local reverse dictionary and a bitmap array corresponding to the computing node; wherein the mapping set includes a mapping value corresponding to each string; the local reverse dictionary includes a reverse mapping relationship of each mapping value reversely mapped to a string;

[0017] The local reverse dictionary and the bitmap array corresponding to the computing node are reversely checked to obtain the single-machine duplicate set corresponding to the computing node.

[0018] In a possible implementation, mapping processing is performed on each string in the first data set based on the computing node to obtain a mapping set, a local reverse dictionary, and a bitmap array corresponding to the computing node, including:

[0019] Performing calculation processing on each character string in the first data set to obtain an integer set corresponding to the first data set; wherein the integer set includes an integer value corresponding to each character string; and the integer value represents a unique identifier of the business data;

[0020] Determine the integer value corresponding to the string as the mapping value corresponding to the string;

[0021] The local reverse dictionary and the bitmap array are determined according to each of the character strings and the mapping values ​​corresponding to the character strings.

[0022] In a possible implementation, performing calculation processing on each character string in the first data set to obtain an integer set corresponding to the first data set includes:

[0023] Encrypting each character string in the first data set to obtain an encrypted set corresponding to the first data set; wherein the encrypted set includes encrypted data corresponding to each character string;

[0024] A hash calculation is performed on each encrypted data in the encryption set to obtain the integer set.

[0025] In a possible implementation, performing reverse lookup processing on the local reverse dictionary and the bitmap array corresponding to the computing node to obtain the single-machine repetition set corresponding to the computing node includes:

[0026] Performing bitwise operation processing on the bitmap array corresponding to the computing node to obtain a mapping value set corresponding to the computing node; wherein the mapping value set includes at least one mapping subset corresponding to the computing node, and the mapping subset includes repeated mapping values;

[0027] According to the local reverse dictionary corresponding to the computing node, reverse query processing is performed on the mapping value set corresponding to the computing node to determine the original data set corresponding to the computing node; wherein the original data set includes a data subset corresponding to each of the mapping subsets; and the data subset includes a character string corresponding to each repeated mapping value in the mapping subset;

[0028] Performing comparison processing on each character string in the data subset to obtain a comparison result of the data subset;

[0029] If it is determined that the comparison result indicates that all character strings in the data subset are the same, then it is determined that all character strings in the data subset are character strings in the single-machine repetition set.

[0030] In a possible implementation manner, the target data source is stored in a single server; determining the target data source includes:

[0031] The target data source is determined from a plurality of data sources according to a data source configuration file, wherein the data source configuration file represents a configuration of a data source.

[0032] In a possible implementation, the data source configuration file includes server information and database information; the server information represents the server configuration of the data source; the database information includes a connection string and instance status of the database where the data source is located.

[0033] In a possible implementation, the target data source includes multiple business data; distributing the multiple first data sets in the target data source to the M computing nodes includes:

[0034] Performing data splitting processing on the target data source to obtain M second data sets corresponding to the target data source; wherein the second data set includes business data in at least one target data source;

[0035] Performing calculation processing on each business data in the second data set to obtain the first data set corresponding to the second data set;

[0036] The M first data sets are evenly distributed to the M computing nodes.

[0037] In a possible implementation manner, each of the multi-machine repetition sets and each of the single-machine repetition sets are merged to obtain a global repetition set, including:

[0038] Performing consistency processing on the multi-machine repeat set and the single-machine repeat set to obtain a processed multi-machine repeat set and a processed single-machine repeat set;

[0039] The processed multi-machine repetition sets and the processed single-machine repetition sets are merged to obtain the global repetition set.

[0040] In a possible implementation manner, merging the processed multi-machine repetition sets and the processed single-machine repetition sets to obtain the global repetition set includes:

[0041] Merging the processed multi-machine repeat set and two processed single-machine repeat sets corresponding to the processed multi-machine repeat set to obtain a first local repeat set corresponding to the processed multi-machine repeat set;

[0042] The first local repetition sets are merged to obtain the global repetition set.

[0043] In a possible implementation manner, merging the processed multi-machine repetition sets and the processed single-machine repetition sets to obtain the global repetition set includes:

[0044] Merging the processed multi-machine repeat sets to obtain second local repeat sets corresponding to the processed multi-machine repeat sets;

[0045] Merging the processed single-machine repetition sets to obtain third local repetition sets corresponding to the processed single-machine repetition sets;

[0046] The second local repetition set and the third local repetition set are combined to obtain the global repetition set.

[0047] In a possible implementation, performing a secondary duplicate check on the duplicate values ​​according to each of the local reverse dictionaries to obtain a multi-machine duplicate set corresponding to the duplicate values ​​includes:

[0048] Determine at least one character string corresponding to the repeated value according to each of the local reverse dictionaries;

[0049] If it is determined that the character strings corresponding to the repetition values ​​are all the same, then it is determined that the character strings corresponding to the repetition values ​​are character strings in the multi-machine repetition set corresponding to the repetition values.

[0050] In a possible implementation, the method further includes:

[0051] According to the bitmap arrays of the M computing nodes, deduplication processing is performed on each character string in the M computing nodes.

[0052] In a possible implementation, performing deduplication processing on each string in the M computing nodes according to the bitmap arrays of the M computing nodes includes:

[0053] According to the bitmap array and local reverse dictionary of the M computing nodes, deduplication calculation processing is performed on each string in the M computing nodes to obtain a non-duplicate data set; wherein the local reverse dictionary includes a reverse mapping relationship in which each mapping value is reversely mapped to a string; and the non-duplicate data set includes strings that are not repeated in the M computing nodes.

[0054] In a possible implementation, the method further includes:

[0055] According to the bitmap arrays and local reverse dictionaries of the M computing nodes, aggregate calculation processing is performed on each string in the M computing nodes to obtain the number of non-repetitive strings corresponding to the M computing nodes; wherein the number of non-repetitive strings is the total number of different strings in the M computing nodes.

[0056] In a second aspect, an embodiment of the present application provides a device for determining duplicate data, including:

[0057] An acquisition module, used to acquire a bitmap array and a single-machine repetition set of M computing nodes based on the merged computing node; wherein the bitmap array represents a mapping processing record of each string representing business data in the computing node; the single-machine repetition set includes repeated strings; M is an integer greater than or equal to 1;

[0058] A first processing module is used to perform two-bit operation processing on the bitmap arrays of the M computing nodes to obtain M repeated values; wherein the repeated values ​​represent repeated mapping values ​​in every two computing nodes; and the mapping values ​​represent unique identifiers of business data;

[0059] The second processing module is used to obtain the local reverse dictionary corresponding to the computing node; and according to each of the local reverse dictionaries, perform a secondary duplicate check on the repeated value to obtain a multi-machine duplicate set corresponding to the repeated value; wherein the local reverse dictionary includes a reverse mapping relationship in which each mapping value is reversely mapped to a string; and the multi-machine duplicate set includes the repeated strings in every two computing nodes;

[0060] The merging module is used to merge the multi-machine repetition sets and the single-machine repetition sets to obtain a global repetition set; wherein the global repetition set includes repeated character strings in all computing nodes.

[0061] In a third aspect, an embodiment of the present application provides a computing device, including: a memory, a processor;

[0062] The memory stores computer-executable instructions;

[0063] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the above first aspect and / or various possible implementations of the first aspect.

[0064] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the first aspect above and / or various possible implementations of the first aspect.

[0065] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the above first aspect and / or various possible implementation methods of the first aspect.

[0066] The method, apparatus and device for determining duplicate data provided in the embodiments of the present application are based on merging computing nodes, and according to the bit operation processing process of the bitmap array of each two computing nodes, a secondary duplication check is performed on the duplicate mapping values ​​in the two computing nodes to obtain a multi-machine duplicate set, and the duplicate character strings in each computing node are merged and restored to obtain all the duplicate business data in each computing node; thereby, the efficiency and accuracy of determining duplicate data are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0068] Figure 1 A schematic diagram of an application scenario provided for this application;

[0069] Figure 2 A flowchart of a method for determining duplicate data provided in an embodiment of the present application;

[0070] Figure 3 A flowchart of another method for determining duplicate data provided in an embodiment of the present application;

[0071] Figure 4 A flowchart of a data distribution process provided in an embodiment of the present application;

[0072] Figure 5 A schematic diagram of a process of constructing a local reverse dictionary provided in an embodiment of the present application;

[0073] Figure 6 A schematic diagram of a secondary weight finding process provided in an embodiment of the present application;

[0074] Figure 7 A flowchart of another method for determining duplicate data provided in an embodiment of the present application;

[0075] Figure 8 A schematic diagram of a process of merging duplicate sets provided in an embodiment of the present application;

[0076] Fig. 9 A schematic diagram of an overall process for determining duplicate data provided in an embodiment of the present application;

[0077] Fig.10 A flowchart of another method for determining duplicate data provided in an embodiment of the present application;

[0078] Fig.11 A schematic diagram of the structure of a device for determining duplicate data provided in an embodiment of the present application;

[0079] Fig.12 A schematic diagram of the structure of a computing device provided in an embodiment of the present application.

[0080] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0081] Here, exemplary embodiments will be described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Instead, they are only examples of devices and methods consistent with some aspects of the present application.

[0082] First, the terms involved in this application are explained:

[0083] Local reverse dictionary (LRD): A mapping dictionary stored in each worker node that maps integer values ​​back to the original string.

[0084] Computing node: It is a generalized node. A thread on a machine can also be considered as an independent computing node.

[0085] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0086] It should be noted that the method, device and equipment for determining duplicate data of the present application can also be used in any field except big data, and the application field of the method, device and equipment for determining duplicate data of the present application is not limited.

[0087] There is a large amount of duplicate data in the financial system. In order to reduce the waste of storage and computing resources caused by duplicate data, it is necessary to identify the duplicate data.

[0088] In one example, data is manually checked line by line, duplicate data is marked, and then deleted.

[0089] However, in the above method, manual processing is used for data sets with large data volumes, which is time-consuming and labor-intensive, thereby reducing the efficiency of determining duplicate data.

[0090] In another example, based on the single-machine Set / Bitmap array duplicate search, the Set collection class naturally has the function of duplicate search. For some scenarios with smaller data volumes, single-machine Set duplicate search can be used directly; Bitmap is used to store data, using computer median storage, with 1 bit representing a number.

[0091] However, in the above methods, using Set and Bitmap to find weights is limited by the resources of a single machine and is difficult to process massive data.

[0092] In another example, based on the global dictionary + Bitmap deduplication search, the distributed database system maintains a global forward dictionary in advance, maps the original data record into a globally unique integer, and then stores the integer in the Bitmap for deduplication calculation.

[0093] However, the above method requires a complex operation to map a string to a globally unique integer, and the original data cannot be obtained from the mapping value; the amount of data is large, and there is concurrent competition; thus, the performance of finding the weight is significantly reduced.

[0094] In response to the above technical problems, the present application proposes the following technical concept: based on merging computing nodes, according to the bit operation processing process of the bitmap array of each two computing nodes, the repeated mapping values ​​in the two computing nodes are checked for duplicates twice to obtain a multi-machine duplicate set, and the repeated character strings in each computing node are merged and restored to obtain all the repeated business data in each computing node. In this way, the problem of low efficiency in determining duplicate data based on manual duplicate detection in the prior art can be solved.

[0095] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0096] Figure 1 A schematic diagram of an application scenario provided by this application. Figure 1 As shown, the specific application scenario of the present application includes a merged computing node 102 and multiple computing nodes 103 in the computing device 101. Through the merged computing node 102 in the computing device 101, the business data in the data center distributed to each computing node 103 is checked for duplicates to obtain duplicate business data.

[0097] Figure 2 A flowchart of a method for determining duplicate data provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, the method includes:

[0098] 201. Based on the merged computing nodes, obtain the bitmap array and the single-machine repetition set of M computing nodes; wherein the bitmap array represents the mapping processing record of each string representing the business data in the computing node; the single-machine repetition set includes repeated strings; and M is an integer greater than or equal to 1.

[0099] Exemplarily, the execution subject of this embodiment may be a computing device, hereinafter referred to as the device. The device may be a physical device or a virtual device that executes the method for determining duplicate data. A merged computing node and M distributed computing nodes are deployed in the device. In each computing node, each string representing business data in the computing node will be mapped, and the mapping processing records of each string will be stored in a bitmap array, such as storing all data records converted to numbers in a Bitmap or Roaring Bitmap. At the same time, based on the duplicate checking method of each computing node, the repeated strings in each computing node can be obtained and sorted into a single-machine duplicate set. Based on the merged computing node, the bitmap arrays and single-machine duplicate sets of all computing nodes are obtained for processing; wherein, M is an integer greater than or equal to 1.

[0100] 202. Perform two-bit operations on the bitmap arrays of the M computing nodes to obtain M repeated values; wherein the repeated values ​​represent repeated mapping values ​​in every two computing nodes; and the mapping values ​​represent unique identifiers of the business data.

[0101] Exemplarily, based on the characteristics of the bitmap array, the device performs two-bit operations on the bitmap array in the M computer nodes through bit operations, and can quickly obtain the repeated mapping values ​​in every two computing nodes in the M computer nodes. Each mapping value is the mapping value corresponding to the corresponding string mapping processing, so as to represent the unique identifier of the corresponding business data.

[0102] For example, for computing node 1, computing node 2, and computing node 3, bit operations are performed on bitmap array 1 of computing node 1 and bitmap array 2 of computing node 2 to obtain repeated values ​​1-2 corresponding to computing node 1 and computing node 2, bit operations are performed on bitmap array 1 of computing node 1 and bitmap array 3 of computing node 3 to obtain repeated values ​​1-3 corresponding to computing node 1 and computing node 3, and bitmap array 3 of computing node 3 and bitmap array 2 of computing node 2 to obtain repeated values ​​3-2 corresponding to computing node 3 and computing node 2.

[0103] 203. Obtain the local reverse dictionary corresponding to the computing node; and perform a secondary duplicate check on the duplicate values ​​according to each local reverse dictionary to obtain a multi-machine duplicate set corresponding to the duplicate values; wherein the local reverse dictionary includes a reverse mapping relationship in which each mapping value is reversely mapped to a string; and the multi-machine duplicate set includes the repeated strings in every two computing nodes.

[0104] Exemplarily, repeated mapping values ​​do not represent repeated source data. Therefore, the device performs a secondary duplicate check on the repeated values ​​corresponding to every two computing nodes, and can perform restoration calculation processing on each repeated value. Based on each computing node in the distributed deployment, there is a one-to-one correspondence between each string and each mapping value, and each mapping value in each computing node is reversely mapped to a reverse mapping relationship of the string, which is stored as a local reverse dictionary corresponding to each computing node. Based on the merged computing nodes, the local reverse dictionary corresponding to each computing node can be obtained. By using the reverse mapping relationship of each mapping value reversely mapped to a string, the corresponding local reverse dictionaries can be queried to obtain the repeated strings in every two computing nodes, that is, the corresponding multi-machine duplicate set.

[0105] 204. Each multi-machine repetition set and each single-machine repetition set are merged to obtain a global repetition set; wherein the global repetition set includes repeated character strings in all computing nodes.

[0106] Exemplarily, after obtaining M multi-machine duplicate sets and M single-machine duplicate sets, the device merges all the multi-machine duplicate sets and single-machine duplicate sets to obtain the final global duplicate set, that is, the repeated character strings in all computing nodes can be obtained to complete the data duplicate search process for all computing nodes.

[0107] In this embodiment, a method for determining duplicate data is provided. Based on merging computing nodes, the repeated mapping values ​​in each two computing nodes are checked for duplicates according to the bit operation processing process of the bitmap array of each two computing nodes and the local reverse dictionary to obtain a multi-machine duplicate set, and the duplicate character strings in each computing node are merged and processed to obtain all the duplicate business data in each computing node; thereby, the efficiency and accuracy of determining duplicate data are improved.

[0108] Figure 3 A flow chart of another method for determining duplicate data provided in an embodiment of the present application is shown as follows: Figure 3 As shown, the method includes:

[0109] 301. Determine a target data source; and distribute multiple first data sets in the target data source to M computing nodes; wherein the first data set includes at least one character string.

[0110] Exemplarily, the device first obtains a target data source that needs to be processed for duplicate detection, where the target data source includes multiple data sets, namely, first data sets, each of which includes at least one character string. Then, the device distributes all first data sets in the target data source to M computing nodes that are distributed. For example, computing nodes with abundant computing power resources can be distributed to more first data sets to improve the efficiency of subsequent computing processing.

[0111] In one example, the target data source is stored in a single server, and determining the target data source in step 301 includes: determining the target data source from multiple data sources according to a data source configuration file; wherein the data source configuration file represents the configuration of the data source.

[0112] In one example, the data source configuration file includes server information and database information; the server information represents the server configuration of the data source; the database information includes the connection string and instance status of the database where the data source is located.

[0113] Specifically, if the target data source processed by the device is a data source stored in a single server, the acquired data source needs to be processed to determine whether the data source is a data source stored in a single server. The device then parses each acquired data source to determine the configuration status of the data source, i.e., the data source configuration file, and determines the target data source from multiple data sources based on the data source configuration file of each data source, wherein each data source configuration file includes server information and database information; the server information represents the server configuration status of the data source; and the database information includes the connection string and instance status of the database where the data source is located.

[0114] For example, the target data source can be determined by checking database configuration files, viewing database connection strings, using database management tools, checking server configurations, operating system commands, viewing database instance status, log files, etc.

[0115] In one example, distributing the multiple first data sets in the target data source to M computing nodes in step 301 includes:

[0116] Step 1: perform data splitting processing on the target data source to obtain M second data sets corresponding to the target data source; wherein the second data set includes business data in at least one target data source.

[0117] Step 2: Calculate and process each business data in the second data set to obtain the first data set corresponding to the second data set.

[0118] Step 3: Distribute the M first data sets evenly to the M computing nodes.

[0119] Specifically, Figure 4 A flow chart of a data distribution process provided in an embodiment of the present application is as follows: Figure 4As shown, the business data in the target data source is first equally split into M data sets corresponding to the target data source, namely, second data sets, each second data set includes at least one business data in the target data source, and each business data in each second data set is calculated and processed by a preset hash calculation method to obtain a character string corresponding to each business data, namely, a first data set corresponding to the M second data sets; then, the device evenly distributes the M first data sets to the M computing nodes in a distributed deployment.

[0120] 302. Based on the computing node, perform a duplicate check on each character string in the received first data set to obtain a single-machine duplicate set and a bitmap array corresponding to the computing node.

[0121] Exemplarily, the device performs a duplicate checking process on each character string in the received data set based on each computing node and according to a preset duplicate checking algorithm to obtain the repeated character strings in the computing node, that is, to obtain the single-machine duplicate set of the computing node; and, for each character string in the data set, a mapping process is performed to obtain a mapped character string, and then, the mapping process of each character string is recorded and stored in the form of a bitmap array for processing.

[0122] In one example, step 302 includes the following steps:

[0123] The first step of step 302 is to map each string in the first data set based on the computing node to obtain a mapping set, a local reverse dictionary and a bitmap array corresponding to the computing node; wherein the mapping set includes a mapping value corresponding to each string; and the local reverse dictionary includes a reverse mapping relationship in which each mapping value is reversely mapped to a string.

[0124] The second step of step 302 is to perform reverse lookup processing on the local reverse dictionary and bitmap array corresponding to the computing node to obtain the single-machine duplicate set corresponding to the computing node.

[0125] Exemplarily, in a specific duplicate checking process, first, each computing node in the device performs mapping processing on each received string based on a preset mapping algorithm such as a hash algorithm to obtain a mapping value corresponding to each string, that is, a mapping set corresponding to the computing node. At the same time, each string is mapped to each corresponding mapping value. This mapping processing record is stored as a bitmap array, and then each mapping value is reversely mapped to the corresponding string. This reverse mapping relationship is stored in the local reverse dictionary of the computing node. The device performs computational processing on the mapping records represented by the bitmap array of each computing node and the local reverse dictionary based on the preset reverse query algorithm to obtain the repeated strings in the computing node; for example, a preset deep learning model can be called, and the mapping records represented by the bitmap array corresponding to each computing node and the local reverse dictionary can be input into the model for model processing to output the single-machine duplicate set of each computing node.

[0126] In one example, the first step of step 302 includes:

[0127] Step 1: Calculate and process each character string in the first data set to obtain an integer set corresponding to the first data set; wherein the integer set includes an integer value corresponding to each character string; and the integer value represents a unique identifier of the business data.

[0128] Step 2: Determine the integer value corresponding to the string, which is the mapping value corresponding to the string.

[0129] Step 3: Determine the local reverse dictionary and bitmap array according to each string and the mapping value corresponding to each string.

[0130] Specifically, Figure 5 A schematic diagram of a process of constructing a local reverse dictionary provided in an embodiment of the present application is shown in FIG. Figure 5 As shown, in the specific mapping process, based on each computing node, according to the preset class hash mapping algorithm, each string in the received data set is traversed in parallel, and each string is mapped to a number to obtain an integer value corresponding to each string, and then an integer set is obtained. Then, the integer value corresponding to each string is determined as the mapping value corresponding to the string, and the class hash mapping processing records between each string and the mapping value corresponding to each string are stored as a bitmap array, and the reverse mapping relationship of each mapping value in the process of reverse class hash mapping to the corresponding string is stored in the local reverse dictionary.

[0131] In one example, step one includes:

[0132] The first step is to encrypt each character string in the first data set to obtain an encrypted set corresponding to the first data set; wherein the encrypted set includes encrypted data corresponding to each character string.

[0133] The second step is to perform hash calculation on each encrypted data in the encryption set to obtain an integer set.

[0134] Specifically, in the process of mapping each string to an integer value, each computing node of the device can first encrypt each received string based on a preset encryption algorithm to obtain an encrypted set, including the encrypted data corresponding to each string, and then perform hash calculation processing on each encrypted data in the encrypted set based on a preset hash algorithm to obtain an integer set corresponding to the encrypted set.

[0135] In one example, the second step of step 302 includes:

[0136] Step 1: Perform bitwise operation on the bitmap array corresponding to the computing node to obtain a mapping value set corresponding to the computing node; wherein the mapping value set includes at least one mapping subset corresponding to the computing node, and the mapping subset includes repeated mapping values.

[0137] Step 2: According to the local reverse dictionary corresponding to the computing node, the mapping value set corresponding to the computing node is reversely checked to determine the original data set corresponding to the computing node; wherein the original data set includes a data subset corresponding to each mapping subset; and the data subset includes a character string corresponding to each repeated mapping value in the mapping subset.

[0138] Step 3: Perform comparison processing on each character string in the data subset to obtain the comparison result of the data subset.

[0139] Step 4: If it is determined that the comparison result indicates that all the character strings in the data subset are the same, then it is determined that all the character strings in the data subset are character strings in the single-machine repeated set.

[0140] Specifically, in the reverse query process, based on each computing node, according to the characteristics of the bitmap array, the bitmap array corresponding to the computing node is processed by bit operation, and multiple mapping subsets corresponding to the computing node can be obtained. Each mapping subset includes repeated mapping values, which can be sorted into corresponding mapping value sets. However, the same mapping value does not mean that the original data is also the same. Therefore, each computing node needs to perform reverse query processing on the repeated mapping values ​​corresponding to the computing node according to the local reverse dictionary of the computing node to obtain the string corresponding to each repeated mapping value, and then obtain each data subset in the corresponding original data set. Then, each string in each data subset is compared to obtain the comparison result of each data subset to determine whether the strings in each data subset are the same. If it is determined that the strings in each data subset are the same, it means that the duplication is confirmed, and the repeated strings in the data subset are stored in the source data duplication set, which is called a single machine duplication set.

[0141] For example, Figure 6 A schematic diagram of a secondary weight finding process provided in an embodiment of the present application is shown in FIG. Figure 6 As shown, for the character string S1 and the character string S2, the bitmap arrays corresponding to the two are processed by bit operation to obtain the same integer value V1 corresponding to the two, the integer value V1 is subjected to bit conflict analysis, and the corresponding local reverse dictionary is called to perform reverse query processing on V1 to obtain the original data S1 and S2 corresponding to V1, and then determine whether S1 and S2 are the same. If they are the same, S1 and S2 are stored in a single-machine duplicate set. If they are not the same, this step is skipped.

[0142] 303. Based on the merged computing nodes, obtain the bitmap arrays and single-machine repetition sets of the M computing nodes.

[0143] Exemplarily, the device obtains the bitmap arrays of all M computing nodes and the single-machine repetition set based on the merged computing nodes for processing.

[0144] 304. Perform two-bit operations on the bitmap arrays of the M computing nodes to obtain M repeated values; wherein the repeated values ​​represent repeated mapping values ​​in every two computing nodes; and the mapping values ​​represent unique identifiers of the business data.

[0145] Exemplarily, the content of this step can refer to step 202 and will not be repeated here.

[0146] 305. Obtain the local reverse dictionary corresponding to the computing node; and perform a second duplicate check on the duplicate values ​​according to each local reverse dictionary to obtain a multi-machine duplicate set corresponding to the duplicate values; wherein the local reverse dictionary includes a reverse mapping relationship in which each mapping value is reversely mapped to a string; and the multi-machine duplicate set includes the repeated strings in every two computing nodes.

[0147] Exemplarily, the content of this step can refer to step 203 and will not be repeated here.

[0148] 306. Merge each multi-machine repetition set and each single-machine repetition set to obtain a global repetition set; wherein the global repetition set includes repeated character strings in all computing nodes.

[0149] Exemplarily, the content of this step can refer to step 204 and will not be repeated here.

[0150] In this embodiment, based on the above embodiment, a distributed reverse local dictionary stores reverse mapping relationships, and the probability of reverse query can be adjusted through the conflict rate of the mapping function; and a distributed bitmap array is combined to store multi-machine data to realize parallel multi-machine operation and duplicate search, thereby improving the efficiency of determining duplicate data.

[0151] Figure 7A flowchart of another method for determining duplicate data provided in an embodiment of the present application is shown as follows: Figure 7 As shown, the method includes:

[0152] 401. Based on the merged computing nodes, obtain the bitmap array and the single-machine repetition set of M computing nodes; wherein the bitmap array represents the mapping processing record of each string representing the business data in the computing node; the single-machine repetition set includes the repeated string; and M is an integer greater than or equal to 1.

[0153] Exemplarily, the content of this step can refer to step 201 and will not be repeated here.

[0154] 402. Perform two-bit operations on the bitmap arrays of the M computing nodes to obtain M repeated values; wherein the repeated values ​​represent repeated mapping values ​​in every two computing nodes; and the mapping values ​​represent unique identifiers of the business data.

[0155] Exemplarily, this step may refer to step 202 and will not be described in detail.

[0156] 403. Determine at least one character string corresponding to the repeated value according to each local reverse dictionary.

[0157] Exemplarily, based on each computing node in the distributed deployment, there is a one-to-one correspondence between each string and each mapping value, and each mapping value in each computing node is reversely mapped to a reverse mapping relationship of the string, which is stored as a local reverse dictionary corresponding to each computing node. Then, based on the merged computing nodes, the local reverse dictionary corresponding to each computing node can be obtained, and reverse query processing can be performed according to the local reverse dictionary of the computing node corresponding to each repeated value, so that multiple strings corresponding to each repeated value can be queried.

[0158] 404. If it is determined that the character strings corresponding to the repeated values ​​are all the same, then it is determined that the character strings corresponding to the repeated values ​​are character strings in the multi-machine repeated set corresponding to the repeated values.

[0159] Exemplarily, based on the merge calculation node, the reverse mapping relationship in each local reverse dictionary is used to find the strings corresponding to each repeated value, and the strings corresponding to each repeated value are compared. If it is determined that the strings corresponding to the repeated values ​​are the same, the strings corresponding to the repeated values ​​can be determined as repeated strings after the second duplicate check, that is, the multi-machine duplicate set corresponding to each repeated value.

[0160] 405. Perform consistency processing on the multi-machine repeat set and the single-machine repeat set to obtain a processed multi-machine repeat set and a processed single-machine repeat set.

[0161] Exemplarily, before merging data sets, in order to ensure the accuracy and completeness of the data, it is necessary to use data processing technology to perform consistency processing on the data in each multi-machine duplicate set and the data in each single-machine duplicate set, for example, ensuring the consistency of field names and data types, processing coding formats, data formats, etc., to obtain processed multi-machine duplicate sets and processed single-machine duplicate sets.

[0162] 406. Merge the processed multi-machine repeat sets and the processed single-machine repeat sets to obtain a global repeat set.

[0163] Exemplarily, the device merges the obtained repeated character strings in each processed multi-machine repetition set and each processed single-machine repetition set to obtain the repeated character strings in all computing nodes, that is, the global repetition set.

[0164] In one example, step 406 includes:

[0165] Step 1: merge the processed multi-machine repeat set and the two single-machine repeat sets corresponding to the processed multi-machine repeat set to obtain a first local repeat set corresponding to the processed multi-machine repeat set.

[0166] Step 2: merge the first local repeated sets to obtain a merged repeated set.

[0167] Specifically, in order to ensure data consistency, the merge calculation node in the device can first determine the two processed single-machine duplicate sets corresponding to each processed multi-machine duplicate set from each processed single-machine duplicate set, and merge the processed multi-machine duplicate set and the two processed single-machine duplicate sets corresponding to the processed multi-machine duplicate set to obtain the first local duplicate set corresponding to the processed multi-machine duplicate set; then, the merge calculation node merges the first local duplicate sets corresponding to each processed multi-machine duplicate set again to obtain a merged duplicate set.

[0168] For example, Figure 8 A schematic diagram of a process of merging duplicate sets provided in an embodiment of the present application is shown in FIG. Figure 8As shown, after data consistency processing, for bitmap array 1 of computing node 1, bitmap array 2 of computing node 2, and bitmap array 3 of computing node 3, according to bitmap array 1, bitmap array 2, and the local reverse dictionary 1-2 corresponding between computing node 1 and computing node 2, the multi-machine duplicate set 1-2 corresponding between computing node 1 and computing node 2 can be obtained; according to bitmap array 1, bitmap array 3, and the local reverse dictionary 1-3 corresponding between computing node 1 and computing node 3, the multi-machine duplicate set 1-3 corresponding between computing node 1 and computing node 3 can be obtained; according to bitmap array 2, bitmap array 3, and the local reverse dictionary 2-3 corresponding between computing node 2 and computing node 3, the multi-machine duplicate set 1-3 corresponding between computing node 1 and computing node 3 can be obtained. Repeat set 2-3, according to bitmap array 2, bitmap array 3 and the local reverse dictionary 2-3 corresponding to computing node 2 and computing node 3, the multi-machine repeat set 2-3 corresponding to computing node 2 and computing node 3 can be obtained; the single-machine repeat set 1 of computing node 1, the single-machine repeat set 3 of computing node 3, and the multi-machine repeat set 1-3 are merged to obtain the first local repeat set 1-3, the single-machine repeat set 1 of computing node 1, the single-machine repeat set 3 of computing node 3, and the multi-machine repeat set 1-3 are merged to obtain the first local repeat set 1-3, the single-machine repeat set 2 of computing node 2, the single-machine repeat set 3 of computing node 3, and the multi-machine repeat set 2-3 are merged to obtain the first local repeat set 2-3, the single-machine repeat set 2 of computing node 2, the single-machine repeat set 3 of computing node 3, and the multi-machine repeat set 2-3 are merged to obtain the first local repeat set 2-3; finally, the three first local repeat sets are merged to obtain a global repeat set.

[0169] In one example, step 406 further includes:

[0170] Step 3: Merge the processed multi-machine repeat sets to obtain the second local repeat sets corresponding to the processed multi-machine repeat sets.

[0171] Step 4: Merge the processed single-machine repetition sets to obtain the third local repetition sets corresponding to the processed single-machine repetition sets.

[0172] Step 5: Merge the second local repeat set and the third local repeat set to obtain a global repeat set.

[0173] Specifically, the merge calculation node can first merge the processed multi-machine duplicate sets to obtain the corresponding second local duplicate set, including the repeated strings obtained by the second duplicate check; then merge the processed single-machine duplicate sets to obtain the third local duplicate set, including the repeated strings obtained by the first duplicate check, and then merge the strings in the second local duplicate set with the strings in the third local duplicate set to obtain the global duplicate set.

[0174] For example, as shown in Table 1 below, for multiple data strings in the target data source, the length of each data string is 20. By comparing the effects of the methods provided in the embodiments of the present application and other duplicate detection and deduplication methods, it is shown that the performance of the distributed local reverse dictionary method provided in the embodiments of the present application is better than other methods; by increasing the number of threads, the performance of the distributed local reverse dictionary method can be further improved.

[0175] Table 1 Performance comparison of different methods

[0176]

[0177] For example, Fig. 9 A schematic diagram of an overall process of determining duplicate data provided in an embodiment of the present application, such as Fig. 9 As shown, for the Mth computing node, there are strings A1 and A2, and the strings A1 and A2 are mapped to a bitmap array M of VA1 and VA2, and the reverse mapping relationship between VA1 reversely mapped to A1 and VA2 reversely mapped to A2 in the reverse dictionary M is obtained. The single-machine duplicate set M can be obtained by confirming the bit conflict in the bitmap array bit operation; for the Nth computing node, there are strings B1 and B2, and the strings B1 and B2 are mapped to a bitmap array N of VB1 and VB2, and the reverse mapping relationship between VB1 reversely mapped to B1 and VB2 reversely mapped to B2 in the reverse dictionary N is obtained. The single-machine duplicate set N can be obtained by confirming the bit conflict in the bitmap array bit operation; then, the single-machine duplicate set M and the single-machine duplicate set N are subjected to a secondary check for duplicate bit values ​​to obtain a multi-machine duplicate set, and then a global duplicate set can be obtained through a merge process.

[0178] In this embodiment, based on the above embodiment, each multi-machine duplicate set and the two single-machine duplicate sets corresponding to the multi-machine duplicate set are merged in parallel, or each multi-machine duplicate set and each single-machine duplicate set are merged in parallel respectively; finally, all are merged and restored through a multiple merging process, which can ensure the consistency and accuracy of the duplicate data, so as to efficiently realize fast duplicate detection.

[0179] Fig.10 A flowchart of another method for determining duplicate data provided in an embodiment of the present application is shown as follows: Fig.10 As shown, the method includes:

[0180] 501. Based on the merged computing nodes, obtain the bitmap array and the single-machine repetition set of M computing nodes; wherein the bitmap array represents the mapping processing record of each string representing the business data in the computing node; the single-machine repetition set includes the repeated string; and M is an integer greater than or equal to 1.

[0181] Exemplarily, the content of this step can refer to step 201 and will not be repeated here.

[0182] 502. Perform two-bit operations on the bitmap arrays of the M computing nodes to obtain M repeated values; wherein the repeated values ​​represent repeated mapping values ​​in every two computing nodes; and the mapping values ​​represent unique identifiers of the business data.

[0183] Exemplarily, this step may refer to step 202 and will not be described in detail.

[0184] 503. Obtain the local reverse dictionary corresponding to the computing node; and perform a second duplicate check on the duplicate values ​​according to each local reverse dictionary to obtain a multi-machine duplicate set corresponding to the duplicate values; wherein the local reverse dictionary includes a reverse mapping relationship in which each mapping value is reversely mapped to a string; and the multi-machine duplicate set includes the repeated strings in every two computing nodes.

[0185] Exemplarily, this step may refer to step 203 and will not be described in detail.

[0186] 504. Merge each multi-machine repetition set and each single-machine repetition set to obtain a global repetition set; wherein the global repetition set includes repeated character strings in all computing nodes.

[0187] Exemplarily, this step may refer to step 204 and will not be described in detail.

[0188] 505. De-duplicate the strings in the M computing nodes according to the bitmap arrays of the M computing nodes.

[0189] For example, if the device needs to retain the deduplication results, it needs to output all non-duplicate data sets in the end. It only needs to traverse the bitmap array of the M computing nodes, such as Bitmap, to deduplicate each string in the M computing nodes, and then output all non-duplicate strings.

[0190] In one example, step 505 includes: performing deduplication calculation processing on each string in the M computing nodes according to the bitmap array and the local reverse dictionary of the M computing nodes to obtain a non-duplicate data set; wherein the local reverse dictionary includes a reverse mapping relationship in which each mapping value is reversely mapped to a string; and the non-duplicate data set includes non-duplicate strings in the M computing nodes.

[0191] Specifically, the device can obtain the bitmap array Bitmap of M computing nodes, and obtain the local reverse dictionary of the M computing nodes. The local reverse dictionary includes the reverse mapping relationship of each mapping value in the bitmap array to a string. Then, according to the mapping value stored in each Bitmap, each local reverse dictionary is queried, and all data outputs are restored to obtain a non-duplicate string in the M computing nodes.

[0192] In one example, step 505 also includes: performing aggregation calculation processing on each string in the M computing nodes according to the bitmap array and local reverse dictionary of the M computing nodes to obtain the number of non-duplicates corresponding to the M computing nodes; wherein the number of non-duplicates is the total number of different strings in the M computing nodes.

[0193] Specifically, if you want to determine the number of different values ​​in each computing node, the device can perform aggregation calculation processing on each string in the M computing nodes based on the bitmap array and local reverse dictionary of the M computing nodes, based on the reverse lookup algorithm and the bitmap aggregation algorithm, and output the total number of different strings in the M computing nodes, that is, the number without duplication.

[0194] In this embodiment, based on the above embodiment, through the bitmap array and the local reverse dictionary, each string in each computing node can be deduplicated, so that a non-repeated string can be obtained, thereby reducing the waste of computing resources and storage resources.

[0195] Fig.11 A schematic diagram of a structure of a device for determining duplicate data provided in an embodiment of the present application is shown in FIG. Fig.11 As shown, the device comprises:

[0196] Acquisition module 601 is used to acquire the bitmap array and single-machine repetition set of M computing nodes based on the merged computing nodes; wherein the bitmap array represents the mapping processing records of each string representing business data in the computing node; the single-machine repetition set includes repeated strings; and M is an integer greater than or equal to 1.

[0197] The first processing module 602 is used to perform two-bit operations on the bitmap arrays of the M computing nodes to obtain M repeated values; wherein the repeated values ​​represent the repeated mapping values ​​in every two computing nodes; and the mapping values ​​represent the unique identifier of the business data.

[0198] The second processing module 603 is used to obtain the local reverse dictionary corresponding to the computing node; and according to each local reverse dictionary, perform a secondary duplicate check on the duplicate values ​​to obtain a multi-machine duplicate set corresponding to the duplicate values; wherein the local reverse dictionary includes a reverse mapping relationship in which each mapping value is reversely mapped to a string; and the multi-machine duplicate set includes the repeated strings in every two computing nodes.

[0199] The merging module 604 is used to merge the multi-machine repetition sets and the single-machine repetition sets to obtain a global repetition set; wherein the global repetition set includes repeated character strings in all computing nodes.

[0200] In one possible implementation, the acquisition module 601 is specifically used to: determine a target data source; and distribute multiple first data sets in the target data source to M computing nodes; wherein the first data set includes at least one string; based on the computing nodes, perform duplicate checking on each string in the received first data set to obtain a single-machine duplicate set and a bitmap array corresponding to the computing node; based on merging the computing nodes, obtain the bitmap array and single-machine duplicate set of the M computing nodes.

[0201] In one possible implementation, the acquisition module 601 is specifically used to: based on the computing node, map each string in the first data set to obtain a mapping set, a local reverse dictionary, and a bitmap array corresponding to the computing node; wherein the mapping set includes a mapping value corresponding to each string; the local reverse dictionary includes a reverse mapping relationship in which each mapping value is reversely mapped to a string; and reversely query the local reverse dictionary and the bitmap array corresponding to the computing node to obtain a single-machine repetition set corresponding to the computing node.

[0202] In a possible implementation, the acquisition module 601 is specifically used to: perform calculation processing on each string in the first data set to obtain an integer set corresponding to the first data set; wherein the integer set includes an integer value corresponding to each string; the integer value represents a unique identifier of the business data; determine the integer value corresponding to the string as a mapping value corresponding to the string;

[0203] According to each character string and the mapping value corresponding to each character string, a local reverse dictionary and a bitmap array are determined.

[0204] In one possible implementation, the acquisition module 601 is specifically used to: encrypt each string in the first data set to obtain an encrypted set corresponding to the first data set; wherein the encrypted set includes encrypted data corresponding to each string; and perform hash calculation on each encrypted data in the encrypted set to obtain an integer set.

[0205] In a possible implementation, the acquisition module 601 is specifically used to: perform bitwise operation processing on the bitmap array corresponding to the computing node to obtain a mapping value set corresponding to the computing node; wherein the mapping value set includes at least one mapping subset corresponding to the computing node, and the mapping subset includes repeated mapping values; according to the local reverse dictionary corresponding to the computing node, reverse query processing is performed on the mapping value set corresponding to the computing node to determine the original data set corresponding to the computing node; wherein the original data set includes a data subset corresponding to each mapping subset; the data subset includes a character string corresponding to each repeated mapping value in the mapping subset; compare each character string in the data subset to obtain a comparison result of the data subset; if it is determined that the comparison result indicates that each character string in the data subset is the same, then it is determined that each character string in the data subset is a character string in a single-machine repeated set.

[0206] In a possible implementation, the target data source is stored in a single server; the acquisition module 601 is further specifically used to: determine the target data source from multiple data sources according to a data source configuration file; wherein the data source configuration file represents the configuration of the data source.

[0207] In a possible implementation, the data source configuration file includes server information and database information; the server information represents the server configuration of the data source; the database information includes the connection string and instance status of the database where the data source is located.

[0208] In a possible implementation, the target data source includes multiple business data; the acquisition module 601 is also specifically used to: perform data splitting processing on the target data source to obtain M second data sets corresponding to the target data source; wherein the second data set includes business data in at least one target data source; perform calculation processing on each business data in the second data set to obtain a first data set corresponding to the second data set; and evenly distribute the M first data sets to M computing nodes.

[0209] In one possible implementation, the merging module 604 is specifically used to: perform consistency processing on the multi-machine duplicate sets and the single-machine duplicate sets to obtain processed multi-machine duplicate sets and processed single-machine duplicate sets; and merge the processed multi-machine duplicate sets and the processed single-machine duplicate sets to obtain a global duplicate set.

[0210] In one possible implementation, the merging module 604 is specifically used to: merge the processed multi-machine repeat set and the two processed single-machine repeat sets corresponding to the processed multi-machine repeat set to obtain the first local repeat set corresponding to the processed multi-machine repeat set; merge each first local repeat set to obtain a global repeat set.

[0211] In a possible implementation, the merging module 604 is also specifically used to: merge the processed multi-machine repeat sets to obtain the second local repeat sets corresponding to the processed multi-machine repeat sets; merge the processed single-machine repeat sets to obtain the third local repeat sets corresponding to the processed single-machine repeat sets; merge the second local repeat set and the third local repeat set to obtain the global repeat set.

[0212] In a possible implementation, the second processing module 603 is specifically used to: determine at least one string corresponding to the repeated value according to each local reverse dictionary; if it is determined that the strings corresponding to the repeated values ​​are the same, then determine that the strings corresponding to the repeated values ​​are the strings in the multi-machine repetition set corresponding to the repeated values.

[0213] In a possible implementation, the device is further configured to: perform deduplication processing on each character string in the M computing nodes according to the bitmap arrays of the M computing nodes.

[0214] In a possible implementation, the device is also specifically used to: perform deduplication calculations on each string in the M computing nodes according to the bitmap arrays and local reverse dictionaries of the M computing nodes to obtain a non-duplicate data set; wherein the local reverse dictionary includes a reverse mapping relationship in which each mapping value is reversely mapped to a string; and the non-duplicate data set includes non-duplicate strings in the M computing nodes.

[0215] In a possible implementation, the device is also specifically used to: perform aggregation calculation processing on each string in the M computing nodes according to the bitmap array and local reverse dictionary of the M computing nodes to obtain the number of non-duplicates corresponding to the M computing nodes; wherein the number of non-duplicates is the total number of different strings in the M computing nodes.

[0216] The device of this embodiment can execute the technical solution in the above method. Its specific implementation process and technical principles are the same and will not be repeated here.

[0217] It should be noted that it should be understood that the division of the various modules of the above device is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. And these modules can all be implemented in the form of software called by processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software called by processing elements, and some modules can be implemented in the form of hardware. Each module can be a separately established processing element, or it can be integrated in a chip of the above device. In addition, it can also be stored in the memory of the above device in the form of program code, and called and executed by a processing element of the above device. The functions of each module. In addition, all or part of these modules can be integrated together, or they can be implemented independently. The processing element here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each module above can be completed by an integrated logic circuit of hardware in the processor element or instructions in the form of software.

[0218] Fig.12 A schematic diagram of a computing device provided in an embodiment of the present application is shown in FIG. Fig.12 As shown, the computing device includes: a memory 701, a processor 702; the memory 701; and a memory for storing instructions executable by the processor 702.

[0219] The processor 702 is configured to execute the method provided in the above embodiment.

[0220] The computing device further includes a receiver 703 and a transmitter 704. The receiver 703 is used to receive instructions and data sent by other devices, and the transmitter 704 is used to send instructions and data to external devices.

[0221] An embodiment of the present application also provides a chip for executing instructions, which is used to execute the technical solution of the processing method in the above embodiment.

[0222] The embodiment of the present application also provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions are executed on a computer, the computer executes the technical solution of the processing method of the above embodiment.

[0223] An embodiment of the present application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium. When at least one processor executes the computer program, the technical solution of the processing method in the above embodiment can be implemented.

[0224] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0225] In addition, each functional module in each embodiment of the present application can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The above-mentioned module-composed unit can be implemented in the form of hardware or in the form of hardware plus software functional units.

[0226] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the methods of various embodiments of the present application.

[0227] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented as being executed by a hardware processor, or may be implemented by a combination of hardware and software modules in the processor.

[0228] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present application.

[0229] In the above embodiments, the description of each embodiment has its own emphasis. For the part not described in detail in a certain embodiment, please refer to the relevant description of other embodiments. The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0230] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0231] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A method for determining duplicate data, characterized in that: include: Based on the merged computing nodes, a bitmap array and a single-machine repetition set of M computing nodes are obtained; wherein the bitmap array represents a mapping processing record of each string representing business data in the computing node; the single-machine repetition set includes repeated strings; and M is an integer greater than or equal to 1; Perform two-bit operations on the bitmap arrays of the M computing nodes to obtain M repeated values; wherein the repeated values ​​represent repeated mapping values ​​in every two computing nodes; and the mapping values ​​represent unique identifiers of business data; Obtaining a local reverse dictionary corresponding to the computing node; and performing a secondary duplicate check on the duplicate value according to each local reverse dictionary to obtain a multi-machine duplicate set corresponding to the duplicate value; wherein the local reverse dictionary includes a reverse mapping relationship in which each mapping value is reversely mapped to a string; and the multi-machine duplicate set includes strings repeated in every two computing nodes; Each of the multi-machine repetition sets and each of the single-machine repetition sets are merged to obtain a global repetition set; wherein the global repetition set includes repeated character strings in all computing nodes.

2. The method according to claim 1, characterized in that Based on the merged computing nodes, the bitmap array and single-machine repetition set of M computing nodes are obtained, including: Determine a target data source; and distribute a plurality of first data sets in the target data source to the M computing nodes; wherein the first data set includes at least one of the character strings; Based on the computing node, each character string in the received first data set is checked for duplicates to obtain a single-machine duplicate set and a bitmap array corresponding to the computing node; Based on the merged computing node, a bitmap array and a single-machine repetition set of the M computing nodes are obtained.

3. The method according to claim 2, characterized in that Based on the computing node, each character string in the received first data set is checked for duplicates to obtain a single machine duplicate set and a bitmap array corresponding to the computing node, including: Based on the computing node, mapping processing is performed on each string in the first data set to obtain a mapping set, a local reverse dictionary and a bitmap array corresponding to the computing node; wherein the mapping set includes a mapping value corresponding to each string; the local reverse dictionary includes a reverse mapping relationship of each mapping value reversely mapped to a string; The local reverse dictionary and the bitmap array corresponding to the computing node are reversely checked to obtain the single-machine duplicate set corresponding to the computing node.

4. The method according to claim 3, characterized in that Based on the computing node, mapping processing is performed on each string in the first data set to obtain a mapping set, a local reverse dictionary, and a bitmap array corresponding to the computing node, including: Performing calculation processing on each character string in the first data set to obtain an integer set corresponding to the first data set; wherein the integer set includes an integer value corresponding to each character string; and the integer value represents a unique identifier of the business data; Determine the integer value corresponding to the string as the mapping value corresponding to the string; The local reverse dictionary and the bitmap array are determined according to each of the character strings and the mapping values ​​corresponding to the character strings.

5. The method according to claim 4, characterized in that Performing calculation processing on each character string in the first data set to obtain an integer set corresponding to the first data set includes: Encrypting each character string in the first data set to obtain an encrypted set corresponding to the first data set; wherein the encrypted set includes encrypted data corresponding to each character string; A hash calculation is performed on each encrypted data in the encryption set to obtain the integer set.

6. The method according to claim 3, characterized in that Perform reverse query processing on the local reverse dictionary and bitmap array corresponding to the computing node to obtain a single-machine duplicate set corresponding to the computing node, including: Performing bitwise operation processing on the bitmap array corresponding to the computing node to obtain a mapping value set corresponding to the computing node; wherein the mapping value set includes at least one mapping subset corresponding to the computing node, and the mapping subset includes repeated mapping values; According to the local reverse dictionary corresponding to the computing node, reverse query processing is performed on the mapping value set corresponding to the computing node to determine the original data set corresponding to the computing node; wherein the original data set includes a data subset corresponding to each of the mapping subsets; and the data subset includes a character string corresponding to each repeated mapping value in the mapping subset; Performing comparison processing on each character string in the data subset to obtain a comparison result of the data subset; If it is determined that the comparison result indicates that all character strings in the data subset are the same, then it is determined that all character strings in the data subset are character strings in the single-machine repetition set.

7. The method according to claim 2, characterized in that: The target data source is stored in a single server; Identify target data sources, including: The target data source is determined from a plurality of data sources according to a data source configuration file, wherein the data source configuration file represents a configuration status of a data source.

8. The method according to claim 7, characterized in that The data source configuration file includes server information and database information; the server information represents the server configuration of the data source; the database information includes the connection string and instance status of the database where the data source is located.

9. The method according to claim 2, characterized in that: The target data source includes a plurality of business data; distributing a plurality of first data sets in the target data source to the M computing nodes includes: Performing data splitting processing on the target data source to obtain M second data sets corresponding to the target data source; wherein the second data set includes business data in at least one target data source; Performing calculation processing on each business data in the second data set to obtain the first data set corresponding to the second data set; The M first data sets are evenly distributed to the M computing nodes.

10. The method according to claim 1, characterized in that Each of the multi-machine repetition sets and each of the single-machine repetition sets are merged to obtain a global repetition set, including: Performing consistency processing on the multi-machine repeat set and the single-machine repeat set to obtain a processed multi-machine repeat set and a processed single-machine repeat set; The processed multi-machine repetition sets and the processed single-machine repetition sets are merged to obtain the global repetition set.

11. The method according to claim 10, characterized in that The processed multi-machine repetition sets and the processed single-machine repetition sets are merged to obtain the global repetition set, including: Merging the processed multi-machine repeat set and two processed single-machine repeat sets corresponding to the processed multi-machine repeat set to obtain a first local repeat set corresponding to the processed multi-machine repeat set; The first local repetition sets are merged to obtain the global repetition set.

12. The method according to claim 10, characterized in that The processed multi-machine repetition sets and the processed single-machine repetition sets are merged to obtain the global repetition set, including: Merging the processed multi-machine repeat sets to obtain second local repeat sets corresponding to the processed multi-machine repeat sets; Merging the processed single-machine repetition sets to obtain third local repetition sets corresponding to the processed single-machine repetition sets; The second local repetition set and the third local repetition set are combined to obtain the global repetition set.

13. The method according to claim 1, characterized in that According to each of the local reverse dictionaries, the duplicate values ​​are subjected to a secondary duplicate check process to obtain a multi-machine duplicate set corresponding to the duplicate values, including: Determine at least one character string corresponding to the repeated value according to each of the local reverse dictionaries; If it is determined that the character strings corresponding to the repetition values ​​are all the same, then it is determined that the character strings corresponding to the repetition values ​​are character strings in the multi-machine repetition set corresponding to the repetition values.

14. The method according to any one of claims 1 to 13, characterized in that The method further comprises: According to the bitmap arrays of the M computing nodes, deduplication processing is performed on each character string in the M computing nodes.

15. The method according to claim 14, characterized in that According to the bitmap arrays of the M computing nodes, deduplication processing is performed on each string in the M computing nodes, including: According to the bitmap array and local reverse dictionary of the M computing nodes, deduplication calculation processing is performed on each string in the M computing nodes to obtain a non-duplicate data set; wherein the local reverse dictionary includes a reverse mapping relationship in which each mapping value is reversely mapped to a string; and the non-duplicate data set includes strings that are not repeated in the M computing nodes.

16. The method according to claim 15, characterized in that The method further comprises: According to the bitmap arrays and local reverse dictionaries of the M computing nodes, aggregate calculation processing is performed on each string in the M computing nodes to obtain the number of non-repetitive strings corresponding to the M computing nodes; wherein the number of non-repetitive strings is the total number of different strings in the M computing nodes.

17. A device for determining duplicate data, characterized in that: include: An acquisition module, used to acquire a bitmap array and a single-machine repetition set of M computing nodes based on the merged computing node; wherein the bitmap array represents a mapping processing record of each string representing business data in the computing node; the single-machine repetition set includes repeated strings; M is an integer greater than or equal to 1; A first processing module is used to perform two-bit operation processing on the bitmap arrays of the M computing nodes to obtain M repeated values; wherein the repeated values ​​represent repeated mapping values ​​in every two computing nodes; and the mapping values ​​represent unique identifiers of business data; The second processing module is used to obtain the local reverse dictionary corresponding to the computing node; and according to each of the local reverse dictionaries, perform a secondary duplicate check on the repeated value to obtain a multi-machine duplicate set corresponding to the repeated value; wherein the local reverse dictionary includes a reverse mapping relationship in which each mapping value is reversely mapped to a string; and the multi-machine duplicate set includes the repeated strings in every two computing nodes; The merging module is used to merge the multi-machine repetition sets and the single-machine repetition sets to obtain a global repetition set; wherein the global repetition set includes repeated character strings in all computing nodes.

18. A computing device, characterized in that include: Memory, processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method according to any one of claims 1 to 16.

19. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the method according to any one of claims 1 to 16.

20. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 16 is implemented.

Citation Information

Cited By

  • Method and apparatus for determining duplicate data, and device

    WO2026036630A1