Method and apparatus for determining duplicate data, and device

Abstract

The embodiments of the present application relate to the field of big data. Provided are a method and apparatus for determining duplicate data, and a device. The method comprises: on the basis of a merged computing node, acquiring bitmap arrays and single-machine duplicate sets of M computing nodes; performing pairwise bitwise operation processing on the bitmap arrays of the M computing nodes, so as to obtain M duplicate values; acquiring local reverse dictionaries corresponding to the computing nodes, and on the basis of the local reverse dictionaries, performing secondary duplicate check processing on the duplicate values, so as to obtain multi-machine duplicate sets corresponding to the duplicate values; and merging the multi-machine duplicate sets and the single-machine duplicate sets, so as to obtain a global duplicate set. The method in the present application improves data duplicate detection efficiency.

Description

Method, device and equipment for determining repeated data

[0001] The present application claims priority to the Chinese patent application No. 202411133960.5, filed on August 16, 2024, and entitled "Method, device and equipment for determining repeated data", the whole content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to the field of big data, and in particular to a method, device and equipment for determining repeated data. BACKGROUND

[0003] There are a large number of repeated data in the financial system. In order to reduce the waste of storage and computing resources caused by repeated data, it is necessary to determine the repeated data.

[0004] In the prior art, the repeated data is marked and deleted by manually checking the data line by line.

[0005] However, in the above-mentioned manner, for a large amount of data set, the manual processing method is time-consuming and laborious, thereby reducing the determination efficiency of the repeated data. SUMMARY

[0006] The embodiments of the present application provide a method, device and equipment for determining repeated data, to solve the technical problem of reducing the determination efficiency of repeated data caused by the manual processing method.

[0007] In a first aspect, the embodiments of the present application provide a method for determining repeated data, comprising: obtaining a bitmap array and a single-machine repeated set of M computing nodes based on a merge computing node; wherein the bitmap array represents the mapping processing record of each string representing business data in the computing node; the single-machine repeated set includes repeated strings; M is an integer greater than or equal to 1; performing two-by-two bit operation processing on the bitmap array of the M computing nodes to obtain M repeated values; wherein the repeated value represents the repeated mapping value in each two computing nodes; the mapping value represents the unique identifier of the business data; obtaining the local reverse dictionary corresponding to the computing node; and performing secondary duplicate checking on the repeated value according to each local reverse dictionary to obtain a multi-machine repeated set corresponding to the repeated value; wherein the local reverse dictionary includes the reverse mapping relationship of each mapping value to a string; the multi-machine repeated set includes the repeated string in each two computing nodes; and performing merge processing on each multi-machine repeated set and each single-machine repeated set to obtain a global repeated set; wherein the global repeated set includes the repeated string in all computing nodes.

[0008] In a possible implementation, the bitmap array and the single-machine duplicate set of the M computing nodes are obtained based on the merging computing node, including: determining a target data source; and distributing a plurality of first data sets in the target data source to the M computing nodes; wherein the first data sets include at least one string; performing duplicate checking on each string in the received first data set based on the computing node to obtain the single-machine duplicate set and the bitmap array corresponding to the computing node; and obtaining the bitmap array and the single-machine duplicate set of the M computing nodes based on the merging computing node.

[0009] In a possible implementation, the duplicate checking is performed on each string in the received first data set based on the computing node to obtain the single-machine duplicate set and the bitmap array corresponding to the computing node, including: performing mapping processing on each string in the first data set based on the computing node to obtain a mapping set, a local reverse dictionary, and a bitmap array corresponding to the computing node; wherein the mapping set includes a mapping value corresponding to each string; the local reverse dictionary includes a reverse mapping relationship of each mapping value being reversely mapped into a string; and performing reverse lookup processing on the local reverse dictionary and the bitmap array corresponding to the computing node to obtain the single-machine duplicate set corresponding to the computing node.

[0010] In a possible implementation, the mapping processing is performed on each string in the first data set based on the computing node to obtain the mapping set, the local reverse dictionary, and the bitmap array corresponding to the computing node, including: performing computing processing on each string in the first data set to obtain an integer set corresponding to the first data set; wherein the integer set includes an integer value corresponding to each string; and the integer value represents a unique identifier of business data; determining the integer value corresponding to the string as a mapping value corresponding to the string; and determining the local reverse dictionary and the bitmap array according to each string and the mapping value corresponding to each string.

[0011] In a possible implementation, the computing processing is performed on each string in the first data set to obtain the integer set corresponding to the first data set, including: performing encryption processing on each string in the first data set to obtain an encryption set corresponding to the first data set; wherein the encryption set includes encryption data corresponding to each string; and performing hash calculation on each encryption data in the encryption set to obtain the integer set.

[0012] In a possible implementation, the local reverse dictionary and the bitmap array corresponding to the computing node are subjected to reverse lookup processing to obtain a single-machine repeated set corresponding to the computing node, including: performing bit operation processing on the bitmap array corresponding to the computing node to obtain a mapping value set corresponding to the computing node; wherein the mapping value set includes at least one mapping subset corresponding to the computing node, and the mapping subset includes repeated mapping values; performing reverse lookup processing on the mapping value set corresponding to the computing node according to the local reverse dictionary to determine an original data set corresponding to the computing node; wherein the original data set includes a data subset corresponding to each mapping subset; the data subset includes a string corresponding to each repeated mapping value in the mapping subset; performing comparison processing on each string in the data subset to obtain a comparison result of the data subset; and if it is determined that the comparison result represents that each string in the data subset is the same, it is determined that each string in the data subset is a string in the single-machine repeated set.

[0013] In a possible implementation, the target data source is stored in a single server; and determining the target data source includes: determining the target data source from a plurality of data sources according to a data source configuration file; wherein the data source configuration file represents the configuration of the data source.

[0014] In a possible implementation, the data source configuration file includes server information and database information; the server information represents the server configuration of the data source; and the database information includes a connection string and an instance state of a database where the data source is located.

[0015] In a possible implementation, the target data source includes a plurality of business data; and distributing a plurality of first data sets in the target data source to the M computing nodes includes: performing data splitting processing on the target data source to obtain M second data sets corresponding to the target data source; wherein each second data set includes business data in the target data source; performing computing processing on each business data in the second data set to obtain the first data set corresponding to the second data set; and distributing the M first data sets to the M computing nodes evenly.

[0016] In a possible implementation, each multi-machine repeated set and each single-machine repeated set is subjected to merging processing to obtain a global repeated set, including: performing consistency processing on the multi-machine repeated set and the single-machine repeated set to obtain a processed multi-machine repeated set and a processed single-machine repeated set; and performing merging processing on each processed multi-machine repeated set and each processed single-machine repeated set to obtain the global repeated set.

[0017] In a possible implementation, the merging processing is performed on each of the processed multi-machine repeat set and each of the processed single-machine repeat set to obtain the global repeat set, including: performing the merging processing on the processed multi-machine repeat set and two processed single-machine repeat sets corresponding to the processed multi-machine repeat set to obtain a first local repeat set corresponding to the processed multi-machine repeat set; and performing the merging processing on each of the first local repeat set to obtain the global repeat set.

[0018] In a possible implementation, the merging processing is performed on each of the processed multi-machine repeat set and each of the processed single-machine repeat set to obtain the global repeat set, including: performing the merging processing on each of the processed multi-machine repeat set to obtain a second local repeat set corresponding to each of the processed multi-machine repeat set; performing the merging processing on each of the processed single-machine repeat set to obtain a third local repeat set corresponding to each of the processed single-machine repeat set; and performing the merging processing on the second local repeat set and the third local repeat set to obtain the global repeat set.

[0019] In a possible implementation, the secondary repeat checking processing is performed on the repeat value according to each of the local reverse dictionaries to obtain a multi-machine repeat set corresponding to the repeat value, including: determining at least one string corresponding to the repeat value according to each of the local reverse dictionaries; and determining each string corresponding to the repeat value as a string in the multi-machine repeat set corresponding to the repeat value if each string corresponding to the repeat value is determined to be the same.

[0020] In a possible implementation, the method further includes: performing the deduplication processing on each string in the M computing nodes according to the bitmap array of the M computing nodes.

[0021] In a possible implementation, the deduplication processing is performed on each string in the M computing nodes according to the bitmap array of the M computing nodes, including: performing the deduplication calculation processing on each string in the M computing nodes according to the bitmap array of the M computing nodes and a local reverse dictionary to obtain a non-repeated data set; wherein the local reverse dictionary includes an inverse mapping relationship in which each mapping value is inversely mapped to a string; and the non-repeated data set includes strings without repetition in the M computing nodes.

[0022] In a possible implementation, the method further includes: performing the aggregation calculation processing on each string in the M computing nodes according to the bitmap array of the M computing nodes and a local reverse dictionary to obtain a non-repeated number corresponding to the M computing nodes; wherein the non-repeated number is a total number of different strings in the M computing nodes.

[0023] In a second aspect, an embodiment of the present application provides a repeated data determination apparatus, comprising: an acquisition module, configured to acquire, based on a merging calculation node, a bitmap array of M calculation nodes and a single-machine repeated set; wherein the bitmap array represents mapping processing records of each string representing service data in the calculation nodes; the single-machine repeated set comprises repeated strings; M is an integer greater than or equal to 1; a first processing module, configured to perform two-by-two bit operation processing on the bitmap arrays of the M calculation nodes to obtain M repeated values; wherein the repeated value represents a repeated mapping value in each two calculation nodes; the mapping value represents a unique identifier of service data; a second processing module, configured to acquire a local reverse dictionary corresponding to the calculation node, and perform secondary duplicate checking processing on the repeated values according to each local reverse dictionary to obtain a multi-machine repeated set corresponding to the repeated values; wherein the local reverse dictionary comprises a reverse mapping relationship that each mapping value is reversely mapped into a string; the multi-machine repeated set comprises strings that are repeated in each two calculation nodes; and a merging module, configured to perform merging processing on each multi-machine repeated set and each single-machine repeated set to obtain a global repeated set; wherein the global repeated set comprises strings that are repeated in all calculation nodes.

[0024] In a third aspect, an embodiment of the present application provides a computing device, comprising: a memory and a processor; the memory stores computer execution instructions; and the processor executes the computer execution instructions stored in the memory, so that the processor executes the first aspect and / or various possible implementation manners of the first aspect.

[0025] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, wherein the computer readable storage medium stores computer execution instructions, and the computer execution instructions are executed by a processor to implement the first aspect and / or various possible implementation manners of the first aspect.

[0026] In a fifth aspect, an embodiment of the present application provides a computer program product, comprising a computer program, wherein the computer program is executed by a processor to implement the first aspect and / or various possible implementation manners of the first aspect.

[0027] The repeated data determination method, apparatus and device provided by the embodiments of the present application perform secondary duplicate checking on repeated mapping values in each two calculation nodes based on the bit operation processing of the bitmap arrays of the two calculation nodes, obtain a multi-machine repeated set, and perform merging and restoration processing on the strings that are repeated in each calculation node, to obtain all repeated service data in each calculation node, thereby improving the repeated data determination efficiency and accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0028] FIG. 1 is a schematic diagram of an application scenario provided by the present application;

[0029] FIG. 2 is a flow diagram of a method for determining duplicate data according to an embodiment of the present application;

[0030] FIG. 3 is a flow diagram of another method for determining duplicate data according to an embodiment of the present application;

[0031] FIG. 4 is a flow diagram of a data distribution process according to an embodiment of the present application;

[0032] FIG. 5 is a flow diagram of a process for constructing a local reverse dictionary according to an embodiment of the present application;

[0033] FIG. 6 is a flow diagram of a secondary duplicate search process according to an embodiment of the present application;

[0034] FIG. 7 is a flow diagram of another method for determining duplicate data according to an embodiment of the present application;

[0035] FIG. 8 is a flow diagram of a process for merging duplicate sets according to an embodiment of the present application;

[0036] FIG. 9 is a flow diagram of an overall process for determining duplicate data according to an embodiment of the present application;

[0037] FIG. 10 is a flow diagram of another method for determining duplicate data according to an embodiment of the present application;

[0038] FIG. 11 is a block diagram of a device for determining duplicate data according to an embodiment of the present application;

[0039] FIG. 12 is a block diagram of a computing device according to an embodiment of the present application.

[0040] The above-described drawings show certain embodiments of the application. Additional aspects of the application will be described in more detail in the following description. DETAILED DESCRIPTION

[0041] The illustrative examples described herein will now be described with reference to the accompanying drawings, of which:

[0042] First, the terms used in the present application are explained:

[0043] Local reverse dictionary (LRD): a mapping dictionary that reverses an integer value to the original string, and is stored independently on each worker node.

[0044] Computing node: a general node, a thread on a machine can also be considered as an independent computing node.

[0045] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use, processing, transmission, provision, disclosure and application of related data comply with relevant laws, regulations and standards, necessary security measures are taken, do not violate public order and good customs, and provide corresponding operation portal for user to choose authorization or refusal.

[0046] It should be noted that the duplicate data determination method, device and equipment of the present application can also be used in any field other than big data, and the application field of the duplicate data determination method, device and equipment of the present application is not limited.

[0047] There is a large amount of duplicate data in the financial system, in order to reduce the waste of storage and computing resources caused by duplicate data, it is necessary to determine the duplicate data.

[0048] In one example, the duplicate data is marked and deleted by manually checking the data line by line.

[0049] However, in the above-mentioned manner, for a large amount of data set, the manual processing method is time-consuming and laborious, and thus the determination efficiency of the duplicate data is reduced.

[0050] In another example, based on single machine Set / Bitmap array search, Set collection class naturally has the function of searching for duplicates, and for some small data volume scenes, single machine Set can be directly used to search for duplicates; Bitmap is used to store data, and 1 bit represents a number in computer bit storage.

[0051] However, in the above-mentioned manner, the use of Set and Bitmap to search for duplicates is limited by single machine resources, and it is difficult to process massive data.

[0052] In yet another example, based on global dictionary + Bitmap search, a distributed database system maintains a global forward dictionary in advance, which maps the original data record to a globally unique integer, and then stores the integer in Bitmap for deduplication calculation.

[0053] However, in the above manner, a complex operation is required to map the string to a globally unique integer, and the original data cannot be obtained from the mapping value; the data volume is large, and there is concurrent competition; further, the performance of searching for duplicates is significantly reduced.

[0054] To solve the above technical problems, the present application proposes the following technical concept: based on the merging calculation node, according to the bit operation processing process of the bitmap array of each two calculation nodes, the duplicate mapping values in the two calculation nodes are processed for secondary duplicate checking, and the multi-machine duplicate set is obtained, and the duplicate strings in each calculation node are merged and restored to obtain all the duplicate business data in each calculation node. In this way, the problem of low efficiency of duplicate data determination in the prior art based on manual duplicate searching can be solved.

[0055] The technical solutions of the present application and how the technical solutions of the present application solve the above technical problems will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of the present application will be described below with reference to the accompanying drawings.

[0056] FIG. 1 is a schematic diagram of an application scenario provided by the present application. As shown in FIG. 1, the specific application scenario of the present application includes one merging calculation node 102 in the computing device 101 and a plurality of calculation nodes 103. Through the merging calculation node 102 in the computing device 101, the duplicate checking processing is performed on the business data in the data set distributed to each calculation node 103 to obtain the duplicate business data.

[0057] FIG. 2 is a flowchart of a duplicate data determination method provided by an embodiment of the present application, as shown in FIG. 2, the method includes:

[0058] 201, based on the merging calculation node, obtain the bitmap array and the single-machine duplicate set of M calculation nodes; wherein the bitmap array represents the mapping processing record of each string representing the business data in the calculation node; the single-machine duplicate set includes duplicate strings; M is an integer greater than or equal to 1.

[0059] For example, the execution subject of this embodiment can be a computing device, hereinafter referred to as the device. The device can be a physical device or a virtual device that executes the method for determining duplicate data. The device deploys a merged computing node and M distributed computing nodes. In each computing node, each string representing business data in the computing node is mapped and the mapping processing records of each string are stored in a bitmap array, such as storing all data converted to numbers in a Bitmap or Roaring Bitmap. At the same time, based on the deduplication method of each computing node, the duplicate strings in each computing node can be obtained and organized into a single-machine duplicate set. Based on the merged computing node, the bitmap arrays and single-machine duplicate sets of all computing nodes are obtained for processing; where M is an integer greater than or equal to 1.

[0060] 202. Perform two-bit operations on the bitmap array of M computing nodes to obtain M duplicate values; where the duplicate value represents the duplicate mapping value in every two computing nodes; the mapping value represents the unique identifier of the business data.

[0061] For example, based on the characteristics of bitmap arrays, the device can perform bitwise operations on the bitmap arrays in M ​​computer nodes to quickly obtain the repeated mapping values ​​in every two computer nodes. Each mapping value is the mapping value corresponding to the corresponding string mapping processing, which represents the unique identifier of the corresponding business data.

[0062] For example, for compute nodes 1, 2, and 3, bitwise operations are performed on bitmap array 1 of compute node 1 and bitmap array 2 of compute node 2 to obtain the corresponding repeated values ​​1-2 for compute nodes 1 and 2. Bitwise operations are performed on bitmap array 1 of compute node 1 and bitmap array 3 of compute node 3 to obtain the corresponding repeated values ​​1-3 for compute nodes 1 and 3. Bitwise operations are performed on bitmap array 3 of compute node 3 and bitmap array 2 of compute node 2 to obtain the corresponding repeated values ​​3-2 for compute nodes 3 and 2.

[0063] 203. Obtain the local reverse dictionary corresponding to the computing node; and perform secondary deduplication processing on the duplicate values ​​according to each local reverse dictionary to obtain the multi-machine duplicate set corresponding to the duplicate values; wherein, the local reverse dictionary includes the reverse mapping relationship of each mapping value to the string; the multi-machine duplicate set includes the duplicate strings in every two computing nodes.

[0064] Exemplarily, the mapping value repetition does not represent source data repetition, and therefore the device performs secondary duplicate checking on the repeated values corresponding to each pair of computing nodes. The device can perform reduction calculation processing on each repeated value, and based on each computing node in the distributed deployment, there is a one-to-one correspondence between each string and each mapping value, and an inverse mapping relationship of each mapping value in each computing node being inversely mapped to a string is stored as a local inverse dictionary corresponding to each computing node. Then, based on the merged computing node, the local inverse dictionary corresponding to each computing node can be obtained, and the corresponding local inverse dictionaries are queried by using the inverse mapping relationship of each mapping value being inversely mapped to a string, so that the repeated strings in each pair of computing nodes can be obtained, that is, the corresponding multi-machine repetition set is obtained.

[0065] 204、performing merging processing on each multi-machine repetition set and each single-machine repetition set to obtain a global repetition set; wherein the global repetition set includes the repeated strings in all computing nodes.

[0066] Exemplarily, after obtaining M multi-machine repetition sets and M single-machine repetition sets, the device performs merging processing on all multi-machine repetition sets and single-machine repetition sets to obtain a final global repetition set, that is, the repeated strings in all computing nodes can be obtained, so as to complete the data duplicate searching process of all computing nodes.

[0067] In this embodiment, a determination method of repeated data is provided, based on the merged computing node, secondary duplicate checking is performed on the repeated mapping values in each pair of computing nodes according to the bit operation processing process of the bitmap array of each pair of computing nodes and the local inverse dictionary, a multi-machine repetition set is obtained, and merging processing is performed on the repeated strings in each computing node to obtain all repeated business data in each computing node. Therefore, the determination efficiency and accuracy of the repeated data are improved.

[0068] FIG. 3 is a flowchart of another determination method of repeated data provided by an embodiment of the present application. As shown in FIG. 3, the method includes the following steps.

[0069] 301, determine a target data source; and distribute a plurality of first data sets in the target data source to M computing nodes; wherein the first data set includes at least one string.

[0070] Exemplarily, the device first obtains a target data source that needs to be subjected to duplicate searching processing. The target data source includes a plurality of data sets, that is, first data sets, and each first data set includes at least one string. Then, the device distributes all first data sets in the target data source to M computing nodes in the distributed deployment; for example, a computing node with rich computing resource can be distributed to more first data sets to improve the subsequent calculation processing efficiency.

[0071] In one example, the target data source is stored in a single server, and the determining of the target data source in step 301 comprises: determining the target data source from the plurality of data sources according to a data source configuration file, wherein the data source configuration file represents the configuration of the data source.

[0072] In one example, the data source configuration file comprises server information and database information, the server information represents the server configuration of the data source, and the database information comprises a connection string and an instance state of a database where the data source is located.

[0073] Specifically, if the target data source for the processing of the device is a data source stored in a single server, the obtained data source needs to be processed to determine whether the data source is stored in a single server, and then the device parses each data source to determine the configuration of the data source, i.e., a data source configuration file, and determines the target data source from the plurality of data sources according to the data source configuration file of each data source. Each data source configuration file comprises server information and database information, the server information represents the server configuration of the data source, and the database information comprises a connection string and an instance state of a database where the data source is located.

[0074] For example, the target data source is determined by checking a database configuration file, viewing a database connection string, using a database management tool, checking server configuration, operating system commands, viewing a database instance state, log files, etc.

[0075] In one example, the distributing of the plurality of first data sets in the target data source to the M computing nodes in step 301 comprises:

[0076] Step one, performing data splitting processing on the target data source to obtain M second data sets corresponding to the target data source, wherein each second data set comprises at least one business data in the target data source.

[0077] Step two, performing computing processing on each business data in the second data set to obtain a first data set corresponding to the second data set.

[0078] Step three, distributing the M first data sets to the M computing nodes evenly.

[0079] Specifically, FIG. 4 is a flow diagram of a data distribution process provided by an embodiment of the present application. As shown in FIG. 4, first, the business data in the target data source is equally split into M data sets corresponding to the target data source, i.e., second data sets, each of which includes at least one piece of business data in the target data source. Each piece of business data in each second data set is calculated and processed by a preset hash calculation method to obtain a corresponding string for each piece of business data, i.e., a first data set corresponding to the M second data sets. Then, the M first data sets are evenly distributed to M computing nodes deployed in a distributed manner.

[0080] 302. Based on the computing node, each string in the received first data set is processed for duplicate checking to obtain a single-machine duplicate set corresponding to the computing node and a bitmap array.

[0081] For example, based on each computing node, each string in the received data set is processed for duplicate checking according to a preset duplicate checking algorithm to obtain a string duplicated in the computing node, i.e., a single-machine duplicate set of the computing node. For each string in the data set, mapping processing is performed to obtain a mapped string, and the mapping processing process of each string is recorded and stored in the form of a bitmap array for processing.

[0082] In one example, step 302 includes the following steps:

[0083] The first step of step 302 is to perform mapping processing on each string in the first data set based on the computing node to obtain a mapping set, a local reverse dictionary, and a bitmap array corresponding to the computing node. The mapping set includes a mapping value corresponding to each string. The local reverse dictionary includes a reverse mapping relationship of each mapping value being reversely mapped into a string.

[0084] The second step of step 302 is to perform reverse lookup processing on the local reverse dictionary and the bitmap array corresponding to the computing node to obtain a single-machine duplicate set corresponding to the computing node.

[0085] Exemplarily, in a specific duplicate checking process, first, each computing node in the device performs mapping processing on each received string based on a preset mapping algorithm such as a hash algorithm, to obtain a mapping value corresponding to each string, that is, a mapping set corresponding to the computing node, and stores the mapping processing record that each string is mapped to each corresponding mapping value as a bitmap array; then, the reverse mapping relationship that each mapping value is reversely mapped to the corresponding string is stored in the local reverse dictionary of the computing node. The device performs calculation processing on the mapping record represented by the bitmap array of each computing node and the local reverse dictionary based on a preset reverse checking algorithm, that is, the repeated strings in the computing node can be obtained; for example, a preset deep learning model can be called, and the mapping record represented by the bitmap array of each computing node and the local reverse dictionary are input into the model for model processing to output the single-machine repeated set of each computing node.

[0086] In one example, the first step of step 302 includes:

[0087] Step one, performing calculation processing on each string in the first data set to obtain an integer set corresponding to the first data set; wherein the integer set includes an integer value corresponding to each string; the integer value represents a unique identifier of the business data.

[0088] Step two, determining that the integer value corresponding to the string is the mapping value corresponding to the string.

[0089] Step three, determining the local reverse dictionary and the bitmap array according to the strings and the mapping values corresponding to the strings.

[0090] Specifically, FIG. 5 is a flowchart of a process of constructing a local reverse dictionary according to an embodiment of the present application. As shown in FIG. 5, in a specific mapping process, based on each computing node, each string in the received data set is traversed in parallel according to a preset hash-like mapping algorithm, and each string is mapped to a number to obtain an integer value corresponding to each string, and then an integer set is obtained. Then, the integer value corresponding to each string is determined as the mapping value corresponding to the string, and the hash-like mapping processing record between each string and the mapping value corresponding to the string is stored as a bitmap array, and the reverse mapping relationship that each mapping value is reversely mapped to each corresponding string is stored in the local reverse dictionary.

[0091] In one example, step one includes:

[0092] First step, performing encryption processing on each string in the first data set to obtain an encryption set corresponding to the first data set; wherein the encryption set includes encryption data corresponding to each string.

[0093] The second step is to hash each encrypted data in the encrypted set to obtain an integer set.

[0094] Specifically, in the process of mapping each string to an integer value, each computing node of the device can first encrypt each received string based on a preset encryption algorithm to obtain an encrypted set including encrypted data corresponding to each string, and then hash each encrypted data in the encrypted set based on a preset hash algorithm to obtain an integer set corresponding to the encrypted set.

[0095] In one example, the second step of step 302 includes:

[0096] Step one: performing bit operation processing on the bitmap array corresponding to the computing node to obtain a mapping value set corresponding to the computing node; wherein the mapping value set includes at least one mapping subset corresponding to the computing node, and the mapping subset includes repeated mapping values.

[0097] Step two: performing reverse lookup processing on the mapping value set corresponding to the computing node according to the local reverse dictionary corresponding to the computing node to determine an original data set corresponding to the computing node; wherein the original data set includes a data subset corresponding to each mapping subset; and the data subset includes a string corresponding to each repeated mapping value in the mapping subset.

[0098] Step three: comparing each string in the data subset to obtain a comparison result of the data subset.

[0099] Step four: if it is determined that the comparison result indicates that each string in the data subset is the same, then it is determined that each string in the data subset is a string in the single-machine repeated set.

[0100] Specifically, in the reverse lookup processing, based on each computing node, the bitmap array corresponding to the computing node is processed by bit operation according to the characteristics of the bitmap array, so as to obtain a plurality of mapping subsets corresponding to the computing node, each mapping subset including repeated mapping values, and thus a mapping value set corresponding to the computing node is obtained. However, the same mapping value does not mean the same original data, so each computing node needs to perform reverse lookup processing on the repeated mapping values corresponding to the computing node according to the local reverse dictionary of the computing node to obtain a string corresponding to each repeated mapping value, so as to obtain each data subset in the corresponding original data set. Then, each string in each data subset is compared to obtain a comparison result of each data subset, so as to determine whether the strings in each data subset are the same. If it is determined that each string in each data subset is the same, it means that the repetition is confirmed, and the repeated string in the data subset is stored in the source data repeated set, which is referred to as a single-machine repeated set.

[0101] For example, FIG. 6 is a flowchart of a secondary duplicate searching process according to an embodiment of the present application. As shown in FIG. 6, for the string S1 and the string S2, the bitmap arrays corresponding to the two strings are subjected to a bitwise operation to obtain the same integer value V1 corresponding to the two strings. The integer value V1 is subjected to a bit conflict analysis, and a corresponding local reverse dictionary is called to perform a reverse lookup on V1 to obtain the original data S1 and S2 corresponding to V1. Then, it is determined whether S1 and S2 are the same. If yes, the S1 and S2 are stored in the single-machine duplicate set. If no, the step is skipped.

[0102] 303. Based on the merging calculation node, the bitmap array and the single-machine duplicate set of the M calculation nodes are obtained.

[0103] For example, based on the merging calculation node, the bitmap array and the single-machine duplicate set of all the M calculation nodes are obtained for processing.

[0104] 304. The bitmap arrays of the M calculation nodes are subjected to a pairwise bitwise operation to obtain M duplicate values. The duplicate value represents a duplicate mapping value in each two calculation nodes. The mapping value represents a unique identifier of the business data.

[0105] For example, the content of this step can be referred to step 202, which will not be repeated here.

[0106] 305. The local reverse dictionary corresponding to the calculation node is obtained. The duplicate values are subjected to a secondary duplicate checking based on the local reverse dictionaries to obtain a multi-machine duplicate set corresponding to the duplicate values. The local reverse dictionary includes a reverse mapping relationship in which each mapping value is reversely mapped to a string. The multi-machine duplicate set includes strings that are duplicated in each two calculation nodes.

[0107] For example, the content of this step can be referred to step 203, which will not be repeated here.

[0108] 306. The global duplicate set is obtained by merging the multi-machine duplicate sets and the single-machine duplicate sets. The global duplicate set includes strings that are duplicated in all the calculation nodes.

[0109] For example, the content of this step can be referred to step 204, which will not be repeated here.

[0110] In this embodiment, on the basis of the above-mentioned embodiment, the distributed reverse local dictionary stores the reverse mapping relationship, and the probability of reverse lookup is adjusted by the conflict rate of the mapping function. In combination with the distributed bitmap array storing the multi-machine data, parallel multi-machine bitwise operation searching is implemented, and thus the determination efficiency of the duplicate data is improved.

[0111] FIG. 7 is a flowchart of another method for determining duplicate data according to an embodiment of the present application. As shown in FIG. 7, the method includes the following steps.

[0112] 401. Based on the merged computing nodes, obtain the bitmap arrays and single-machine repeat sets of M computing nodes; where the bitmap arrays represent the mapping processing records of the strings representing business data in each computing node; the single-machine repeat sets include repeated strings; M is an integer greater than or equal to 1.

[0113] For example, the content of this step can be found in step 201, and will not be repeated here.

[0114] 402. Perform two-bit operations on the bitmap array of M computing nodes to obtain M duplicate values; where the duplicate value represents the duplicate mapping value in every two computing nodes; the mapping value represents the unique identifier of the business data.

[0115] For example, this step can be referred to as step 202, and will not be repeated here.

[0116] 403. Based on the local reverse dictionaries, determine at least one string corresponding to the duplicate value.

[0117] For example, based on each computing node in a distributed deployment, there exists a one-to-one correspondence between each string and each mapping value. The reverse mapping relationship of each mapping value in each computing node is reverse mapped to the string and stored as a local reverse dictionary corresponding to each computing node. Then, based on the merged computing nodes, the local reverse dictionary corresponding to each computing node can be obtained. And based on the local reverse dictionary of the computing node corresponding to each duplicate value, a reverse query process can be performed to find multiple strings corresponding to each duplicate value.

[0118] 404. If it is determined that all strings corresponding to the duplicate values ​​are the same, then each string corresponding to the duplicate value is determined to be a string in the set of duplicate values.

[0119] For example, based on the merged computing node, the reverse mapping relationship in each local reverse dictionary is used to find the strings corresponding to each duplicate value, and the strings corresponding to each duplicate value are compared. If it is determined that all the strings corresponding to the duplicate value are the same, then the strings corresponding to the duplicate value can be identified as the duplicate strings after secondary deduplication, which is the multi-machine duplicate set corresponding to each duplicate value.

[0120] 405. Perform consistency processing on the multi-machine duplicate set and the single-machine duplicate set to obtain the processed multi-machine duplicate set and the processed single-machine duplicate set.

[0121] Exemplarily, before the data set is merged, in order to ensure the accuracy and integrity of the data, the data in each multi-machine repeat set and the data in each single-machine repeat set need to be processed for consistency by using data processing technology. For example, the consistency of field names and data types is ensured, encoding formats, data formats, etc. are processed, and the processed multi-machine repeat set and the processed single-machine repeat set are obtained.

[0122] 406. The processed multi-machine repeat sets and the processed single-machine repeat sets are merged to obtain a global repeat set.

[0123] Exemplarily, the device merges the repeated strings in each processed multi-machine repeat set and each processed single-machine repeat set to obtain the repeated strings in all computing nodes, i.e., the global repeat set.

[0124] In one example, step 406 includes:

[0125] Step one, the processed multi-machine repeat set and the two processed single-machine repeat sets corresponding to the processed multi-machine repeat set are merged to obtain a first local repeat set corresponding to the processed multi-machine repeat set.

[0126] Step two, the first local repeat sets are merged to obtain a merged repeat set.

[0127] Specifically, in order to ensure data consistency, the merging computing node in the device can first determine the two processed single-machine repeat sets corresponding to each processed multi-machine repeat set from the processed single-machine repeat sets, and then merge the processed multi-machine repeat set and the two processed single-machine repeat sets corresponding to the processed multi-machine repeat set to obtain a first local repeat set corresponding to the processed multi-machine repeat set. Then, the merging computing node merges the obtained first local repeat sets corresponding to each processed multi-machine repeat set to obtain a merged repeat set.

[0128] For example, FIG. 8 is a flow diagram of a merging and repeating set process according to an embodiment of the present application. As shown in FIG. 8, after data consistency processing, for the bitmap array 1 of the computing node 1, the bitmap array 2 of the computing node 2, and the bitmap array 3 of the computing node 3, according to the bitmap array 1, the bitmap array 2, and the local reverse dictionary 1-2 between the computing node 1 and the computing node 2, the multi-computer repeating set 1-2 between the computing node 1 and the computing node 2 can be obtained; according to the bitmap array 1, the bitmap array 3, and the local reverse dictionary 1-3 between the computing node 1 and the computing node 3, the multi-computer repeating set 1-3 between the computing node 1 and the computing node 3 can be obtained; and according to the bitmap array 2, the bitmap array 3, and the local reverse dictionary 2-3 between the computing node 2 and the computing node 3, the multi-computer repeating set 2-3 between the computing node 2 and the computing node 3 can be obtained. The single-computer repeating set 1 of the computing node 1, the single-computer repeating set 3 of the computing node 3, and the multi-computer repeating set 1-3 are merged to obtain the first local repeating set 1-3; the single-computer repeating set 1 of the computing node 1, the single-computer repeating set 2 of the computing node 2, and the multi-computer repeating set 1-2 are merged to obtain the first local repeating set 1-2; the single-computer repeating set 2 of the computing node 2, the single-computer repeating set 3 of the computing node 3, and the multi-computer repeating set 2-3 are merged to obtain the first local repeating set 2-3; and finally, the three first local repeating sets are merged to obtain the global repeating set.

[0129] In one example, step 406 further includes:

[0130] Step three, merging and processing each processed multi-computer repeating set to obtain a second local repeating set corresponding to each processed multi-computer repeating set.

[0131] Step four, merging and processing each processed single-computer repeating set to obtain a third local repeating set corresponding to each processed single-computer repeating set.

[0132] Step five, merging and processing the second local repeating set and the third local repeating set to obtain the global repeating set.

[0133] Specifically, the merging computing node can first merge and process each processed multi-computer repeating set to obtain a corresponding second local repeating set, including the second time of repeating strings; then merge and process each processed single-computer repeating set to obtain a third local repeating set, including the first time of repeating strings; and finally, merge and process the strings in the second local repeating set and the strings in the third local repeating set to obtain the global repeating set.

[0134] For example, as shown in Table 1 below, for a plurality of data strings in a target data source, each data string has a length of 20, by comparing the effects between the method provided in the embodiments of the present application and other duplicate searching methods, it is shown that the performance of the distributed local reverse dictionary method provided in the embodiments of the present application is superior to other methods; by increasing the number of threads, the performance of the distributed local reverse dictionary method can be further improved.

[0135] Table 1: Performance comparison table of different methods

[0136] For another example, FIG. 9 is a schematic diagram of a whole process for determining duplicate data provided in the embodiments of the present application, as shown in FIG. 9, for the Mth computing node, there are strings A1 and A2, the bitmap array M of A1 and A2 is mapped to VA1 and VA2, and the reverse mapping relationship between VA1 and A1 and VA2 and A2 in the reverse dictionary M is obtained. By bit conflict confirmation in bitmap array bit operation, the single-machine duplicate set M can be obtained; for the Nth computing node, there are strings B1 and B2, the bitmap array N of B1 and B2 is mapped to VB1 and VB2, and the reverse mapping relationship between VB1 and B1 and VB2 and B2 in the reverse dictionary N is obtained. By bit conflict confirmation in bitmap array bit operation, the single-machine duplicate set N can be obtained; further, the secondary duplicate bit value checking of the single-machine duplicate set M and the single-machine duplicate set N is performed to obtain the multi-machine duplicate set, and after merging processing, the global duplicate set can be obtained.

[0137] In the embodiments, on the basis of the above-mentioned embodiments, the multi-machine duplicate set and the two single-machine duplicate sets corresponding to the multi-machine duplicate set are merged in parallel, or each multi-machine duplicate set is merged in parallel and each single-machine duplicate set is merged in parallel; finally, a plurality of merging processes of full merging and restoration processing are performed, which can guarantee the consistency and accuracy of the duplicate data, so as to efficiently realize fast duplicate searching.

[0138] FIG. 10 is a flowchart of another method for determining duplicate data provided in the embodiments of the present application, as shown in FIG. 10, the method comprises:

[0139] 501. Based on the merging computing node, the bitmap array and the single-machine duplicate set of the M computing nodes are obtained; wherein the bitmap array represents the mapping processing record of each string representing business data in the computing node; the single-machine duplicate set includes the duplicate strings; M is an integer greater than or equal to 1.

[0140] For example, the content of this step can be referred to step 201, which will not be repeated here.

[0141] 502. Perform two-to-two operations on the bitmap array of M computing nodes to obtain M duplicate values; where the duplicate value represents the duplicate mapping value in every two computing nodes; the mapping value represents the unique identifier of the business data.

[0142] For example, this step can be referred to as step 202, and will not be repeated here.

[0143] 503. Obtain the local reverse dictionary corresponding to the computing node; and perform secondary deduplication processing on the duplicate values ​​according to each local reverse dictionary to obtain the multi-machine duplicate set corresponding to the duplicate values; wherein, the local reverse dictionary includes the reverse mapping relationship of each mapping value to the string; the multi-machine duplicate set includes the duplicate strings in every two computing nodes.

[0144] For example, this step can be referred to step 203, and will not be repeated here.

[0145] 504. Merge the multi-machine duplicate sets and the single-machine duplicate sets to obtain the global duplicate set; the global duplicate set includes the strings that are repeated in all computing nodes.

[0146] For example, this step can be referred to step 204, and will not be repeated here.

[0147] 505. Based on the bitmap array of M computing nodes, perform deduplication on each string in the M computing nodes.

[0148] For example, if the device needs to retain the deduplication results, it will eventually need to output all unique datasets. This can be achieved by iterating through the bitmap arrays of the M computing nodes to deduplicate each string in the M computing nodes, thus outputting all unique strings.

[0149] In one example, step 505 includes: performing deduplication on each string in the M computing nodes based on the bitmap array of the M computing nodes and the local reverse dictionary to obtain a unique dataset; wherein, the local reverse dictionary includes the reverse mapping relationship of each mapping value to a string; the unique dataset includes strings that are not repeated in the M computing nodes.

[0150] Specifically, the device can obtain the bitmap array Bitmap of M computing nodes and the local reverse dictionary of M computing nodes. The local reverse dictionary includes the reverse mapping relationship of each mapping value in the bitmap array to a string. Then, based on the mapping value stored in each Bitmap, the device can query each local reverse dictionary, restore all data output, and obtain the string that is not repeated in the M computing nodes.

[0151] In one example, the step 505 further includes: performing, according to the bitmap array of the M computing nodes and the local reverse dictionary, aggregation calculation processing on each string in the M computing nodes to obtain a number of unique values corresponding to the M computing nodes; wherein the number of unique values is a total number of different strings in the M computing nodes.

[0152] Specifically, to determine the number of different values in each computing node, the device can perform, according to the bitmap array of the M computing nodes and the local reverse dictionary, aggregation calculation processing on each string in the M computing nodes based on the reverse lookup algorithm and the bitmap aggregation algorithm, and output the total number of different strings in the M computing nodes, i.e., obtain the number of unique values.

[0153] In the embodiment, on the basis of the above-mentioned embodiment, the bitmap array and the local reverse dictionary can be used to perform a deduplication operation on each string in each computing node, so that the strings without duplication can be obtained, and the waste of computing resources and storage resources can be reduced.

[0154] FIG. 11 is a structural schematic diagram of a repeated data determination device provided by an embodiment of the present application. As shown in FIG. 11, the device includes:

[0155] The acquisition module 601 is configured to acquire, based on the merged computing node, a bitmap array of the M computing nodes and a single-machine repeated set; wherein the bitmap array represents mapping processing records of each string representing the business data in the computing node; the single-machine repeated set includes repeated strings; and M is an integer greater than or equal to 1.

[0156] The first processing module 602 is configured to perform two-by-two bit operation processing on the bitmap array of the M computing nodes to obtain M repeated values; wherein the repeated value represents a repeated mapping value in each two computing nodes; and the mapping value represents a unique identifier of the business data.

[0157] The second processing module 603 is configured to acquire a local reverse dictionary corresponding to the computing node; and perform secondary duplicate checking processing on the repeated values according to the local reverse dictionaries to obtain a multi-machine repeated set corresponding to the repeated values; wherein the local reverse dictionary includes a reverse mapping relationship in which each mapping value is reversely mapped to a string; and the multi-machine repeated set includes strings repeated in each two computing nodes.

[0158] The merging module 604 is configured to perform merging processing on each multi-machine repeated set and each single-machine repeated set to obtain a global repeated set; wherein the global repeated set includes strings repeated in all computing nodes.

[0159] In a possible implementation, the obtaining module 601 is specifically configured to: determine a target data source; and distribute a plurality of first data sets in the target data source to M computing nodes; wherein the first data sets each include at least one string; perform duplicate checking on each string in the received first data set based on the computing node, to obtain a single-machine duplicate set and a bitmap array corresponding to the computing node; and obtain the bitmap array and the single-machine duplicate set of the M computing nodes based on a merging computing node.

[0160] In a possible implementation, the obtaining module 601 is specifically configured to: perform mapping processing on each string in the first data set based on the computing node, to obtain a mapping set, a local reverse dictionary, and a bitmap array corresponding to the computing node; wherein the mapping set includes a mapping value corresponding to each string; the local reverse dictionary includes a reverse mapping relationship of each mapping value being reversely mapped to a string; and perform reverse lookup processing on the local reverse dictionary and the bitmap array corresponding to the computing node, to obtain a single-machine duplicate set corresponding to the computing node.

[0161] In a possible implementation, the obtaining module 601 is specifically configured to: perform computing processing on each string in the first data set, to obtain an integer set corresponding to the first data set; wherein the integer set includes an integer value corresponding to each string; and the integer value represents a unique identifier of the business data; and determine the integer value corresponding to the string as a mapping value corresponding to the string.

[0162] The local reverse dictionary and the bitmap array are determined according to the strings and the mapping values corresponding to the strings.

[0163] In a possible implementation, the obtaining module 601 is specifically configured to: perform encryption processing on each string in the first data set, to obtain an encryption set corresponding to the first data set; wherein the encryption set includes encryption data corresponding to each string; and perform hash calculation on each encryption data in the encryption set, to obtain the integer set.

[0164] In a possible implementation, the obtaining module 601 is specifically configured to: perform bit operation processing on the bitmap array corresponding to the computing node, to obtain a mapping value set corresponding to the computing node; wherein the mapping value set includes at least one mapping subset corresponding to the computing node, and the mapping subset includes duplicate mapping values; perform reverse lookup processing on the mapping value set corresponding to the computing node according to the local reverse dictionary corresponding to the computing node, to determine an original data set corresponding to the computing node; wherein the original data set includes a data subset corresponding to each mapping subset; and the data subset includes a string corresponding to each duplicate mapping value in the mapping subset; perform comparison processing on each string in the data subset, to obtain a comparison result of the data subset; and if it is determined that the comparison result represents that each string in the data subset is the same, it is determined that each string in the data subset is a string in the single-machine duplicate set.

[0165] In a possible implementation, the target data source is stored in a single server; the acquisition module 601 is further configured to: determine the target data source from the plurality of data sources according to a data source configuration file; and wherein the data source configuration file represents a configuration of the data source.

[0166] In a possible implementation, the data source configuration file includes server information and database information; the server information represents a server configuration of the data source; and the database information includes a connection string and an instance state of a database where the data source is located.

[0167] In a possible implementation, the target data source includes a plurality of business data; the acquisition module 601 is further configured to: perform data splitting processing on the target data source to obtain M second data sets corresponding to the target data source; wherein each of the second data sets includes business data in at least one of the target data sources; perform calculation processing on each of the business data in the second data sets to obtain a first data set corresponding to the second data set; and distribute the M first data sets to M computing nodes evenly.

[0168] In a possible implementation, the merging module 604 is configured to: perform consistency processing on the multi-machine repeated sets and the single-machine repeated sets to obtain processed multi-machine repeated sets and processed single-machine repeated sets; and perform merging processing on each of the processed multi-machine repeated sets and each of the processed single-machine repeated sets to obtain the global repeated set.

[0169] In a possible implementation, the merging module 604 is configured to: perform merging processing on the processed multi-machine repeated set and two processed single-machine repeated sets corresponding to the processed multi-machine repeated set to obtain a first local repeated set corresponding to the processed multi-machine repeated set; and perform merging processing on each of the first local repeated sets to obtain the global repeated set.

[0170] In a possible implementation, the merging module 604 is further configured to: perform merging processing on each of the processed multi-machine repeated sets to obtain a second local repeated set corresponding to each of the processed multi-machine repeated sets; perform merging processing on each of the processed single-machine repeated sets to obtain a third local repeated set corresponding to each of the processed single-machine repeated sets; and perform merging processing on the second local repeated set and the third local repeated set to obtain the global repeated set.

[0171] In a possible implementation, the second processing module 603 is configured to: determine at least one string corresponding to the repeated value according to each of the local reverse dictionaries; and if each of the strings corresponding to the repeated value is determined to be the same, determine each of the strings corresponding to the repeated value as a string in the multi-machine repeated set corresponding to the repeated value.

[0172] In a possible implementation, the apparatus is further configured to perform deduplication processing on each string in the M computing nodes according to the bitmap array of the M computing nodes.

[0173] In a possible implementation, the apparatus is further configured to perform deduplication processing on each string in the M computing nodes according to the bitmap array of the M computing nodes.

[0174] In a possible implementation, the apparatus is further configured to perform deduplication processing on each string in the M computing nodes according to the bitmap array of the M computing nodes.

[0175] The apparatus of the embodiment can perform the technical solutions in the above method, and the specific implementation process and technical principles are the same, which will not be repeated here.

[0176] It should be noted that the division of each module of the above apparatus is only a logical function division, and all or part of the modules can be integrated into one physical entity, or can be physically separated. The modules can all be implemented in the form of software called by a processing element; or all be implemented in the form of hardware; or part of the modules are implemented in the form of software called by a processing element, and part of the modules are implemented in the form of hardware. Each module can be a separately established processing element, or can be integrated in a chip of the apparatus, in addition, the functions of each module can be stored in the memory of the apparatus in the form of program code, and called and executed by a processing element of the apparatus. In addition, all or part of the modules can be integrated together, or can be independently implemented. The processing element herein can be an integrated circuit with signal processing capability. In the implementation process, each step of the above method or each module can be completed by the integrated logic circuit of hardware or the instruction of software in the processing element.

[0177] FIG. 12 is a structural schematic diagram of a computing device provided by an embodiment of the present application. As shown in FIG. 12, the computing device includes a memory 701, a processor 702, and a memory 701.

[0178] The processor 702 is configured to perform the method provided by the above embodiment.

[0179] The computing device further includes a receiver 703 and a transmitter 704. The receiver 703 is configured to receive instructions and data transmitted by other devices, and the transmitter 704 is configured to transmit instructions and data to external devices.

[0180] The embodiment of the present application further provides a chip for running instructions, which is used for executing the technical solution of the processing method in the above embodiment.

[0181] The embodiment of the present application further provides a computer readable storage medium, which stores computer instructions, and when the computer instructions are run on a computer, the computer executes the technical solution of the processing method in the above embodiment.

[0182] The embodiment of the present application further provides a computer program product, which includes a computer program stored in a computer readable storage medium, at least one processor can read the computer program from the computer readable storage medium, and when the at least one processor executes the computer program, the technical solution of the processing method in the above embodiment can be implemented.

[0183] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other manners. For example, the above described device embodiments are merely illustrative, and for example, the division of the modules is merely a logical function division, and there can be another division manner in actual implementation, for example, a plurality of modules can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual coupling or direct coupling or communication connection between the modules can be indirect coupling or communication connection through some interfaces, devices or modules, and can be electrical, mechanical or other forms.

[0184] In addition, each function module in each embodiment of the present application can be integrated in a processing unit, or each module can exist independently physically, or two or more modules can be integrated in one unit. The above-mentioned modules can be realized in the form of hardware, or in the form of hardware plus software function modules.

[0185] The integrated modules realized in the form of software function modules can be stored in a computer readable storage medium. The above-mentioned software function modules stored in a storage medium include a plurality of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute part of the steps of the method of each embodiment of the present application.

[0186] It should be appreciated that the above-mentioned processor can be a central processing unit (CPU), and can also be other general-purpose processors, a digital signal processor (DSP), an application specific integrated circuit (ASIC), or the like. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the application can be directly embodied as the execution of the hardware processor, or the execution of the combination of hardware and software modules in the processor.

[0187] It should be noted that, for each of the above-mentioned method embodiments, in order to simply describe, each of them is expressed as a series of action combinations, but those skilled in the art should know that the application is not limited by the action order described, because according to the application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the application.

[0188] In the above embodiments, the description of each embodiment has its own emphasis, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments. Each technical feature of the above-mentioned embodiments can be combined arbitrarily, and in order to make the description simple, not all possible combinations of each technical feature in the above-mentioned embodiments are described, however, as long as the combination of these technical features does not exist contradictory, it should be considered as the scope of the disclosure.

[0189] Those skilled in the art will readily conceive other embodiments of the application upon considering the specification and practicing the application disclosed herein. The application is intended to cover any variations, uses, or adaptive changes of the application that follow the general principles of the application and include common knowledge or conventional technical means in the art that are not disclosed in the application. The specification and examples are only considered as exemplary, and the true scope and spirit of the application are indicated by the following claims.

[0190] It should be understood that the application is not limited to the precise construction that has been described above and illustrated in the accompanying drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the application is limited only by the claims that follow

[0191] The above merely provides the specific implementation of the present application, but the protection scope of the present application is not limited to this. Any person skilled in the art can easily think of the changes or replacements within the technical range disclosed by the present application, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method of determining duplicate data, characterized by, The method comprises the following steps: Based on the merging computing node, the bitmap array and the single machine repeat set of M computing nodes are obtained; wherein the bitmap array represents the mapping processing record of each string representing business data in the computing node; the single machine repeat set includes repeated strings; M is an integer greater than or equal to 1; The bitmap array of the M computing nodes is processed by two-bit operation to obtain M repeated values; wherein the repeated value represents the repeated mapping value in each two computing nodes; the mapping value represents the unique identifier of the business data; The local reverse dictionary corresponding to the computing node is obtained; and the repeated values are processed by secondary duplicate checking according to each local reverse dictionary to obtain the multi-machine repeat set corresponding to the repeated values; wherein the local reverse dictionary includes the reverse mapping relationship of each mapping value to the string; the multi-machine repeat set includes the repeated string in each two computing nodes; Each multi-machine repeat set and each single machine repeat set is merged to obtain a global repeat set; wherein the global repeat set includes the repeated string in all computing nodes.

2. The method of claim 1, wherein, Based on the merging computing node, the bitmap array and the single machine repeat set of M computing nodes are obtained, comprising: Determine the target data source; and distribute a plurality of first data sets in the target data source to the M computing nodes; wherein the first data set includes at least one string; Based on the computing node, each string in the received first data set is processed by duplicate checking to obtain the single machine repeat set and the bitmap array corresponding to the computing node; Based on the merging computing node, the bitmap array and the single machine repeat set of the M computing nodes are obtained.

3. The method of claim 2, wherein, Based on the computing node, each string in the received first data set is processed by duplicate checking to obtain the single machine repeat set and the bitmap array corresponding to the computing node, comprising: Based on the computing node, each string in the first data set is processed by mapping to obtain the mapping set, the local reverse dictionary and the bitmap array corresponding to the computing node; wherein the mapping set includes the mapping value corresponding to each string; the local reverse dictionary includes the reverse mapping relationship of each mapping value to the string; The local reverse dictionary and the bitmap array corresponding to the computing node are processed by reverse checking to obtain the single machine repeat set corresponding to the computing node.

4. The method of claim 3, wherein, Based on the computing node, each string in the first data set is processed by mapping to obtain the mapping set, the local reverse dictionary and the bitmap array corresponding to the computing node, comprising: Each string in the first data set is processed by calculation to obtain the integer set corresponding to the first data set; wherein the integer set includes the integer value corresponding to each string; the integer value represents the unique identifier of the business data; Determine the integer value corresponding to the string as the mapping value corresponding to the string; According to each string and the mapping value corresponding to each string, the local reverse dictionary and the bitmap array are determined.

5. The method of claim 4, wherein, The computing processing is performed on each string in the first data set to obtain an integer set corresponding to the first data set, including: The encryption processing is performed on each string in the first data set to obtain an encryption set corresponding to the first data set; wherein the encryption set includes encryption data corresponding to each string; The hash calculation is performed on each encryption data in the encryption set to obtain the integer set.

6. The method of claim 3, wherein, The local reverse dictionary and the bitmap array corresponding to the computing node are subjected to reverse lookup processing to obtain a single-computer repeated set corresponding to the computing node, including: The bitmap array corresponding to the computing node is subjected to bit operation processing to obtain a mapping value set corresponding to the computing node; wherein the mapping value set includes at least one mapping subset corresponding to the computing node, and the mapping subset includes repeated mapping values; According to the local reverse dictionary corresponding to the computing node, the mapping value set corresponding to the computing node is subjected to reverse lookup processing to determine an original data set corresponding to the computing node; wherein the original data set includes a data subset corresponding to each mapping subset; and the data subset includes strings corresponding to each repeated mapping value in the mapping subset; The comparison processing is performed on each string in the data subset to obtain a comparison result of the data subset; If it is determined that the comparison result represents that each string in the data subset is the same, it is determined that each string in the data subset is a string in the single-computer repeated set.

7. The method according to any one of claims 2-6, characterized in that, The target data source is stored in a single server; The target data source is determined, including: According to the data source configuration file, the target data source is determined from a plurality of data sources; wherein the data source configuration file represents the configuration of the data source.

8. The method of claim 7, wherein, The data source configuration file includes server information and database information; the server information represents the server configuration of the data source; and the database information includes a connection string and an instance state of a database where the data source is located.

9. The method according to any one of claims 2-6, characterized in that, The target data source includes a plurality of business data; and the plurality of first data sets in the target data source are distributed to the M computing nodes, including: The data splitting processing is performed on the target data source to obtain M second data sets corresponding to the target data source; wherein the second data sets include business data in at least one target data source; The computing processing is performed on each business data in the second data set to obtain the first data set corresponding to the second data set; The M first data sets are evenly distributed to the M computing nodes.

10. The method according to any one of claims 1-9, characterized in that, The plurality of multi-computer repeated sets and the plurality of single-computer repeated sets are subjected to merging processing to obtain a global repeated set, including: The consistency processing is performed on the multi-computer repeated sets and the single-computer repeated sets to obtain processed multi-computer repeated sets and processed single-computer repeated sets; The merging processing is performed on each of the processed multi-computer repeated sets and each of the processed single-computer repeated sets to obtain the global repeated set.

11. The method of claim 10, wherein, The merging processing is performed on each of the processed multi-computer repeated sets and each of the processed single-computer repeated sets to obtain the global repeated set, including: The processed multi-machine repeated set and the two processed single-machine repeated sets corresponding to the processed multi-machine repeated set are merged to obtain a first local repeated set corresponding to the processed multi-machine repeated set; The first local repeated sets are merged to obtain the global repeated set.

12. The method of claim 10, wherein, The processed multi-machine repeated sets and the processed single-machine repeated sets are merged to obtain the global repeated set, including: The processed multi-machine repeated sets are merged to obtain second local repeated sets corresponding to the processed multi-machine repeated sets; The processed single-machine repeated sets are merged to obtain third local repeated sets corresponding to the processed single-machine repeated sets; The second local repeated sets and the third local repeated sets are merged to obtain the global repeated set.

13. The method of any of claims 1-12, wherein, According to the local reverse dictionaries, the duplicate values are subjected to secondary duplicate checking to obtain a multi-machine repeated set corresponding to the duplicate values, including: According to the local reverse dictionaries, at least one string corresponding to the duplicate value is determined; If it is determined that each string corresponding to the duplicate value is the same, it is determined that each string corresponding to the duplicate value is a string in the multi-machine repeated set corresponding to the duplicate value.

14. The method of any one of claims 1-13, wherein, The method further includes: According to the bitmap array of the M computing nodes, each string in the M computing nodes is subjected to duplicate removal processing.

15. The method of claim 14, wherein, According to the bitmap array of the M computing nodes, each string in the M computing nodes is subjected to duplicate removal processing, including: According to the bitmap array of the M computing nodes and the local reverse dictionary, each string in the M computing nodes is subjected to duplicate removal calculation processing to obtain a duplicate-free data set; wherein the local reverse dictionary includes an inverse mapping relationship in which each mapping value is inversely mapped to a string; and the duplicate-free data set includes strings that are not duplicated in the M computing nodes.

16. The method of claim 15, wherein, The method further includes: According to the bitmap array of the M computing nodes and the local reverse dictionary, each string in the M computing nodes is subjected to aggregation calculation processing to obtain a duplicate-free number corresponding to the M computing nodes; wherein the duplicate-free number is the total number of different strings in the M computing nodes.

17. An apparatus for determining duplicate data, the apparatus comprising: including: The acquisition module is configured to acquire, based on a merging computing node, a bitmap array and a single-machine repeated set of M computing nodes; wherein the bitmap array represents mapping processing records of each string representing business data in a computing node; the single-machine repeated set includes repeated strings; M is an integer greater than or equal to 1; The first processing module is configured to perform two-by-two bit operation processing on the bitmap array of the M computing nodes to obtain M duplicate values; wherein the duplicate value represents a duplicated mapping value in each two computing nodes; and the mapping value represents a unique identifier of business data; The first processing module is configured to perform two-by-two bit operation processing on the bitmap array of the M computing nodes to obtain M duplicate values; wherein the duplicate value represents a duplicated mapping value in each two computing nodes; and the mapping value represents a unique identifier of business data; The second processing module is configured to acquire local reverse dictionaries corresponding to the computing nodes, and perform secondary duplicate checking on the duplicate values according to the local reverse dictionaries to obtain a multi-computer duplicate set corresponding to the duplicate values, wherein the local reverse dictionary includes a reverse mapping relationship of each mapping value being reversely mapped into a string, and the multi-computer duplicate set includes strings that are duplicated in each two computing nodes. The merging module is configured to perform merging processing on the multi-computer duplicate sets and the single-computer duplicate sets to obtain a global duplicate set, wherein the global duplicate set includes strings that are duplicated in all the computing nodes.

18. A computing device, comprising: The computer program product comprises a computer readable storage medium and a computer program. The computer readable storage medium stores computer execution instructions. The processor executes the computer execution instructions stored in the memory, so that the processor executes the method in any one of claims 1-16. The computer readable storage medium stores computer execution instructions.

19. A computer-readable storage medium, characterized in that, 20. A computer program product, comprising a computer program, wherein the computer program is executed by a processor to implement the method in any one of claims 1-16. ​

Images