Distributed data deduplication method and product

Abstract

The embodiment of the invention provides a distributed data deduplication method and product. The method comprises the steps of obtaining a data key of data to be subjected to deduplication; distributing the data to be subjected to duplicate removal to the ith sub-bucket according to the mixed hash value of the data key; mapping the mixed hash value according to a hash function attribute value configured for the ith sub-bucket to obtain k position numbers; reading numerical values of positions corresponding to the k position numbers from a bit array which is stored at the current moment and corresponds to the ith sub-bucket to obtain k numbers; if it is confirmed that the k numbers are all first numerical values, it is confirmed that the data to be subjected to duplicate removal belong to duplicate data; and if at least one of the k digits is confirmed to be a second numerical value, confirming that the data to be subjected to duplicate removal belongs to new data. According to the embodiment of the invention, the speed and accuracy of big data deduplication processing are remarkably improved, and the problem of single-node calculation overload (different buckets can be distributed to different calculation nodes) is effectively solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of big data processing, and in particular, the embodiments of the present application relate to a distributed data deduplication method and product. BACKGROUND

[0002] Data deduplication is a common problem in the field of big data. The significance of data deduplication at least lies in eliminating the impact of dirty data, i.e., repeatedly reported data or repeatedly delivered data, generated by unreliable data sources, so as to make the results of calculation more accurate.

[0003] With the sharp increase in data volume and real-time data processing scenarios, the previous big data deduplication scheme cannot meet the existing business scenarios and often causes single-node calculation overload, thereby significantly affecting the processing speed and accuracy of big data deduplication. SUMMARY

[0004] The purpose of the embodiments of the present application is to provide a distributed data deduplication method and product. The embodiments of the present application draw on the idea of handling hash conflicts in computer architecture (such as XOR of high and low bits of a hash table) and apply it to a completely different field, i.e., the distributed data bucketing process. This means of promoting a bottom-level bit operation skill to solve top-level architecture problems of a distributed system significantly improves the speed and accuracy of big data deduplication processing and effectively overcomes the problem of single-node calculation overload (different buckets can be allocated to different computing nodes).

[0005] In a first aspect, the embodiments of the present application provide a distributed data deduplication method, which comprises: obtaining a data key of to-be-deduplicated data, wherein the data key is a primary key or a pseudo primary key obtained according to a target business field; distributing the to-be-deduplicated data to an i-th bucket according to a mixed hash value of the data key, wherein i is an integer greater than or equal to 1, and the mixed hash value is obtained by fusing different parts of a hash value of the data key; mapping the mixed hash value according to a hash function attribute value configured for the i-th bucket to obtain k position numbers, wherein k is an integer greater than 1; reading values at positions corresponding to the k position numbers from a bit array corresponding to the i-th bucket stored at a current time to obtain k numbers, wherein the values of the bits in the bit array corresponding to the i-th bucket stored at the current time are determined by all historical non-duplicate data allocated to the i-th bucket or are determined by initialization; if it is confirmed that the k numbers are all first values, it is confirmed that the to-be-deduplicated data belongs to duplicate data; and if it is confirmed that at least one of the k numbers is a second value, it is confirmed that the to-be-deduplicated data belongs to new data.

[0006] Embodiments of the present application use a mixed hash value to distribute data to be deduplicated to a suitable bucket for processing, which can overcome the performance bottleneck of single node processing and improve the probability of distributing the same data to the same bucket, thereby ultimately improving the processing speed and accuracy of data deduplication.

[0007] In some embodiments, before the distributing the data to be deduplicated to the i-th sub-bucket according to the mixed hash value of the data key, the method further comprises: calculating a hash value of the data key; splitting the hash value into a high bit part and a low bit part; and performing an exclusive OR operation on the high bit part and the low bit part to obtain the mixed hash value.

[0008] In some embodiments, the distributing the data to be deduplicated to the i-th sub-bucket according to the mixed hash value of the data key comprises: performing a modulo operation on the mixed hash value and the total number of sub-buckets to obtain a bucket number corresponding to the i-th sub-bucket.

[0009] The traditional sub-bucketing can easily cause data skew (e.g., user IDs concentrated in a particular range), resulting in uneven node load. The exclusive OR algorithm of the embodiments of the present application breaks the original data distribution rule and avoids skew. The bit operation of the exclusive OR algorithm of the embodiments of the present application has a much higher processing speed than the process using a more complex Hash function, thereby improving data processing speed. The bucket number algorithm of the embodiments of the present application can ensure that the same data is always mapped to the same bucket, thereby improving the accuracy of the data deduplication result.

[0010] In some embodiments, before the reading the values of the positions corresponding to the k position numbers from the bit array corresponding to the i-th sub-bucket stored at the current time, the method comprises: at a p-th historical time, assigning j-th historical non-repeated data to the i-th sub-bucket, where p and j are integers greater than or equal to 1; using k hash functions to map the mixed hash value of the j-th historical non-repeated data to obtain k updated position numbers; setting the numbers of the positions corresponding to the k updated position numbers in the bit array corresponding to the i-th sub-bucket stored at the p-th historical time to the first value; and repeating the above process until the above operations are performed for all historical non-repeated data before the current time, thereby obtaining the bit array corresponding to the i-th sub-bucket stored at the current time.

[0011] Embodiments of the present application provide a method for updating each bit data in the bit array of the i-th sub-bucket, i.e., new data entering the bucket sets the numbers of the mapped positions to the first value, otherwise the data of these positions is the second value, thereby enabling the technical solution of the present application to determine whether the data to be analyzed that needs to be determined recently belongs to repeated data according to the values of each bit in the latest bit array, significantly reducing the complexity of repeated data identification and improving the processing speed.

[0012] In some embodiments, the method further comprises: initializing a Bloom filter for the ith sub-bucket, and setting the following parameters for the Bloom filter: size of the bit array and number of hash functions; for each piece of historical data allocated to the ith sub-bucket: obtaining a data key of the historical data; mapping a hybrid hash value of the historical data using a set number of hash functions to obtain k historical position numbers; reading values at positions corresponding to the k historical position numbers from a bit array corresponding to the ith sub-bucket stored at a historical corresponding time to obtain k historical numbers, wherein the historical corresponding time is a time at which the ith sub-bucket receives the historical data; if there is a second value in the k historical numbers, setting the numbers of the k historical positions in the bit array corresponding to the ith sub-bucket stored at the historical corresponding time to the first value and outputting the historical data; and if the values of the k historical numbers are all the first value, confirming that the historical data is repeated data to be discarded.

[0013] Some embodiments of the present application provide a process of setting values of corresponding bits in a bit array corresponding to an ith sub-bucket according to historical data allocated to the ith sub-bucket, through which each piece of data allocated to the ith sub-bucket can be recorded in real time through the bit array, and the accuracy of the identification result of whether subsequent data is repeated data can be improved.

[0014] In some embodiments, the first value is a number 1, and the second value is a number 0, and each bit of the bit array is initialized to 0.

[0015] In some embodiments, the obtaining of the data key of the data to be deduplicated comprises: confirming that the data to be deduplicated has a primary key corresponding to a unique business identifier; and taking the primary key as the data key.

[0016] Some embodiments of the present application can be applied to the case where there is a primary key, and the generality of the technical solution is improved.

[0017] In some embodiments, the obtaining of the data key of the data to be deduplicated comprises: when it is confirmed that the data to be deduplicated does not have a primary key corresponding to a unique business identifier, selecting a plurality of target business fields; calculating an MD5 value by splicing the plurality of target business fields to obtain a pseudo-primary key, and taking the pseudo-primary key as the data key.

[0018] Some embodiments of the present application can be applied to the case where there is no primary key, and the generality of the technical solution is improved.

[0019] In a second aspect, some embodiments of the present application provide a computer readable storage medium having stored thereon a computer program which, when executed by a processor, can implement the method according to any one of the embodiments included in the first aspect.

[0020] In a third aspect, some embodiments of the present application provide a computer program product comprising computer program instructions, which, when read and executed by a processor, can implement the method according to any one of the embodiments included in the first aspect.

[0021] In a fourth aspect, some embodiments of the present application provide an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, can implement the method according to any one of the embodiments included in the first aspect.

[0022] In a fifth aspect, some embodiments of the present application provide a device for distributed data deduplication, the device comprising: a data key obtaining module configured to obtain a data key of to-be-deduplicated data, wherein the data key is a primary key or a pseudo primary key obtained according to a target business field; a bucket number determining module configured to allocate the to-be-deduplicated data to an i-th bucket according to a hybrid hash value of the data key, wherein i is an integer greater than or equal to 1, and the hybrid hash value is obtained by fusing different parts of a hash value of the data key; a position number determining module configured to map the hybrid hash value according to a hash function attribute value configured for the i-th bucket to obtain k position numbers, wherein k is an integer greater than 1; a bit array number reading module configured to read values at positions corresponding to the k position numbers from a bit array corresponding to the i-th bucket stored at a current time to obtain k numbers, wherein the values of each bit in the bit array corresponding to the i-th bucket stored at the current time are determined by all historical non-duplicate data allocated to the i-th bucket or are determined by initialization; and a duplication result determining module configured to: if it is confirmed that the k numbers are all first values, then confirm that the to-be-deduplicated data belongs to duplicate data; and if it is confirmed that at least one of the k numbers is a second value, then confirm that the to-be-deduplicated data belongs to new data. BRIEF DESCRIPTION OF DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments of the present application. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation on the scope, and for those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0024] Figure 1 A composition diagram of a distributed deduplication system provided by an embodiment of the present application is shown in FIG. 1. Figure 2 A flowchart of a method of distributed data deduplication provided by an embodiment of the present application is shown in FIG. 2. Figure 3 A flowchart of a method of distributed data deduplication provided by an embodiment of the present application is shown in FIG. 3. Figure 4 A composition module diagram of an apparatus of distributed data deduplication provided by an embodiment of the present application is shown in FIG. 4. Figure 5 A composition diagram of an electronic device provided by an embodiment of the present application is shown in FIG. 5. DETAILED DESCRIPTION

[0025] The technical solutions in the embodiments of the present application will be described below with reference to the accompanying drawings.

[0026] It should be noted that similar reference numerals and letters represent similar items in the following drawings, and thus, once an item is defined in one drawing, it need not be further defined and explained in subsequent drawings. Meanwhile, in the description of the present application, the terms "first", "second", and the like are merely used to distinguish description, and cannot be understood as indicating or implying relative importance.

[0027] Compared with the data deduplication scheme provided by the related art, the data deduplication method provided by the embodiments of the present application uses a distributed computing idea, specifically adopts a mixed hash value to distribute data to a corresponding bucket, so that different buckets can use different computing nodes for operation processing to realize computing parallelization, avoiding single-node computing overload, and meanwhile, the data deduplication method of the embodiments of the present application supports data deduplication in both cases of having a primary key and not having a primary key, and the technical solution has stronger universality. For example, some embodiments of the present application perform high-low bit XOR operation on the hash value of the primary key or pseudo-primary key corresponding to the data to obtain a mixed hash value, try to uniformly hash, distribute the mixed hash values in the same bucket (the same bucket corresponds to the same computing resource), and then use distributed and parallel computing to realize data deduplication.

[0028] Please refer to Figure 1 , Figure 1 An architecture diagram of a distributed deduplication system provided by some embodiments of the present application is shown in FIG. 6, which includes a data source layer, a data collection layer, a distributed computing layer, a control and management layer, and a result storage layer. The data source layer includes one or more data source servers, which can generate original logs / data (i.e. data to be deduplicated or historical data or historical non-duplicate data). The data collection layer receives data pushed by the data source layer and integrates a message queue cluster (such as kafka decoupling and buffering). The i-th computing server (i takes a value of 1, 2, …, n, where n is the number of computing servers) in the distributed computing layer is connected to the data collection layer and the (i+1)-th computing server is connected to the i-th computing server. The control and management layer is connected to the data collection layer and the distributed computing layer. The result storage layer is connected to the distributed computing layer. Figure 1CPU of the first computing server of the i-th computing server (1 to N) is used for hybrid hash value calculation and repeated data determination, and the memory of the i-th computing server is used to carry a plurality of buckets (for example, Figure 1 The first computing server of the i-th computing server (1 to N) carries a bit array BF and the like of the 1-N buckets, and a result storage layer is used to receive non-repeated data written from each computing server. The result storage layer is provided with a distributed file system (for example, the system includes HDFS or S3, etc.), and the distributed file storage system is used to store the final result after deduplication. Figure 1 The control and management layer of the i-th computing server is provided with a management server, which is deployed with a monitoring system and a resource scheduler. All computing nodes can be dynamically managed through the monitoring system and the resource scheduler. The data type output by the data collection layer can be consumption data or other types of data. If the data collection layer outputs consumption data, it is not difficult to understand that the data to be deduplicated, historical data or historical non-repeated data involved in the embodiments of the application are also consumption data.

[0029] Corresponding to Figure 1 The data processing flow of the distributed deduplication system includes: the data to be deduplicated is generated on a source server and pushed to a Kafka cluster. A plurality of computing servers pull data from the Kafka in parallel. Each computing server independently performs the following operations on each piece of data to be deduplicated: a. Determine the data key. b. The CPU performs hybrid hash value calculation to determine the target bucket number. c. Check the numbers in a plurality of positions in the bit array of the corresponding bucket in the memory. d. According to the judgment result, output the data or discard. All non-repeated data is written to a distributed file system. The management server monitors the entire process to ensure healthy operation.

[0030] The embodiments of the application can efficiently utilize modern distributed computing hardware resources (multi-core CPU, large memory, high-speed network).

[0031] The following will be combined Figure 2 with an example to illustrate the implementation process of the distributed deduplication method performed by any one computing server on any piece of data to be deduplicated.

[0032] As shown in Figure 2 , the embodiments of the application provide a distributed data deduplication method, which includes: S110, obtaining a data key of data to be deduplicated, wherein the data key is a primary key or a pseudo primary key obtained according to a target business field.

[0033] It should be noted that in some embodiments of the application, the data to be deduplicated can be obtained from Figure 1The data key of the to-be-de-duplicated data is a primary key (for data having a primary key) in some embodiments of the present application, and is a pseudo primary key generated according to a plurality of target business fields in some other embodiments of the present application.

[0034] For example, in some embodiments of the present application, the S110 exemplarily comprises: confirming that the to-be-de-duplicated data has a primary key corresponding to a unique business identifier; and taking the primary key as the data key. That is, some embodiments of the present application can be applied to a case where a primary key exists, thereby improving the universality of the technical solution.

[0035] For example, in some embodiments of the present application, the S110 exemplarily comprises: when it is confirmed that the to-be-de-duplicated data does not have a primary key corresponding to a unique business identifier, selecting a plurality of target business fields; calculating an MD5 value after splicing the plurality of target business fields to obtain the pseudo primary key, and taking the pseudo primary key as the data key. That is, some embodiments of the present application can be applied to a case where a primary key does not exist, thereby improving the universality of the technical solution.

[0036] S120, according to the mixed hash value of the data key, the to-be-de-duplicated data is distributed to the i-th sub-bucket, wherein i is an integer greater than or equal to 1, and the mixed hash value is obtained by fusing different parts of the hash value of the data key.

[0037] It should be noted that the purpose of the embodiments of the present application in adopting the mixed hash value is to ensure uniform distribution of data and avoid skewing. In the embodiments of the present application, the mixed hash value is obtained by fusing different parts (for example, different parts include high-bit parts, low-bit parts, or middle-bit parts; different parts can include high-bit parts and low-bit parts) of the hash value of the data key, and the fusing processing includes exclusive OR, addition after bit rotation, and AND or other bit operation processing.

[0038] For example, in some embodiments of the present application, the S120 exemplarily comprises: calculating the hash value of the data key; splitting the hash value into a high-bit part and a low-bit part; performing exclusive OR operation on the high-bit part and the low-bit part to obtain the mixed hash value; and performing modulo operation on the mixed hash value and the total number of sub-buckets to obtain a bucket number corresponding to the i-th sub-bucket. Traditional sub-bucketing is prone to cause data skewing (such as user IDs concentrated in a specific range), resulting in uneven node load. The exclusive OR algorithm of the embodiments of the present application breaks the original data distribution rule and avoids skewing; the bit operation of the exclusive OR of the embodiments of the present application has a much higher processing speed than the process of fusing the hash value by using a more complex Hash function, thereby improving the data processing speed.

[0039] Embodiments of the present application determine the number of the corresponding bucket of each data to be deduplicated according to the mixed hash value, so that the same data is always mapped to the same bucket, thereby improving the accuracy of the deduplication result.

[0040] In S130, the mixed hash value is mapped according to the hash function attribute value configured for the i-th bucket, to obtain k position numbers, where k is an integer greater than 1.

[0041] For example, the hash function attribute value can be the number of hash functions. For example, if the hash function attribute value is 4, S130 exemplarily includes: using four hash functions to map the mixed hash value to obtain four position numbers, which are used to indicate the corresponding digits of the bit array corresponding to the i-th bucket.

[0042] In some embodiments of the present application, S130 exemplarily includes: calculating the hash value of the data key; splitting the hash value into a high bit part and a low bit part; performing XOR operation on the high bit part and the low bit part to obtain the mixed hash value; performing k hash function mappings according to the mixed hash value to obtain k position numbers. It should be noted that the k position numbers are used to specify the bit number of the bit array.

[0043] In S140, the values at the positions corresponding to the k position numbers in the bit array corresponding to the i-th bucket stored at the current time are read to obtain k numbers, where the values of each bit in the bit array corresponding to the i-th bucket stored at the current time are determined by all historical non-repeated data allocated to the i-th bucket or are determined by initialization.

[0044] In some embodiments of the present application, the data to be deduplicated is the first data to be processed allocated to the i-th bucket, and the bit array corresponding to the i-th bucket stored at the current time is a bit array initialized to all zeros. For example, if the size of the bit array is 8, the value of the bit array corresponding to the i-th bucket stored at the current time is: 00000000, where the k position numbers obtained by calculation are used to indicate multiple bits in the eight bits, for example, the k position numbers are: 1, 3 and 5, and the k numbers obtained at this place are the numbers at the first bit, the third bit and the fifth bit of the bit array, respectively.

[0045] In some embodiments of this application, if the data to be deduplicated is not the first data to be deduplicated allocated to the i-th bucket, then the bit array stored at the current time corresponding to the i-th bucket is determined by all non-duplicate data to be deduplicated (i.e., historical non-duplicate data) allocated to that bucket before the current time. For example, in some embodiments of this application, before reading the value of the position corresponding to the k position numbers from the bit array stored at the current time corresponding to the i-th bucket, the method includes the following process of obtaining the bit array stored at the current time corresponding to the i-th bucket: at the p-th historical time, allocate the j-th historical non-duplicate data to the i-th bucket, where p and j are both integers greater than or equal to 1; use k hash functions (k is set when constructing the i-th bucket, and the value of k is an integer greater than 1) to hash the j-th historical non-duplicate data (which may be generated by...). Figure 1 The mixed hash value (obtained by referring to the algorithm for obtaining the mixed hash value described above) of historical non-duplicate data (i.e., data to be deduplicated that was allocated to the i-th bucket before the current time and determined to be non-duplicate data) generated by the data source layer is mapped to obtain k update position numbers; the numbers corresponding to the k update position numbers in the bit array stored at the p-th historical time corresponding to the i-th bucket are set to the first value; the above process is repeated until the above operation is performed on all historical non-duplicate data before the current time, to obtain the bit array stored at the current time corresponding to the i-th bucket. Embodiments of this application provide a method for updating the bits in the bit array corresponding to the i-th bucket, where new data entering the bucket sets the numbers at multiple mapped positions to the first value; otherwise, the data at these positions is set to the second value. This allows the technical solution of this application to determine whether the recently analyzed data to be analyzed belongs to duplicate data based on the latest bit array values, significantly reducing the complexity of duplicate data identification.

[0046] S151, if it is confirmed that all k numbers obtained in S140 are the first value, then the data to be deduplicated is confirmed to be duplicate data. Alternatively, S152, if it is confirmed that at least one of the k numbers obtained in S140 is the second value, then the data to be deduplicated is confirmed to be new data.

[0047] The embodiments of this application use mixed hash values ​​to allocate the data to be deduplicated to appropriate buckets for processing. This can overcome the performance bottleneck of single-node processing and increase the probability of allocating the same data to the same bucket, thereby ultimately improving the processing speed and accuracy of data deduplication.

[0048] The following exemplifies the processing method for historical data allocated to the i-th bucket. In some embodiments of this application, the method further includes: initializing a Bloom filter for the i-th bucket and setting the following parameters for the Bloom filter: the size of the bit array and the number of hash functions; for each piece of historical data allocated to the i-th bucket: obtaining the data key of the historical data; mapping the mixed hash value of the historical data using the set k number of hash functions to obtain k historical position numbers; reading the values ​​corresponding to the k historical position numbers from the bit array stored at the corresponding historical time corresponding to the i-th bucket to obtain k historical numbers, wherein the corresponding historical time is the time when the i-th bucket receives the historical data; if there is a second value among the k historical numbers, then setting the numbers of the k historical positions in the bit array stored at the corresponding historical time corresponding to the i-th bucket to the first value and outputting the historical data; if the values ​​of the k historical numbers are all the first value, then confirming that the historical data is duplicate data and needing to be discarded. Some embodiments of this application provide a technical solution for determining whether each piece of historical data allocated to the i-th bucket is duplicate data.

[0049] For example, in some embodiments of this application, the first value is the number 1, and the second value is the number 0, and all digits of the bit group are initialized to 0.

[0050] The following is combined Figure 3 The present application provides exemplary methods for distributed data deduplication in some embodiments.

[0051] Some embodiments of the present application provide a distributed data deduplication method. The method can be based on a big data distributed computing framework. For data to be deduplicated, high and low bit XOR value hash is performed for bucketing (supporting two scenarios: with primary key (directly bucketing) and without primary key (bucketing after MD5 of core field)). Then, for the data in each bucket, separate deduplication is performed. For data with unique business identifier, deduplication is performed according to the unique identifier. For data without unique identifier, deduplication is performed using MD5 of multiple business fields. The deduplication logic is to retain a bit array for each bucket. First, the bit array is initialized. Each bit of the bit array in the initialization state is set to 0. For the unique identifier (i.e., primary key) in the set to be deduplicated, the unique identifier is hashed. Then, the high 16 bits of the hash value are XORed with the low 16 bits of the hash value to obtain a new value (as an example of a hybrid hash value). The new value is sequentially mapped through k hash functions. Each mapping produces a hash value, which corresponds to a bit in the bit array. Then, the corresponding position of the bit array is marked as 1 (as an example of the first value). When querying whether an element exists in the set, the same method is used to hash the unique identifier and obtain a hybrid hash value through high and low bit XOR. Then, the hybrid hash value is mapped to 3 positions in the bit array through 3 hash functions (as an example of k position numbers). If any of the 3 positions is not 1, it can be determined that the element does not exist in the set. Conversely, if all 3 positions are 1, the element (i.e., the data to be deduplicated) may exist in the set. The calculation is distributed and parallelized as much as possible to avoid single node overload. The data is preprocessed for bucketing to distribute the calculation to each node to achieve maximum parallelization, reuse, and data balancing. It should be noted that the method of using high 16 bit XOR low 16 bit when calculating the hash value in some embodiments of the present application is to reduce collision and balance data distribution. For a given 32-bit hashCode, this technique achieves this purpose by left shifting the high 16-bit hashCode by 16 bits without changing it, and then XORing it with the original hashCode. This method can maintain the distribution of hash values while avoiding collisions between different keys as much as possible.

[0052] As shown in Figure 3 , the data to be deduplicated in some embodiments of the present application includes offline data and real-time data. The unique identifiers corresponding to the data are extracted. In the case of having a primary key (i.e., unique identifier, which needs to perform the step of judging whether it has a unique identifier), high and low bit XOR Hash is performed for bucketing calculation according to the primary key (for example, the bucketing numbers include: A, B, C, and D). In the case of not having a primary key, MD5 is performed on the core business field (for example, the bucketing numbers include: A, B, C, and D). Figure 3 Figure 3 Figure 3 ​​The key business field MD5 is shown, and then the MD5 is subjected to high-low bit XOR Hash bucket calculation. Data with the same Hash value must be distributed into the same bucket, and then deduplication is performed in the same bucket. Finally, the deduplicated data is summarized. Figure 3 An example of a bit array corresponding to a respective bucket stored at a current moment corresponding to four sub-buckets is schematically shown. The value of the bit array corresponding to bucket A is: 101000010, the value of the bit array corresponding to bucket B is: 011000111, the value of the bit array corresponding to bucket C is: 000110000, and the value of the bit array corresponding to the fourth bucket is: 100000001. That is, in some embodiments of the present application, the method for distributed data deduplication includes: placing the unique identifier of the data in the bucket into the array, and if there is no unique identifier, taking several core fields to perform MD5, and then storing it into the array; polling the array, and then taking the unique identifier to perform hash calculation, and then performing XOR calculation on the high 16 bits of the hash value and the low 16 bits of the hash value to obtain a new value; and obtaining three position numbers according to the new value by three hash function mappings. If the three values corresponding to the three position numbers in the bit array are all 1, it indicates that the data is duplicated, and the data is marked as duplicated data.

[0053] Please refer to Figure 4 , Figure 4 An apparatus for distributed data deduplication is shown, and it should be understood that the apparatus corresponds to the above-mentioned Figure 2 or Figure 3 method embodiments, and can perform each step involved in the above-mentioned method embodiments. The specific functions of the apparatus can be referred to the description above, and the detailed description is appropriately omitted here to avoid repetition. The apparatus includes at least one software function module stored in the form of software or firmware in the memory or solidified in the operating system of the apparatus. The distributed data deduplication apparatus includes: A data key acquisition module 210 is configured to acquire a data key of data to be deduplicated, wherein the data key is a primary key or a pseudo primary key obtained according to a target business field.

[0054] A sub-bucket number determination module 220 is configured to distribute the data to be deduplicated to an i-th sub-bucket according to a mixed Hash value of the data key, wherein i is an integer greater than or equal to 1, and the mixed Hash value is obtained by fusion processing different parts of a Hash value of the data key.

[0055] A position number determination module 230 is configured to map the mixed Hash value according to a Hash function attribute value configured for the i-th sub-bucket to obtain k position numbers, wherein k is an integer greater than 1. The bit array number reading module 240 is configured to read the values ​​corresponding to the k position numbers from the bit array corresponding to the i-th bucket stored at the current time, and obtain k numbers. The values ​​of each bit in the bit array corresponding to the i-th bucket stored at the current time are determined by all historical non-repeating data allocated to the i-th bucket or by initialization.

[0056] The duplicate result determination module 250 is configured to: if it is confirmed that all k numbers are first values, then the data to be deduplicated is confirmed to be duplicate data; if it is confirmed that at least one of the k numbers is a second value, then the data to be deduplicated is confirmed to be new data.

[0057] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the device described above can be referred to the corresponding process in the aforementioned method, and will not be elaborated further here.

[0058] Some embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can implement the method described in any of the embodiments of the distributed data deduplication method described above.

[0059] Some embodiments of this application provide a computer program product, including computer program instructions, which, when read and executed by a processor, can implement the method described in any of the embodiments of the distributed data deduplication method described above.

[0060] like Figure 5 As shown, some embodiments of this application provide an electronic device 400, which includes a memory 410, a processor 420, and a computer program stored in the memory 410 and executable on the processor 420. When the processor 420 reads and executes the computer program via a bus 430, it can implement the data synchronization method as described in any of the above embodiments.

[0061] Processor 420 can process digital signals and may include various computing architectures. For example, it may be a complex instruction set computer architecture, a reduced instruction set computer architecture, or an architecture that implements multiple instruction set combinations. In some examples, processor 420 may be a microprocessor.

[0062] Memory 410 can be used to store instructions executed by processor 420 or data related to the execution of instructions. These instructions and / or data may include code used to implement some or all of the functions of one or more modules described in the embodiments of this application. The processor 420 of the embodiments of this disclosure can be used to execute the instructions in memory 410 to implement… Figure 2or Figure 3 Memory 410 includes dynamic random access memory, static random access memory, flash memory, optical memory, or other memory well known to those skilled in the art.

[0063] In several embodiments provided in the present application, it should be understood that the disclosed apparatus and method can also be implemented by other means. The apparatus embodiments described above are only illustrative, for example, the flowcharts and block diagrams in the drawings show the possible implementation architecture, function and operation of the apparatus, method and computer program product according to the embodiments of the present application. In this regard, each block in the flowchart or block diagram can represent a module, a program segment or a part of code, which includes one or more executable instructions for implementing the specified logic function. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in different order from that shown in the drawings. For example, two consecutive blocks can actually be executed substantially in parallel, and they can also be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and the combination of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system for performing the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0064] In addition, the functional modules in the various embodiments of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0065] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0066] The above merely provides an example of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application. It should be noted that similar reference numerals and letters represent similar items in the following drawings, and thus, once an item is defined in one drawing, it need not be further defined and explained in subsequent drawings.

[0067] The above merely provides an example of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application. It should be noted that similar reference numerals and letters represent similar items in the following drawings, and thus, once an item is defined in one drawing, it need not be further defined and explained in subsequent drawings.

[0068] It should be noted that the relational terms herein such as first and second and the like are used solely to distinguish one entity or action from another, without necessarily requiring or implying any actual relationship or order between such entities or actions. Moreover, the terms "comprises", "comprising", or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, exclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

Claims

1. A method for distributed data deduplication, the method comprising: The method comprises: obtaining a data key of to-be-de-duplicated data, wherein the data key is a primary key or a pseudo primary key obtained according to a target business field; allocating the to-be-de-duplicated data to an i-th sub-bucket according to a mixed hash value of the data key, wherein i is an integer greater than or equal to 1, and the mixed hash value is obtained by fusing different parts of a hash value of the data key; mapping the mixed hash value using a hash function attribute value configured for the i-th sub-bucket to obtain k position numbers, wherein k is an integer greater than 1; reading values of positions corresponding to the k position numbers from a bit array corresponding to the i-th sub-bucket stored at a current time to obtain k numbers, wherein the values of bits in the bit array corresponding to the i-th sub-bucket stored at the current time are determined by all historical non-repeated data allocated to the i-th sub-bucket or are determined by initialization; if it is confirmed that the k numbers are all first values, it is confirmed that the to-be-de-duplicated data belongs to repeated data; if it is confirmed that at least one of the k numbers is a second value, it is confirmed that the to-be-de-duplicated data belongs to new data.

2. The method of claim 1, wherein, Before the to-be-de-duplicated data is allocated to the i-th sub-bucket according to the mixed hash value of the data key, the method further comprises: calculating a hash value of the data key; splitting the hash value into a high-bit part and a low-bit part; performing exclusive or operation on the high-bit part and the low-bit part to obtain the mixed hash value.

3. The method of claim 2, wherein, The allocating of the to-be-de-duplicated data to the i-th sub-bucket according to the mixed hash value of the data key comprises: performing modulo operation on the mixed hash value and a total number of sub-buckets to obtain a bucket number corresponding to the i-th sub-bucket.

4. The method of claim 1, wherein, Before the values of positions corresponding to the k position numbers are read from the bit array corresponding to the i-th sub-bucket stored at the current time, the method comprises: at a p-th historical time, allocating j-th historical non-repeated data to the i-th sub-bucket, wherein p and j are both integers greater than or equal to 1; mapping a mixed hash value of the j-th historical non-repeated data using k hash functions to obtain k updated position numbers; setting numbers of positions corresponding to the k updated position numbers in a bit array corresponding to the i-th sub-bucket stored at the p-th historical time to the first values; repeating the above process until the above operation is performed on all historical non-repeated data before the current time to obtain the bit array corresponding to the i-th sub-bucket stored at the current time.

5. The method of claim 1, wherein, The method further comprises: initializing a Bloom filter for the i-th sub-bucket and setting the following parameters for the Bloom filter: size of a bit array and number of hash functions; for each piece of historical data allocated to the i-th sub-bucket: obtaining a data key of the historical data; mapping a mixed hash value of the historical data using a set number of hash functions to obtain k historical position numbers; reading values of positions corresponding to the k historical position numbers in a bit array corresponding to the i-th sub-bucket stored at a historical corresponding time, to obtain k historical numbers, wherein the historical corresponding time is a time at which the i-th sub-bucket receives the historical data; if there is the second value in the k historical numbers, setting the k historical numbers in the bit array corresponding to the i-th sub-bucket stored at the historical corresponding time to the first value and outputting the historical data; if values of the k historical numbers are all the first value, confirming that the historical data is repeated data to be discarded.

6. The method of claim 2, wherein, The method further includes: confirming that the data to be deduplicated has a primary key corresponding to a unique business identifier; taking the primary key as the data key.

7. The method of claim 2, wherein, The method further includes: when it is confirmed that the data to be deduplicated does not have a primary key corresponding to a unique business identifier, selecting a plurality of target business fields; splicing the plurality of target business fields to calculate an MD5 value to obtain a pseudo-primary key, and taking the pseudo-primary key as the data key.

8. A computer-readable storage medium, characterized in that, A computer program is stored thereon, and when the program is executed by a processor, the method of any one of claims 1-7 can be implemented.

9. A computer program product, characterised in that, Computer program instructions are included, and when the computer program instructions are read and run by a processor, the method of any one of claims 1-7 can be implemented.

10. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein, When the processor executes the computer program, the method of any one of claims 1-7 can be implemented.

Images