Data deduplication method and device, computer equipment and storage medium

Abstract

The invention relates to a data deduplication method and device, computer equipment and a storage medium. The method comprises the following steps: dividing a plurality of pieces of data to be subjected to duplicate removal into at least one data subset; each data subset corresponds to an independent probability filter; for each data subset, determining a mapping value of each data to be subjected to duplicate removal in the data subset according to a probability filter corresponding to the data subset; according to the mapping value of each piece of data to be subjected to duplicate removal in the data subset, respectively and correspondingly determining a data filtering result of each piece of data to be subjected to duplicate removal in the data subset; the data filtering result is used for representing whether the to-be-deduplicated data exists in a probability filter or not; and performing duplicate removal processing on the data subset according to the data filtering result of the data to be subjected to duplicate removal in the data subset. According to the invention, the data deduplication precision and efficiency can be improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data deduplication method, device, computer equipment and storage medium. Background Art

[0002] With the development of big data technology, data deduplication has become a key link in data preprocessing, which is crucial to reducing storage overhead and improving data deduplication efficiency.

[0003] The core idea of ​​data deduplication is to identify and delete duplicate data through some method. Common data deduplication methods include hash tables, sorting methods, Bloom filters, etc. Take Bloom filters as an example. Bloom filters are a probabilistic data structure with extremely high space efficiency. They can determine whether an element exists in a collection with very little memory consumption and time cost, so as to quickly search and deduplicate data. However, the false alarm rate of Bloom filters is high, resulting in low data deduplication accuracy. Summary of the invention

[0004] Based on this, it is necessary to provide a data deduplication method, device, computer equipment and storage medium that can improve deduplication efficiency and accuracy in response to the above technical problems.

[0005] In a first aspect, the present application provides a data deduplication method, the method comprising:

[0006] Divide the multiple data to be deduplicated into at least one data subset; each data subset corresponds to an independent probability filter;

[0007] For each data subset, determining a mapping value of each to-be-deduplicated data in the data subset according to a probability filter corresponding to the data subset;

[0008] According to the mapping values ​​of each to-be-deduplicated data in the data subset, respectively correspondingly determining the data filtering results of each to-be-deduplicated data in the data subset; the data filtering results are used to indicate whether the to-be-deduplicated data exists in the probability filter;

[0009] Deduplication processing is performed on the data subset according to the data filtering results of each to-be-deduplicated data in the data subset.

[0010] In one of the embodiments, dividing the plurality of to-be-deduplicated data into at least one data subset includes:

[0011] Respectively obtain data features of each of the to-be-deduplicated data;

[0012] Determine the local sensitive hash value of each of the to-be-deduplicated data based on the local sensitive hash function according to the data characteristics of each of the to-be-deduplicated data;

[0013] According to the local sensitive hash value of each of the to-be-deduplicated data, each of the to-be-deduplicated data is divided into at least one data subset.

[0014] In one embodiment, the probability filter includes a Bloom filter, and the mapping value includes a hash value, wherein determining the mapping value of each to-be-deduplicated data in the data subset according to the probability filter corresponding to the data subset includes:

[0015] Determining a hash function corresponding to the data subset according to the Bloom filter corresponding to the data subset;

[0016] The hash value of each to-be-deduplicated data in the data subset is determined respectively according to the hash function.

[0017] In one embodiment, the probability filter includes a bit array, and the data filtering results of each to-be-deduplicated data in the data subset are determined according to the mapping values ​​of each to-be-deduplicated data in the data subset, respectively, including:

[0018] Traversing the values ​​of the mapping values ​​of each to-be-deduplicated data in the data subset at the corresponding positions in the bit array respectively;

[0019] For any one of the data to be deduplicated in the data subset, if there is 0 among the traversed numerical values ​​corresponding to the data to be deduplicated, it is determined that the data filtering result of the data to be deduplicated is not present in the probability filter; if there is no 0 among the traversed numerical values ​​corresponding to the data to be deduplicated, it is determined that the data filtering result of the data to be deduplicated is present in the probability filter.

[0020] In one embodiment, performing deduplication processing on the data subset according to the data filtering result of each to-be-deduplicated data in the data subset includes:

[0021] For any one of the to-be-deduplicated data in the data subset, if the data filtering result of the to-be-deduplicated data is that the to-be-deduplicated data exists in the probability filter, storing the to-be-deduplicated data in the duplicate database corresponding to the data subset;

[0022] When the data filtering result of the to-be-deduplicated data is that the to-be-deduplicated data does not exist in the probability filter, the to-be-deduplicated data is stored in a result database corresponding to the data subset.

[0023] In one embodiment, the method further comprises:

[0024] Determining the false alarm rate of the probability filter according to the result database and the duplicate database corresponding to the data subset;

[0025] When the false alarm rate exceeds the allowable error range of the probability filter, the probability filter is adjusted, and the adjusted probability filter is used to determine the data filtering results of subsequent data to be deduplicated until the false alarm rate meets the allowable error range.

[0026] In one embodiment, adjusting the probability filter of the data subset includes:

[0027] adjusting a filter parameter of a probability filter of the data subset, the filter parameter comprising at least one of a number of hash functions and a bit array length;

[0028] Wherein, in the case where the filtering parameters include the number of hash functions and the length of the bit array, the number of hash functions and the length of the bit array have the same change trend.

[0029] In a second aspect, the present application provides a data deduplication device, the device comprising:

[0030] A partitioning module, used for dividing a plurality of to-be-deduplicated data into at least one data subset; each data subset corresponds to an independent probability filter;

[0031] A mapping module, used for determining, for each data subset, a mapping value of each to-be-deduplicated data in the data subset according to a probability filter corresponding to the data subset;

[0032] A filtering module, used for determining data filtering results of each to-be-deduplicated data in the data subset according to the mapping values ​​of each to-be-deduplicated data in the data subset; the data filtering results are used to indicate whether the to-be-deduplicated data exists in the probability filter;

[0033] The deduplication module is used to perform deduplication processing on the data subset according to the data filtering results of each data to be deduplicated in the data subset.

[0034] In a third aspect, the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0035] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the above method when executed by a processor.

[0036] The above-mentioned data deduplication method, device, computer equipment and storage medium divide multiple data to be deduplicated into at least one data subset. For each data subset, according to the probability filter corresponding to the data subset, the mapping value of each data to be deduplicated in the data subset is determined. According to the mapping value of each data to be deduplicated in the data subset, the data filtering results of each data to be deduplicated in the data subset are respectively determined. According to the data filtering results of each data to be deduplicated in the data subset, the data subset is deduplicated. The data deduplication method provided in the embodiment of the present application divides the data to be deduplicated into at least one data subset, realizes a more refined distinction between similar data, and applies an independent probability filter to each data subset, that is, for each data subset, an independent probability filter is adaptively and characteristically designed to avoid using a unified probability filter to perform deduplication processing on different data with large differences, reduce the misjudgment of duplicate data by the probability filter, not only improves the deduplication efficiency, but also significantly improves the accuracy of data deduplication. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] FIG1 is a schematic diagram of a flow chart of a data deduplication method according to an embodiment;

[0038] FIG2 is a second flow chart of a data deduplication method in one embodiment;

[0039] FIG3 is a third flow chart of a data deduplication method in one embodiment;

[0040] FIG4 is a fourth flow chart of a data deduplication method in one embodiment;

[0041] FIG5 is a fifth flow chart of a method for deduplication of data in one embodiment;

[0042] FIG6 is a sixth flow chart of a data deduplication method in one embodiment;

[0043] FIG7 is a seventh flow chart of a data deduplication method in one embodiment;

[0044] FIG8 is a structural block diagram of a data deduplication device in one embodiment;

[0045] FIG. 9 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0047] In one embodiment, as shown in FIG1 , a data deduplication method is provided. This embodiment is illustrated by applying the method to a terminal. It is understandable that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps S102 to S108.

[0048] S102: Divide a plurality of to-be-deduplicated data into at least one data subset.

[0049] The data to be deduplicated can be any type of data. In the embodiment of the present application, the data to be deduplicated refers to data with deduplication requirements. For example, the data to be deduplicated is data collected by an Internet platform, data collected by a customer relationship management system (CMR), data collected by an application, etc. The specific data can be determined by the actual data deduplication scenario and is not limited here.

[0050] Each data subset includes at least one data to be deduplicated. Each data subset has a certain degree of similarity between the data to be deduplicated, or in other words, each data subset has a high or identical similarity of at least one data feature of the data to be deduplicated. Exemplarily, multiple data to be deduplicated can be divided into at least one data subset according to preset rules, wherein the preset rules are pre-set and can be set according to the actual scenario, and are not limited here.

[0051] Each data subset corresponds to an independent probability filter. The probability filters of different data subsets may be the same or different. In an embodiment of the present application, the probability filter can be characterized for the data subset. The probability filter is used to determine whether the data to be deduplicated in the data subset is repeated. The probability filter is a probabilistic data structure that can determine that a data to be deduplicated exists in the probability filter with very little memory consumption and time cost, thereby quickly searching and deduplicating the data. Exemplarily, the probability filter includes at least one of a Bloom filter, a counting Bloom filter, a cuckoo filter, a dynamic Bloom filter, and a quotient filter.

[0052] S104: For each data subset, according to the probability filter corresponding to the data subset, determine the mapping value of each data in the data subset.

[0053] The mapping value refers to the value obtained after calculating the data to be deduplicated according to a preset algorithm, wherein the preset algorithm may include a preset function, for example, a hash function. The mapping value of each data to be deduplicated is determined according to the corresponding probability filter. The length of the mapping value of each data to be deduplicated in the same data subset is the same. The length of the mapping value of each data to be deduplicated in different data subsets may be the same or different, which is specifically determined according to the probability filter corresponding to each data subset and is not limited here. The probability filter can be understood as mapping each data to be deduplicated in the data subset to a value with a preset fixed length, namely, the mapping value.

[0054] S106: Determine data filtering results for each of the to-be-deduplicated data in the data subset according to the mapping values ​​of each of the to-be-deduplicated data in the data subset.

[0055] There is a preset mapping relationship between the mapping value and the data filtering result. Exemplarily, the data filtering results of each data to be deduplicated in the data subset can be determined respectively according to the mapping value and mapping relationship of each data to be deduplicated in the data subset. The data filtering result is used to indicate whether the data to be deduplicated exists in the probability filter. The data filtering result includes that the data to be deduplicated exists in the probability filter, or that the data to be deduplicated does not exist in the probability filter.

[0056] S108: Deduplication processing is performed on the data subset according to the data filtering results of each to-be-deduplicated data in the data subset.

[0057] In the case where the data filter result of the data to be deduplicated is that the data to be deduplicated exists in the probability filter, it means that the data to be deduplicated is duplicate data in the data subset, and the data to be deduplicated needs to be deleted from the data subset. In the case where the data filtering result of the data to be deduplicated is that the data to be deduplicated does not exist in the probability filter, it means that the data to be deduplicated is not duplicate data in the data subset, and the data to be deduplicated does not need to be deleted from the data subset. Exemplarily, according to the data filtering results of each data to be deduplicated in the data subset, the duplicate data in the data subset is deleted, and a corresponding data set without duplicate data is obtained.

[0058] The above-mentioned data deduplication method divides multiple data to be deduplicated into at least one data subset. For each data subset, according to the probability filter corresponding to the data subset, the mapping value of each data to be deduplicated in the data subset is determined. According to the mapping value of each data to be deduplicated in the data subset, the data filtering results of each data to be deduplicated in the data subset are respectively determined. According to the data filtering results of each data to be deduplicated in the data subset, the data subset is deduplicated. The data deduplication method provided in the embodiment of the present application divides the data to be deduplicated into at least one data subset, realizes a more refined distinction between similar data, and applies an independent probability filter to each data subset, that is, for each data subset, an independent probability filter is adaptively and characteristically designed, so as to avoid using a unified probability filter to perform deduplication processing on different data with large differences, reduce the misjudgment of duplicate data by the probability filter, not only improve the deduplication efficiency, but also significantly improve the accuracy of data deduplication.

[0059] In one embodiment, as shown in FIG. 2 , step S102 , dividing a plurality of to-be-deduplicated data into at least one data subset, includes the following steps S202 to S206 .

[0060] S202: respectively obtain data features of each piece of data to be deduplicated.

[0061] The data features of the data to be deduplicated are used to describe the properties and characteristics of the data to be deduplicated. Exemplarily, the data features include at least one of a data source, a timestamp, a data type, and a data format. Among them, the data source refers to the origin of the data to be deduplicated, for example, the data source of the data to be deduplicated is server A. The timestamp refers to the time when the data to be deduplicated is generated and / or modified. The data type is used to define the type and range of the data to be deduplicated, for example, the data type of the data to be deduplicated is a floating point type. The data format refers to the specific representation of the data to be deduplicated during storage and / or transmission, for example, the data format of the data to be deduplicated is a binary format.

[0062] S204: Determine the local sensitive hash value of each to-be-deduplicated data based on the local sensitive hash function according to the data characteristics of each to-be-deduplicated data.

[0063] Exemplarily, the locality sensitive hash function includes at least one of MinHash, SimHash, LSH Forest, Euclidean LSH, Cosine LSH, and Multi-Probe LSH.

[0064] Taking the data characteristics of the data to be deduplicated including the data source as an example, after respectively obtaining the data source of each data to be deduplicated based on the above step S202, the local sensitive hash value of each data to be deduplicated is determined respectively according to the data source of each data to be deduplicated based on the local sensitive hash function.

[0065] It should be noted that in actual applications, a suitable locality-sensitive hash function can be selected according to specific scenarios. The above is only an exemplary description and should not be understood as a specific limitation on the locality-sensitive hash function.

[0066] S206: Divide each to-be-deduplicated data into at least one data subset according to the local sensitive hash value of each to-be-deduplicated data.

[0067] Exemplarily, the to-be-deduplicated data with the same or similar local sensitive hash values ​​are divided into the same data subset, so that each to-be-deduplicated data is divided into at least one data subset. Wherein, the local sensitive hash values ​​are similar means that the local sensitive hash values ​​meet the preset hash value range, and the hash value ranges corresponding to different data subsets are different.

[0068] The above-mentioned data deduplication method obtains data features of each data to be deduplicated respectively, determines the local sensitive hash value of each data to be deduplicated based on the local sensitive hash function according to the data features of each data to be deduplicated, and divides each data to be deduplicated into at least one data subset according to the local sensitive hash value of each data to be deduplicated. This embodiment introduces the local sensitive hash technology to divide the data to be deduplicated into at least one data subset according to the data features of the data to be deduplicated, thereby realizing mapping similar data to be deduplicated to the same or similar data subsets, that is, quickly finding similar items in a large-scale data set, thereby applying an independent probability filter to each data subset for deduplication processing, which not only maintains the similarity between the data to be deduplicated, but also realizes a more refined distinction between similar data, which helps to reduce misjudgment and significantly improves the accuracy of data deduplication while ensuring query efficiency.

[0069] In one embodiment, the probability filter includes a Bloom filter, and the mapping value includes a hash value. As shown in FIG3 , step S104, according to the probability filter corresponding to the data subset, determines the mapping value of each to-be-deduplicated data in the data subset, including the following steps S302 and S304.

[0070] S302: Determine a hash function corresponding to the data subset according to the Bloom filter corresponding to the data subset.

[0071] Each Bloom filter is provided with at least one hash function. The number of hash functions of different Bloom filters can be the same or different, which can be determined according to the data subset and data deduplication requirements, and is not limited here.

[0072] Exemplarily, the hash function includes at least one of Secure Hash Algorithm 256 (SHA-256), SHA-3, BLAKE2, Murmur Hash, City Hash, FNV-1a, and XX Hash. The types of hash functions of different Bloom filters can be the same or different, which can be determined according to the data subset and data deduplication requirements, and are not limited here.

[0073] S304: Determine the hash value of each to-be-deduplicated data in the data subset according to the hash function.

[0074] In the application, for each data to be deduplicated in each data subset, the hash value of each data to be deduplicated is determined according to the hash function of the corresponding probability filter, so that the data filtering results of each data to be deduplicated in the data subset can be determined according to the hash value of each data to be deduplicated, and then according to the data filtering results of each data to be deduplicated in the data subset, the data subset is deduplicated to achieve data deduplication.

[0075] The above-mentioned data deduplication method determines the hash function corresponding to the data subset according to the Bloom filter corresponding to the data subset, and determines the hash value of each data to be deduplicated in the data subset according to the hash function. This implementation uses a Bloom filter to perform hash processing on the data to be deduplicated, and can determine whether a data to be deduplicated exists in the Bloom filter with very small memory consumption and time cost, thereby quickly searching and deduplicating data, thereby improving data deduplication efficiency.

[0076] In one embodiment, the probability filter includes a bit array. A bit array is a data structure used to store and operate bits (0 or 1). Each element in the bit array is a bit, that is, 0 or 1. For each probability filter, the initial value of each bit of its bit array is 0. The bit array length, or the size of the bit array, is pre-set. The bit array length of each probability filter can be the same or different, which can be determined based on the data subset and data deduplication requirements, and is not limited here.

[0077] Based on the above, as shown in FIG. 4 , step S106 , according to the mapping value of each to-be-deduplicated data in the data subset, respectively determines the data filtering result of each to-be-deduplicated data in the data subset, including the following steps S402 and S404 .

[0078] S402: traverse the values ​​of the mapping values ​​of each to-be-deduplicated data in the data subset at the corresponding positions in the bit array respectively.

[0079] In the application, according to the value of each bit of the mapping value of each to-be-deduplicated data, the value of the corresponding position in the bit array is searched. For example, if the mapping value of the to-be-deduplicated data D1 is 256, the values ​​of the 2nd, 5th and 6th bits in the corresponding bit array are searched.

[0080] S404: For any data to be deduplicated in the data subset, if there is 0 among the numerical values ​​corresponding to the traversed data to be deduplicated, determine that the data filtering result of the data to be deduplicated is not present in the probability filter; if there is no 0 among the numerical values ​​corresponding to the traversed data to be deduplicated, determine that the data filtering result of the data to be deduplicated is present in the probability filter.

[0081] For any data to be deduplicated in the data subset, if there is 0 in each value corresponding to the data to be deduplicated obtained through traversal, it indicates that the data to be deduplicated must not exist in the probability filter. Therefore, the data filtering result of the data to be deduplicated is determined to be not existing in the probability filter. For example, if the values ​​of the 2nd, 5th and 6th bits in the bit array corresponding to the data to be deduplicated D1 are 101 respectively, the data filtering result of the data to be deduplicated D1 is determined to be not existing in the corresponding probability filter.

[0082] For any data to be deduplicated in the data subset, if there is no 0 in the values ​​corresponding to the data to be deduplicated obtained by traversal, it indicates that the data to be deduplicated may exist in the probability filter. Therefore, the data filtering result of the data to be deduplicated is determined to exist in the probability filter. For example, the values ​​of the 2nd, 5th and 6th bits in the bit array corresponding to the data to be deduplicated D1 are 111 respectively, then the data filtering result of the data to be deduplicated D1 is determined to exist in the corresponding probability filter.

[0083] In the application, for the data to be deduplicated whose filtering result is that the data to be deduplicated does not exist in the probability filter, the corresponding position of the data to be deduplicated in the bit array is set to 1, so that the data to be deduplicated is stored in the probability filter, and then used to determine whether there is data that is repeated with the data to be deduplicated in the future.

[0084] The above-mentioned data deduplication method traverses the numerical values ​​of the mapping values ​​of each to-be-deduplicated data in the data subset at the corresponding position in the bit array respectively. For any to-be-deduplicated data in the data subset, when there is 0 in the numerical values ​​corresponding to the traversed data to be deduplicated, it is determined that the data filtering result of the to-be-deduplicated data is not present in the probability filter; and when there is no 0 in the numerical values ​​corresponding to the traversed data to be deduplicated, it is determined that the data filtering result of the to-be-deduplicated data is present in the probability filter. That is, by querying the data using the mapping value of the to-be-deduplicated data, compared with directly using the original data for querying, since the mapping value reduces the data amount compared to the original data, the data query complexity is reduced, the data query rate is increased, and the efficiency of deduplicating the data according to the numerical value obtained by the query is improved.

[0085] In one embodiment, as shown in FIG. 5 , step S108 , performing deduplication processing on the data subset according to the data filtering results of each to-be-deduplicated data in the data subset, includes the following steps S502 and S504 .

[0086] S502: For any to-be-deduplicated data in the data subset, if the data filtering result of the to-be-deduplicated data is that the to-be-deduplicated data exists in the probability filter, the to-be-deduplicated data is stored in a duplicate database corresponding to the data subset.

[0087] S504: When the data filtering result of the to-be-deduplicated data is that the to-be-deduplicated data does not exist in the probability filter, the to-be-deduplicated data is stored in a result database corresponding to the data subset.

[0088] Each data subset corresponds to a duplicate database and a result database. If the data filtering result is present in the probability filter, it means that the data to be deduplicated is duplicate data in the data subset. Therefore, the data to be deduplicated is stored in the duplicate database corresponding to the data subset. If the data filtering result is not present in the probability filter, it means that the data to be deduplicated is not duplicate data in the data subset. Therefore, the data to be deduplicated is stored in the result database corresponding to the data subset. Among them, the data in the duplicate database is duplicated with the data in the result database, that is, the data in the duplicate database is a set of duplicate data in the data subset, and the result database is a set of data after removing duplicate data from the data subset.

[0089] The above-mentioned data deduplication method, for any to-be-deduplicated data in the data subset, when the data filtering result of the to-be-deduplicated data is that it exists in the probability filter, the to-be-deduplicated data is stored in the duplicate database corresponding to the data subset; when the data filtering result of the to-be-deduplicated data is that it does not exist in the probability filter, the to-be-deduplicated data is stored in the result database corresponding to the data subset, thereby realizing deduplication processing for duplicate data in the same data subset, so that the deduplication result database can be applied, avoiding resource consumption caused by duplicate data, helping to improve data processing efficiency and reduce resource consumption.

[0090] In one embodiment, as shown in FIG6 , the data deduplication method further includes the following steps S602 and S604 .

[0091] S602: Determine the false alarm rate of the probability filter according to the result database and the duplicate database corresponding to the data subset.

[0092] Exemplarily, according to the result database and the duplicate database corresponding to the data subset, the false positive data in the duplicate database that does not exist in the result database is determined, and the false positive rate of the probability filter is determined according to the false positive data. For example, the result database includes {1, 2, 3, 4, 5, 6, 7, 8}, the duplicate database includes {0, 1}, wherein the false positive data includes 0, then the false positive rate is 10%.

[0093] S604: When the false alarm rate exceeds the allowable error range of the probability filter, the probability filter is adjusted, and the data filtering result of subsequent data is determined using the adjusted probability filter until the false alarm rate satisfies the allowable error range.

[0094] The allowable error range is pre-set and can be set according to the data deduplication accuracy requirements, and is not limited here. The allowable error range of each probability filter can be the same or different. If the false alarm rate exceeds the allowable error range of the probability filter, it means that the deduplication accuracy of the probability filter does not meet the data deduplication accuracy requirements. For this reason, the probability filter needs to be optimized and adjusted. In the case where the false alarm rate exceeds the allowable error range of the probability filter, the result database and the duplicate database can be updated according to the false alarm data determined in the aforementioned step S602 to obtain accurate result data and duplicate database.

[0095] In the application, the adjusted probability filter is used to process the subsequently collected data to be deduplicated that belongs to the data subset corresponding to the probability filter until the false alarm rate meets the allowable error range. Exemplarily, for the subsequently collected data to be deduplicated (referred to as target data), it is determined that the data subset of the target data is the data subset of the adjusted probability filter, then the mapping value of the target data is determined according to the adjusted probability filter, and whether the target data exists in the data filtering result of the adjusted probability filter is determined according to the mapping value of the target data, and the target data is stored in an updated duplicate database or an updated result database according to the data filtering result, and the false alarm rate of the probability filter is re-determined according to the duplicate database and the result database. If the false alarm rate does not exceed the allowable error range of the probability filter, the result database can be directly applied; if the false alarm rate exceeds the allowable error range of the probability filter, the probability filter is continuously adjusted, and the data filtering results of the subsequent data are determined using the adjusted probability filter until the false alarm rate meets the allowable error range.

[0096] It should be noted that, in the application, each data to be deduplicated in the data subset can be re-processed according to the adjusted probability filter, and the false alarm rate of the adjusted probability filter can be determined according to the re-determined result database and duplicate database. If the false alarm rate does not exceed the allowable error range of the probability filter, the result database can be directly applied; if the false alarm rate exceeds the allowable error range of the probability filter, continue to adjust the probability filter, and use the adjusted probability filter to process the data of the data subset again until the false alarm rate meets the allowable error range.

[0097] It should be noted that the above is only an exemplary description. In actual applications, a suitable method can be selected to optimize and adjust the probability filter according to specific scenarios, and no excessive limitations are made here.

[0098] The above-mentioned data deduplication method determines the false alarm rate of the probability filter according to the result database and the duplicate database corresponding to the data subset. When the false alarm rate exceeds the allowable error range of the probability filter, the probability filter is adjusted, and the adjusted probability filter is used to determine the data filtering results of subsequent data until the false alarm rate meets the allowable error range. This implementation enters into a feedback learning mechanism, records and analyzes the false alarm rate of the probability filter, and optimizes and adjusts the probability filter using the false alarm rate and the allowable error range, thereby reducing the false alarm rate of the probability filter, improving the data deduplication accuracy of the probability filter, and enabling the system to continuously optimize its own performance during continuous operation and reduce the long-term misjudgment rate.

[0099] In one embodiment, step S604, adjusting the probability filter of the data subset, may include: adjusting the filtering parameters of the probability filter of the data subset, wherein the filtering parameters include at least one of the number of hash functions and the length of the bit array.

[0100] Exemplarily, when the false alarm rate exceeds the allowable error range of the probability filter, the number of hash functions is increased or decreased, and / or the length of the bit array is increased or decreased. It is understandable that when the number of hash functions changes, the hash functions can be increased or decreased accordingly according to the change in the number of hash functions. The increased or decreased hash functions can be specifically determined based on factors such as data subsets, false alarm rates, and allowable false alarm ranges, which are not limited here.

[0101] Wherein, when the filtering parameters include the number of hash functions and the length of the bit array, the change trends of the number of hash functions and the length of the bit array are the same. Exemplarily, when the false alarm rate exceeds the allowable error range of the probability filter, the number of hash functions and the length of the bit array are increased respectively. Another exemplary embodiment, when the false alarm rate exceeds the allowable error range of the probability filter, the number of hash functions and the length of the bit array are reduced respectively.

[0102] The above-mentioned data deduplication method, when the false alarm rate exceeds the allowable error range of the probability filter, synchronously adjusts the number of hash functions and the bit array length of the probability filter so that the false alarm rate of the probability filter does not exceed the allowable error range, thereby reducing the false alarm rate and improving the data deduplication accuracy.

[0103] In one embodiment, as shown in FIG. 7 , a data deduplication method is provided, and the data deduplication method includes the following steps S702 to S724 .

[0104] S702: respectively obtain data features of a plurality of data to be deduplicated, wherein the data features include data sources or timestamps.

[0105] S704: Determine the local sensitive hash value of each to-be-deduplicated data based on the local sensitive hash function according to the data characteristics of each to-be-deduplicated data.

[0106] S706: Divide each to-be-deduplicated data into at least one data subset according to the local sensitive hash value of each to-be-deduplicated data.

[0107] S708: Initialize the number of hash functions, the size of the bit array, and the allowable error range of the Bloom filter of each data subset.

[0108] S710: Determine the hash value of each to-be-deduplicated data in the data subset according to the Bloom filter corresponding to each data subset.

[0109] S712: According to the hash value of each to-be-deduplicated data in each data subset, determine whether each to-be-deduplicated data exists in the corresponding Bloom filter. If not, execute step S714; if yes, execute step S716.

[0110] S714: Storing the data to be deduplicated in a result database.

[0111] S716: Storing the data to be deduplicated in the duplication database.

[0112] S718: Determine the false alarm rate of the Bloom filter according to the result database and the duplicate database.

[0113] S720: Determine whether the false alarm rate of the Bloom filter exceeds the allowable error range. If yes, execute step S722; if no, execute step S724.

[0114] S722: Dynamically increase or decrease the number of Bloom filter hash functions and / or the bit array length in small increments, and continue data collection until the allowable error range is met.

[0115] S724: Record the result database, duplicate database, number of hash functions of the Bloom filter, bit array length, false alarm rate, and allowable false alarm range. After multiple data collections, when the false alarm rate is stable, statistical analysis of the error can be gradually reduced.

[0116] The above-mentioned data deduplication method introduces local sensitive hashing technology to divide the data to be deduplicated into data subsets according to certain rules (such as data source, timestamp, etc.), realizes more refined distinction of similar data, and applies independently designed Bloom filters to each data subset, reducing misjudgment, and significantly improving the accuracy of data deduplication while ensuring query efficiency. In addition, a feedback learning mechanism is introduced to record and analyze the misjudgment rate, and use the misjudgment rate to optimize and adjust the selection of the correct hash function and the bit array layout, so that the system can continuously optimize its own performance during continuous operation, reduce the long-term misjudgment rate, and improve the accuracy and efficiency of data deduplication.

[0117] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0118] Based on the same inventive concept, the embodiment of the present application also provides a data deduplication device for implementing the data deduplication method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more data deduplication device embodiments provided below can refer to the limitations of the data deduplication method above, and will not be repeated here.

[0119] In one embodiment, as shown in FIG. 8 , a data deduplication device 800 is provided, including: a partitioning module 801 , a mapping module 802 , a filtering module 803 and a deduplication module 804 .

[0120] Among them, the partitioning module 801 is used to divide multiple data to be deduplicated into at least one data subset; each data subset corresponds to an independent probability filter. The mapping module 802 is used to determine the mapping value of each data to be deduplicated in the data subset according to the probability filter corresponding to the data subset. The filtering module 803 is used to determine the data filtering results of each data to be deduplicated in the data subset according to the mapping value of each data to be deduplicated in the data subset; the data filtering result is used to indicate whether the data to be deduplicated exists in the probability filter. The deduplication module 804 is used to perform deduplication processing on the data subset according to the data filtering results of each data to be deduplicated in the data subset.

[0121] In one embodiment, the partitioning module 801 is also used to respectively obtain data characteristics of each data to be deduplicated, determine the local sensitive hash value of each data to be deduplicated based on the local sensitive hash function according to the data characteristics of each data to be deduplicated, and divide each data to be deduplicated into at least one data subset according to the local sensitive hash value of each data to be deduplicated.

[0122] In one embodiment, the mapping module 802 is further used to determine a hash function corresponding to the data subset according to the Bloom filter corresponding to the data subset, and determine the hash value of each to-be-deduplicated data in the data subset according to the hash function.

[0123] In one embodiment, the filtering module 803 is also used to traverse the numerical values ​​of the mapping values ​​of each to-be-deduplicated data in the data subset at the corresponding position in the bit array respectively. For any to-be-deduplicated data in the data subset, when there is 0 in the numerical values ​​corresponding to the traversed data to be deduplicated, it is determined that the data filtering result of the to-be-deduplicated data is not present in the probability filter; and when there is no 0 in the numerical values ​​corresponding to the traversed data to be deduplicated, it is determined that the data filtering result of the to-be-deduplicated data is present in the probability filter.

[0124] In one embodiment, the deduplication module 804 is also used to store any to-be-deduplicated data in the data subset in a duplication database corresponding to the data subset if the data filtering result of the to-be-deduplicated data is that the data exists in the probability filter, and to store the to-be-deduplicated data in a result database corresponding to the data subset if the data filtering result of the to-be-deduplicated data is that the data does not exist in the probability filter.

[0125] In one embodiment, the data deduplication device 800 also includes an adjustment module, which is used to determine the false alarm rate of the probability filter based on the result database and the duplicate database corresponding to the data subset, and to adjust the probability filter when the false alarm rate exceeds the allowable error range of the probability filter, and use the adjusted probability filter to determine the data filtering results of subsequent data to be deduplicated until the false alarm rate meets the allowable error range.

[0126] In one embodiment, the adjustment module is further used to adjust the filter parameters of the probability filter of the data subset, and the filter parameters include at least one of the number of hash functions and the length of the bit array. Wherein, when the filter parameters include the number of hash functions and the length of the bit array, the change trends of the number of hash functions and the length of the bit array are the same.

[0127] The above-mentioned data deduplication device 800 divides multiple to-be-deduplicated data into at least one data subset through a partitioning module 801, and determines the mapping value of each to-be-deduplicated data in the data subset according to the probability filter corresponding to the data subset through a mapping module 802 for each data subset, and determines the data filtering results of each to-be-deduplicated data in the data subset according to the mapping value of each to-be-deduplicated data in the data subset through a filtering module 803, and performs deduplication processing on the data subset according to the data filtering results of each to-be-deduplicated data in the data subset through a deduplication module 804. The data deduplication device 800 provided in the embodiment of the present application divides the to-be-deduplicated data into at least one data subset, realizes a more refined distinction between similar data, and applies an independent probability filter to each data subset, that is, for each data subset, an independent probability filter is adaptively and characteristically designed, so as to avoid using a unified probability filter to perform deduplication processing on different data with large differences, reduce the misjudgment of duplicate data by the probability filter, and not only improve the deduplication efficiency, but also significantly improve the accuracy of data deduplication.

[0128] Each module in the above-mentioned data deduplication device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0129] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as shown in FIG9 . The computer device includes a processor, a memory, and a network interface connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data to be deduplicated, a result database, a deduplication database, and other data. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a data deduplication method is implemented.

[0130] Those skilled in the art will understand that the structure shown in FIG. 9 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0131] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps of the aforementioned data deduplication method when executing the computer program.

[0132] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the aforementioned data deduplication method are implemented.

[0133] In one embodiment, a computer program product is provided, including a computer program, which implements the steps of the aforementioned data deduplication method when executed by a processor.

[0134] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0135] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data deduplication logic device based on quantum computing, etc., but are not limited to this.

[0136] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0137] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A data deduplication method, characterized in that: The method comprises: Divide the multiple data to be deduplicated into at least one data subset; each data subset corresponds to an independent probability filter; For each data subset, determining a mapping value of each to-be-deduplicated data in the data subset according to a probability filter corresponding to the data subset; According to the mapping values ​​of each to-be-deduplicated data in the data subset, respectively correspondingly determining the data filtering results of each to-be-deduplicated data in the data subset; the data filtering results are used to indicate whether the to-be-deduplicated data exists in the probability filter; Deduplication processing is performed on the data subset according to the data filtering results of each to-be-deduplicated data in the data subset.

2. The method according to claim 1, characterized in that: The step of dividing the plurality of to-be-deduplicated data into at least one data subset comprises: Respectively obtain data features of each of the to-be-deduplicated data; Determine the local sensitive hash value of each of the to-be-deduplicated data based on the local sensitive hash function according to the data characteristics of each of the to-be-deduplicated data; According to the local sensitive hash value of each of the to-be-deduplicated data, each of the to-be-deduplicated data is divided into at least one data subset.

3. The method according to claim 1 or 2, characterized in that: The probability filter includes a Bloom filter, and the mapping value includes a hash value, wherein determining the mapping value of each to-be-deduplicated data in the data subset according to the probability filter corresponding to the data subset includes: Determining a hash function corresponding to the data subset according to the Bloom filter corresponding to the data subset; The hash value of each to-be-deduplicated data in the data subset is determined respectively according to the hash function.

4. The method according to claim 1 or 2, characterized in that: The probability filter includes a bit array, and the data filtering results of each to-be-deduplicated data in the data subset are respectively determined according to the mapping values ​​of each to-be-deduplicated data in the data subset, including: Traversing the values ​​of the mapping values ​​of each to-be-deduplicated data in the data subset at the corresponding positions in the bit array respectively; For any one of the data to be deduplicated in the data subset, if there is 0 among the traversed numerical values ​​corresponding to the data to be deduplicated, it is determined that the data filtering result of the data to be deduplicated is not present in the probability filter; if there is no 0 among the traversed numerical values ​​corresponding to the data to be deduplicated, it is determined that the data filtering result of the data to be deduplicated is present in the probability filter.

5. The method according to claim 1 or 2, characterized in that: The performing deduplication processing on the data subset according to the data filtering result of each to-be-deduplicated data in the data subset comprises: For any one of the to-be-deduplicated data in the data subset, if the data filtering result of the to-be-deduplicated data is that the to-be-deduplicated data exists in the probability filter, storing the to-be-deduplicated data in the duplicate database corresponding to the data subset; When the data filtering result of the to-be-deduplicated data is that the to-be-deduplicated data does not exist in the probability filter, the to-be-deduplicated data is stored in a result database corresponding to the data subset.

6. The method according to claim 5, characterized in that The method further comprises: Determining the false alarm rate of the probability filter according to the result database and the duplicate database corresponding to the data subset; When the false alarm rate exceeds the allowable error range of the probability filter, the probability filter is adjusted, and the adjusted probability filter is used to determine the data filtering results of subsequent data to be deduplicated until the false alarm rate meets the allowable error range.

7. The method according to claim 6, characterized in that The adjusting the probability filter of the data subset comprises: adjusting a filter parameter of a probability filter of the data subset, the filter parameter comprising at least one of a number of hash functions and a bit array length; Wherein, in the case where the filtering parameters include the number of hash functions and the length of the bit array, the number of hash functions and the length of the bit array have the same change trend.

8. A data deduplication device, characterized in that: The device comprises: A partitioning module, used for dividing a plurality of to-be-deduplicated data into at least one data subset; each data subset corresponds to an independent probability filter; A mapping module, used for determining, for each data subset, a mapping value of each to-be-deduplicated data in the data subset according to a probability filter corresponding to the data subset; A filtering module, used for determining data filtering results of each to-be-deduplicated data in the data subset according to the mapping values ​​of each to-be-deduplicated data in the data subset; the data filtering results are used to indicate whether the to-be-deduplicated data exists in the probability filter; The deduplication module is used to perform deduplication processing on the data subset according to the data filtering results of each data to be deduplicated in the data subset.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images