Duplicate data detection method, device, storage medium and computer program product

Abstract

The present application discloses a duplicate data detection method, device, storage medium and computer program product, which relate to the field of data processing technology. A preset bit array is constructed by using multiple preset functions and stored data blocks as the basis for data duplication detection of data blocks to be detected. The preset bit array is used to preliminarily screen out possible duplicate data blocks to be detected, and then a preset hash table is used to perform further refined detection on the possible duplicate data blocks to be detected, thereby realizing hierarchical detection of data. Compared with the hash table lookup traversal and comparison of all data blocks to be detected, the detection efficiency is effectively improved while ensuring the accuracy of detection.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a duplicate data detection method, device, storage medium, and computer program product. Background Art

[0002] Removing duplicate data is a common step in data processing. Duplicate data may lead to problems such as model training deviation and resource waste.

[0003] Generally speaking, duplicate data detection requires establishing a hash table for the existing database to record the hash value of each data item. When new data arrives, the hash value of the new data is calculated and traversed in the hash table. If the same hash value is found, it means that the new data is a duplicate of the data already in the database.

[0004] However, since the maintenance and traversal of the hash table require a large amount of system resources, this data duplication detection method often suffers from problems such as low efficiency and excessive memory usage when faced with large-scale data. Summary of the Invention

[0005] The present application provides a duplicate data detection method, device, storage medium and computer program product to at least solve the problems of low efficiency and excessive memory usage in duplicate data detection in related technologies.

[0006] This application provides a method for detecting duplicate data, including:

[0007] Performing hash calculations on hash values ​​of at least one stored data block based on multiple preset hash functions to determine addresses of multiple first target element bits corresponding to the at least one stored data block;

[0008] Setting the plurality of first target element positions in the preset bit array to non-null values;

[0009] Performing hash calculations on hash values ​​of the data blocks to be detected based on the multiple preset hash functions, and determining addresses of multiple second target element bits corresponding to the data blocks to be detected;

[0010] If there is no null value for the plurality of second target element bits in the preset bit array, determining whether the data block to be detected is a duplicate data block according to a preset hash table.

[0011] The present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any of the above-mentioned duplicate data detection methods when executing the computer program.

[0012] The present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned duplicate data detection methods are implemented.

[0013] The present application also provides a computer program product, including a computer program, which implements the steps of any of the above-mentioned duplicate data detection methods when executed by a processor.

[0014] This application constructs a preset bit array through multiple preset functions and stored data blocks as the basis for data duplication detection of the data blocks to be detected. The preset bit array is used to preliminarily screen out possible duplicate data blocks to be detected, and then the possible duplicate data blocks to be detected are further refined through the preset hash table, thereby realizing hierarchical detection of data. Compared with the hash table traversal and comparison of all data blocks to be detected, it effectively improves the detection efficiency while ensuring the accuracy of the detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0016] Figure 1 A flowchart of a duplicate data detection method provided in an embodiment of the present application;

[0017] Figure 2 A schematic diagram of the structure of a duplicate data detection device provided in an embodiment of the present application;

[0018] Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0019] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0020] It should be noted that, in the description of this application, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. The terms "first," "second," etc., in this application are used to distinguish similar objects, and are not used to describe a particular order or sequence.

[0021] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0022] The embodiments of the present application provide a duplicate data detection method, and the method is described in detail in conjunction with the execution process of the duplicate data detection method.

[0023] Figure 1 This is a flowchart of a duplicate data detection method provided in an embodiment of the present application. This method can be applied to any one of multiple computing nodes, where each node can be any electronic device with data processing capabilities, such as a server, smartphone, PDA, tablet computer, desktop computer, laptop computer, all-in-one computer, etc. It is understood that the duplicate data detection method provided in an embodiment of the present application can also be applied in other scenarios.

[0024] Below Figure 1 The following are the steps to detect duplicate data:

[0025] S101 . Perform hash calculations on hash values ​​of at least one stored data block based on multiple preset hash functions to determine addresses of multiple first target element bits corresponding to the at least one stored data block.

[0026] The multiple preset hash functions are a group of independent hash functions selected to construct the preset bit array, wherein each preset hash function has a low correlation, thereby being able to effectively disperse the spatial distribution of calculation results of the multiple preset hash functions.

[0027] Optionally, the number k of preset hash functions is determined by the length w of the preset bit array and the number v of elements expected to be inserted:

[0028]

[0029] The number of elements expected to be inserted refers to the sum of the number of stored data blocks and the number of non-repeated data blocks to be detected.

[0030] The stored data blocks are data blocks added to the storage space at a historical moment. When the electronic device obtains the stored data blocks, it performs a hash calculation on them to obtain the hash value of the stored data blocks. That is, each stored data block has its own corresponding hash value.

[0031] Optionally, the hash function used to calculate the hash value of the stored data block may be a hash function different from the multiple preset hash functions.

[0032] The first target element bit refers to the mapping position in the preset bit array of the calculation results obtained by performing hash calculations on the hash value of at least one stored data block by multiple preset hash functions. Based on the address of the first target element bit, the corresponding first target element bit can be determined in the preset bit array.

[0033] Specifically, for each stored data block, a hash calculation is performed on the hash value of the stored data block based on multiple preset hash functions to obtain the address of the first target element bit corresponding to each stored data block. The number of addresses of the first target element bit corresponding to each stored data block is consistent with the number of the multiple preset hash functions, that is, for a single stored data block, the address of the first target element bit corresponds to the preset hash function on a one-to-one basis.

[0034] Furthermore, the set of first target element bits corresponding to each stored data block is the address of multiple first target element bits corresponding to at least one stored data block.

[0035] S102: Set the positions of multiple first target elements in the preset bit array to non-null values.

[0036] The preset bit array includes multiple element bits. In an initial state, the element value in each element bit is a null value.

[0037] Optionally, the initial capacity of the preset bit array is set to 1.2 times the estimated data volume to avoid frequent capacity expansion. The error rate of the preset bit array is adjusted based on memory resources and judgment accuracy requirements. The error rate of the preset bit array refers to the probability of determining a non-duplicate data block as a possible duplicate data block.

[0038] Based on the addresses of the plurality of first target element bits, the corresponding plurality of first target element bits are determined in the preset bit array, and the plurality of first target element bits are set to non-null values ​​to obtain the preset bit array corresponding to the currently at least one stored data block.

[0039] Specifically, the element value of the corresponding first target element bit is modified according to the address of the first target element bit corresponding to each stored data block. When a first target element bit has been set to a non-empty value, it does not need to be modified again.

[0040] Optionally, the preset bit array is a binary array with an initial state of 0. For example, if the addresses of multiple first target element bits corresponding to a stored data block are h1, h2, and h3, the element values ​​at the three positions h1, h2, and h3 in the preset bit array are set to 1.

[0041] Among them, the preset bit array is a Bloom filter.

[0042] S103 , performing hash calculations on hash values ​​of the data block to be detected based on multiple preset hash functions, and determining addresses of multiple second target element bits corresponding to the data block to be detected.

[0043] The data block to be detected refers to a newly input new data unit that needs to be checked for data duplication.

[0044] When the data block to be detected is obtained, the electronic device uses the same hash function as that used to calculate the hash value of the stored data block to calculate the hash value of the data block to be detected.

[0045] After obtaining the hash value corresponding to the data block to be detected, the hash value corresponding to the data block to be detected is hashed based on the multiple preset hash functions to obtain multiple addresses of second target element bits, wherein the addresses of the second target element bits correspond one-to-one to the preset hash functions.

[0046] The multiple preset functions used in this step are the same as the multiple preset functions used in the process of constructing the preset bit array according to the stored data block.

[0047] S104: If there is no null value in the plurality of second target element bits in the preset bit array, determine whether the data block to be detected is a duplicate data block according to a preset hash table.

[0048] According to the addresses of the plurality of second target element bits, a plurality of second target element bits in a preset bit array are determined, and it is detected whether the element value in each second target element bit is a null value.

[0049] If the element value in each second target element bit is not a null value, it means that there may be multiple first target element bits corresponding to the stored data block that are the same as the multiple target element bits corresponding to the data block to be detected. Since the multiple preset hash functions used to determine the first target element bit or the second element bit are the same, the hash value of the data block to be detected may be the same as the hash value of a stored data block, and the data block to be detected may be repeated with the stored data block. At this time, it is necessary to combine the preset hash table to further determine whether the data block to be detected is a duplicate data block.

[0050] If there are null values ​​in the plurality of second target element bits, it is determined that the data block to be detected is not a duplicate data block.

[0051] Specifically, if there is at least one second target element bit whose element value is not a null value, it means that the multiple second target element bits corresponding to the data block to be detected are not exactly the same as the first target element bits corresponding to any stored data block, and the hash value of the corresponding data block to be detected is different from the hash value of any stored data block. It can be determined that the data block to be detected is not repeated with any stored data block.

[0052] When a plurality of second target element bits contain null values, or when the data block to be detected is determined not to be a duplicate data block according to a preset hash table, the preset bit array is updated according to the plurality of second target element bits; and the preset hash table is updated according to the data block to be detected and the hash value of the data block to be detected. The preset hash table records the stored data blocks and the hash value corresponding to each stored data block.

[0053] Specifically, the element values ​​of the plurality of second target element positions are set to non-null values; and the data block to be detected and the hash value of the data block to be detected are written into a preset hash table.

[0054] The embodiment of the present application constructs a preset bit array through multiple preset functions and stored data blocks as the basis for data duplication detection of the data blocks to be detected. The preset bit array is used to preliminarily screen out possible duplicate data blocks to be detected, and then the possible duplicate data blocks to be detected are further refined through the preset hash table, thereby realizing hierarchical detection of data. Compared with the hash table traversal and comparison of all data blocks to be detected, the detection efficiency is effectively improved while ensuring the accuracy of the detection.

[0055] In some embodiments, hash values ​​of the data blocks to be detected are respectively hashed based on multiple preset hash functions, and before the addresses of multiple second target element bits corresponding to the data blocks to be detected are determined, the method also includes: dividing the data to be detected into blocks according to the type of the data to be detected to obtain multiple data blocks to be detected; and calculating the hash values ​​of the data blocks to be detected.

[0056] Blocking is the process of dividing the data to be tested into discrete processing units. Different data block strategies are selected for different data types.

[0057] Among them, the data to be detected is divided into blocks according to the type of the data to be detected to obtain multiple data blocks to be detected, including: if the data to be detected is unstructured data, a first sliding window is moved in the data to be detected with a first preset step size, and the data characteristic value in the first sliding window is calculated in real time; when the data characteristic value meets the preset condition, the block position is determined according to the position of the first sliding window to obtain at least one block position; according to the block position, the data to be detected is divided into multiple data blocks to be detected.

[0058] When the data to be detected is unstructured data, such as web page content data, the sliding window blocking mechanism is triggered.

[0059] The data to be detected is taken as the processing object, a first sliding window is adopted, the window is slid according to a preset step size, and the data feature value in each window is calculated, such as the hash value of the data in the first sliding window.

[0060] When the data characteristic value meets the preset condition, the block position is determined according to the current position of the first sliding window.

[0061] For example, the starting position of the current first sliding window is used as the block position; or the ending position of the current first sliding window is used as the block position; or the ending position of the first sliding window before the current window sliding operation is determined according to the ending position of the current first sliding window and the step size of the window sliding, thereby determining the block position.

[0062] Furthermore, the data to be detected before the block position is divided into a data block to be detected.

[0063] Optionally, if the difference between the data feature value corresponding to the current first sliding window and the data feature value in the previous window is greater than a preset threshold, it means that the content of the data in the current first sliding window has changed significantly compared with the data in the previous window, and it is determined to be a semantic boundary, then the starting position of the current first sliding window is used as the block position.

[0064] Accordingly, calculating the hash value of the data block to be detected includes: determining the data hash value corresponding to the first sliding window at the initial position in the data block to be detected as the hash value of the data block to be detected.

[0065] Specifically, for the string s[0..m-1] of the first sliding window at the initial position, calculate the hash value:

[0066]

[0067] Where d is the base of the hash function; q is the modulus of the hash function; s[i] represents the character encoding in the string, usually ASCII code.

[0068] Alternatively, a hash calculation may be performed on the data block to be detected that is segmented based on the first sliding window to obtain a hash value of the data block to be detected.

[0069] Alternatively, if the data to be detected is structured data, the data to be detected is divided into blocks according to a preset number of bytes to obtain a plurality of data blocks to be detected, and the number of bytes of each data block to be detected is the preset number of bytes.

[0070] When the data to be detected is structured data, such as a log file, the data to be detected is divided into a plurality of data blocks to be detected with equal byte numbers according to a preset number of bytes.

[0071] Hash calculations are performed on the multiple data blocks to be detected obtained by segmentation to obtain a hash value corresponding to each data block to be detected.

[0072] This embodiment adaptively determines the data segmentation strategy based on the data type of the data to be detected. It adopts fixed byte number segmentation for structured data and dynamic segmentation for unstructured data, which conforms to the data characteristics of the corresponding data type and improves the flexibility and accuracy of data segmentation.

[0073] In some embodiments, determining whether the data block to be detected is a duplicate data block based on a preset hash table includes: searching the preset hash table for a target data block having the same hash value as the data block to be detected; comparing the data block to be detected and the target data block to obtain a comparison result; and determining whether the data block to be detected is a duplicate data block based on the comparison result.

[0074] If there are no null values ​​for the multiple second target element bits in the preset bit array, it is necessary to perform further refined comparison on the data block to be detected based on the stored data blocks recorded in the preset hash table and the hash values ​​of the stored data blocks.

[0075] First, the hash value of the data block to be checked is searched in a preset hash table to identify a target data block with the same hash value as the data block to be checked. The target data block is then carefully compared with the target data block. The target data block is a data block that may be a duplicate of the data block to be checked.

[0076] Among them, the data block to be detected and the target data block are compared to obtain a comparison result, including: moving the second sliding window in the data block to be detected and the target data block, and calculating in real time the feature value to be detected and the target feature value corresponding to each window movement operation, the feature value to be detected is the data feature value within the second sliding window in the data block to be detected, and the target feature value is the data feature value within the second sliding window in the target data block; comparing the feature value to be detected and the target feature value corresponding to the same window movement operation to obtain a comparison result.

[0077] The second sliding window has a preset window size and a preset moving step size. The data block to be detected and the target data block are processed in parallel based on the second sliding window, and a feature value to be detected of the data in the second sliding window of the data block to be detected and a target feature value of the data in the second sliding window of the target data block corresponding to each window sliding operation are calculated.

[0078] If the feature value to be detected corresponding to at least one window moving operation is different from the target feature value, it is determined that the data block to be detected and the target data block are not repeated; otherwise, it is determined that the data block to be detected and the target data block are repeated.

[0079] The detected eigenvalues ​​corresponding to each window movement operation are compared with the target eigenvalues. If the detected eigenvalues ​​are identical to the target eigenvalues, this indicates that the data in the second sliding window of the current data block to be detected is identical to the data in the second sliding window of the target data block. The next window movement operation is then performed, and the detected eigenvalues ​​corresponding to the next window movement operation are compared with the target eigenvalues. If the data block to be detected and the target data block are processed in parallel based on the second sliding window, and the detected eigenvalues ​​corresponding to each window sliding operation are identical to the target eigenvalues, this indicates that the data in the second sliding window of the data block to be detected is identical to the data in the second sliding window of the target data block, i.e., the data block to be detected and the target data block are duplicated.

[0080] When the characteristic value to be detected is different from the target characteristic value, it means that the data in the second sliding window of the current data block to be detected is different from the data in the second sliding window of the target data block, that is, the data block to be detected and the target data block are not repeated.

[0081] When the detected feature value corresponding to a window moving operation is different from the target feature value, it can be determined that the data block to be detected and the target data block are not repeated. At this time, the window moving operation is stopped and the preset bit array and the preset hash table are updated according to the data block to be detected.

[0082] The disclosed embodiment processes the detection data block and the target data block in parallel through a second sliding window, and determines whether the data block to be detected and the target data block are duplicates based on the comparison result of the characteristic value to be detected and the target characteristic value. Once the characteristic value to be detected and the target characteristic value corresponding to a window movement operation are different, it can be determined that the data block to be detected and the target data block are not duplicates. There is no need to perform full-byte traversal comparison on the data block to be detected and the target data block, which reduces the amount of data calculation and further improves the efficiency of duplicate data detection.

[0083] In some embodiments, a second sliding window is moved between a data block to be detected and a target data block, and a feature value to be detected and a target feature value corresponding to each window movement operation are calculated in real time, including: moving the second sliding window in a preset data block, determining the removed characters and newly added characters in the second sliding window during a single movement, where the preset data block is the data block to be detected or the target data block; updating the first window hash value according to the hash contribution of the removed characters and the newly added characters to obtain a second window hash value, where the first window hash value is the hash value of the data in the second sliding window before the single movement operation, and the second window hash value is the hash value of the data in the second sliding window after the single movement operation.

[0084] The removed characters refer to characters that disappear from the second sliding window due to the movement operation of the second sliding window, and the newly added characters refer to characters that are newly added to the second sliding window due to the movement operation of the second sliding window.

[0085] The first window hash value is the data feature value in the second sliding window before the current window movement operation, and the second window hash value is the data feature value in the current second sliding window. The second window hash value is obtained by updating the first window hash value, rather than directly hashing the data in the current second sliding window.

[0086] The hash value of the first window is updated, including subtracting the hash contribution of the removed characters from the hash value of the first window and adding the hash contribution of the newly added characters.

[0087] Specifically, the base value is multiplied by the removed character to obtain the hash contribution of the removed character, where the base value is calculated based on the cardinality, modulus, and window size of the second sliding window of the hash function; the difference between the hash value of the first window and the hash contribution of the removed character is determined, and the product of the difference and the cardinality is calculated; the product and the sum of the newly added characters are modulo-ed based on the modulus to obtain the hash value of the second window.

[0088] Before the window movement operation of the second sliding window, a pre-calculation is performed based on the base number, modulus, and window size of the hash function to obtain a basic value. The specific calculation process is as follows:

[0089]

[0090] Wherein, h is the base value, d is the cardinality of the hash function, n is the window size of the second sliding window, and q is the modulus of the hash function.

[0091] The calculation process of updating the hash value of the first window to obtain the hash value of the second window is expressed as:

[0092]

[0093] is the hash value of the first window, is the hash value of the second window, s[x] is the character code of the removed character, and s[x+n] is the character code of the added character. Represents the hash contribution of the removed character to the hash value of the first window.

[0094] Calculate the difference between the hash value of the first window and the hash contribution of the removed characters, and further calculate the product of the difference and the base d to perform exponential alignment; further take the modulus of the product and the sum of the newly added characters based on the modulus, add the hash contribution of the newly added characters, and obtain the hash value of the second window.

[0095] Optionally, the hash function's radix must cover the range of character encodings. For example, for ASCII character encoding, set the radix to 256; for Unicode character encoding, set the radix to 65536. The modulus of the hash function should be set to a large prime number to reduce hash collisions. The window size should be adjusted based on the data type. For example, for text data, set it to 64-256 characters; for binary data, set it to 4KB-1MB.

[0096] The embodiment of the present application updates the hash value of the first window to obtain the hash value of the second window, avoiding recalculating the hash value of the entire window during the second sliding window process, thereby effectively reducing the time complexity and improving the comparison efficiency of the data block to be detected and the target data block.

[0097] In some embodiments, multiple nodes are divided into a master node and multiple local nodes. The master node is connected to each local node through a network or other means. Each node maintains its own preset bit array and preset hash table, and maintains a global bit array and a global hash table in the master node. The global bit array and the global hash table are constructed based on a set of stored data blocks of multiple local nodes. The specific construction method is the same as the construction method of the preset bit array and the preset hash table, which will not be repeated here.

[0098] The duplicate data detection method further includes: detecting non-duplicate data blocks to be detected in each local node based on the global bit array and the global hash table, and determining whether the data blocks to be detected are duplicate data blocks.

[0099] Specifically, the non-duplicate data blocks to be detected in each local node are data blocks that have been detected based on the duplicate data detection process in the above embodiment and are determined to be non-duplicates of the data blocks stored in the local node. For such data blocks, a determination is made as to whether the plurality of second target element bits in the global bit array contain null values.

[0100] If there are no null values ​​for the multiple second target element bits in the global bit array, then the global hash table is used to determine whether the data block to be detected is a duplicate data block. Alternatively, if there are null values ​​for the multiple second target element bits in the global bit array, then the data block to be detected is determined to be a non-duplicate data block, stored in a distributed file system or database, and the global bit array and global hash table are updated for subsequent use.

[0101] Optionally, the master node is also used to shard the data to be tested, transferring the sharded data to each local node via a message queue or distributed file system over the network for repeated testing. This reduces the processing burden on a single node and improves the overall processing capacity of multiple nodes. For example, sharding can be performed based on the hash value range of the text; or, for web page data, sharding can be performed based on the web page's Uniform Resource Locator (URL).

[0102] Optionally, multithreading can be used to accelerate data processing on each node, assigning blocks to be checked to multiple duplicate data detection threads. The degree of parallelism between threads can be set to 2-3 times the number of CPU cores on the node to fully utilize cluster resources.

[0103] The disclosed embodiment uses a preset bit array and a global bit array to collaboratively process the module to be detected, and performs a secondary duplicate check on the non-duplicate data blocks of the local node in the global bit array and the global hash table, thereby further improving the accuracy of duplicate data detection; at the same time, multi-threaded parallel processing is used on each node to accelerate duplicate data detection, further improving the efficiency of duplicate data detection.

[0104] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method.

[0105] Figure 2 The structure diagram of the duplicate data detection device provided in the embodiment of the present application. The duplicate data detection device provided in the embodiment of the present application can execute the processing flow provided in the embodiment of the duplicate data detection method, such as Figure 2As shown, the duplicate data detection device 20 includes a first calculation module 21, a setting module 22, a second calculation module 23, and a determination module 24; the first calculation module 21 is used to perform hash calculations on the hash values ​​of at least one stored data block based on multiple preset hash functions, and determine the addresses of multiple first target element bits corresponding to the at least one stored data block; the setting module 22 is used to set the multiple first target element positions in the preset bit array to non-null values; the second calculation module 23 is used to perform hash calculations on the hash values ​​of the data block to be detected based on multiple preset hash functions, and determine the addresses of multiple second target element bits corresponding to the data block to be detected; the determination module 24 is used to determine whether the data block to be detected is a duplicate data block according to a preset hash table if there are no null values ​​in the multiple second target element bits in the preset bit array.

[0106] Optionally, the duplicate data detection device 20 further includes: a block division module for dividing the data to be detected into blocks according to the type of the data to be detected to obtain multiple data blocks to be detected; and a third calculation module for calculating the hash value of the data block to be detected.

[0107] Optionally, the blocking module is used to move the first sliding window in the data to be detected with a first preset step size if the data to be detected is unstructured data, and calculate the data feature value in the first sliding window in real time; when the data feature value meets the preset conditions, determine the blocking position according to the position of the first sliding window to obtain at least one blocking position; and divide the data to be detected into multiple data blocks to be detected according to the blocking position.

[0108] Optionally, the third calculation module is further configured to determine, in the data block to be detected, a data hash value corresponding to the first sliding window at an initial position as the hash value of the data block to be detected.

[0109] Optionally, the blocking module is further used to block the data to be detected according to a preset number of bytes if the data to be detected is structured data, to obtain multiple data blocks to be detected, and the number of bytes of each data block to be detected is the preset number of bytes.

[0110] Optionally, the duplicate data detection device 20 also includes an update module, which is used to determine that the data block to be detected is not a duplicate data block if there are null values ​​in multiple second target element bits; update the preset bit array according to the multiple second target element bits; and update the preset hash table according to the data block to be detected and the hash value of the data block to be detected.

[0111] Optionally, the determination module 24 includes a search unit, a comparison unit, and a determination unit; the search unit is used to search for a target data block having the same hash value as the data block to be detected in a preset hash table; the comparison unit is used to compare the data block to be detected and the target data block to obtain a comparison result; the determination unit is used to determine whether the data block to be detected is a duplicate data block based on the comparison result.

[0112] Optionally, the comparison unit is used to move the second sliding window between the data block to be detected and the target data block, and calculate in real time the feature value to be detected and the target feature value corresponding to each window movement operation, the feature value to be detected is the data feature value within the second sliding window in the data block to be detected, and the target feature value is the data feature value within the second sliding window in the target data block; compare the feature value to be detected and the target feature value corresponding to the same window movement operation to obtain a comparison result.

[0113] Optionally, the comparison unit is also used to move the second sliding window in a preset data block, determine the removed characters and newly added characters in the second sliding window during a single movement, and the preset data block is the data block to be detected or the target data block; according to the hash contribution of the removed characters and the newly added characters, the hash value of the first window is updated to obtain the hash value of the second window, the hash value of the first window is the hash value of the data in the second sliding window before the single movement operation, and the hash value of the second window is the hash value of the data in the second sliding window after the single movement operation.

[0114] Optionally, the comparison unit is further used to multiply the base value by the removed characters to obtain the hash contribution of the removed characters, where the base value is calculated based on the cardinality, modulus, and window size of the second sliding window of the hash function; determine the difference between the hash value of the first window and the hash contribution of the removed characters, and calculate the product of the difference and the cardinality; and take the modulus of the product and the sum of the newly added characters based on the modulus to obtain the hash value of the second window.

[0115] Optionally, the determination unit is further configured to determine that the data block to be detected and the target data block are not repeated if the feature value to be detected corresponding to at least one window moving operation is different from the target feature value; otherwise, determine that the data block to be detected and the target data block are repeated.

[0116] Optionally, the update module is further used to update the preset bit array according to multiple second target element bits if the data block to be detected is not a duplicate data block; and update the preset hash table according to the data block to be detected and the hash value of the data block to be detected.

[0117] Figure 2 The duplicate data detection device of the illustrated embodiment can be used to execute the technical solution of the above-mentioned method embodiment. Its implementation principle and technical effects are similar and will not be described in detail here.

[0118] Figure 3 The electronic device provided in the embodiment of the present application can execute the processing flow provided in the embodiment of the duplicate data detection method, such as Figure 3 As shown, the electronic device 30 includes: a memory 31, a processor 32, a computer program and a communication interface 33; wherein the computer program is stored in the memory 31 and is configured so that the processor 32 executes the duplicate data detection method described above.

[0119] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. The computer program is configured to execute the steps of any of the above-mentioned duplicate data detection method embodiments when running.

[0120] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0121] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps of any of the above-mentioned duplicate data detection method embodiments are implemented.

[0122] An embodiment of the present application further provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any of the above-mentioned duplicate data detection method embodiments are implemented.

[0123] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0124] The above describes in detail a duplicate data detection method, device, storage medium, and computer program product provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only intended to help understand the method and core concept of the present application. It should be noted that, for those skilled in the art, without departing from the principles of the present application, several improvements and modifications may be made to the present application, and such improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A method for detecting duplicate data, characterized in that: include: Performing hash calculations on hash values ​​of at least one stored data block based on multiple preset hash functions to determine addresses of multiple first target element bits corresponding to the at least one stored data block; Setting the plurality of first target element positions in the preset bit array to non-null values; Performing hash calculations on hash values ​​of the data blocks to be detected based on the multiple preset hash functions, and determining addresses of multiple second target element bits corresponding to the data blocks to be detected; If there is no null value in the plurality of second target element bits in the preset bit array, determining whether the data block to be detected is a duplicate data block according to a preset hash table; The step of determining whether the data block to be detected is a duplicate data block according to a preset hash table includes: Searching the preset hash table for a target data block having the same hash value as the data block to be detected; Comparing the data block to be detected and the target data block to obtain a comparison result; Determining whether the data block to be detected is a duplicate data block according to the comparison result; The comparing the data block to be detected and the target data block to obtain a comparison result includes: Moving a second sliding window between the data block to be detected and the target data block, and calculating in real time a feature value to be detected and a target feature value corresponding to each window movement operation, wherein the feature value to be detected is a feature value of data within the second sliding window in the data block to be detected, and the target feature value is a feature value of data within the second sliding window in the target data block; Comparing the to-be-detected feature value and the target feature value corresponding to the same window movement operation to obtain the comparison result; The moving of the second sliding window between the to-be-detected data block and the target data block, and real-time calculation of the to-be-detected feature value and the target feature value corresponding to each window movement operation, include: Moving a second sliding window in a preset data block, and determining removed characters and added characters in the second sliding window during a single movement, wherein the preset data block is the data block to be detected or the target data block; Update the first window hash value according to the hash contribution of the removed character and the newly added character to obtain a second window hash value, where the first window hash value is a hash value of the data in the second sliding window before the single move operation, and the second window hash value is a hash value of the data in the second sliding window after the single move operation; The updating of the first window hash value according to the hash contribution of the removed character and the newly added character to obtain the second window hash value includes: Multiplying a base value by the removed character to obtain a hash contribution of the removed character, wherein the base value is calculated based on a base number and a modulus of a hash function and a window size of the second sliding window; Determine a difference between the first window hash value and the hash contribution of the removed character, and calculate a product of the difference and the base; Taking a modulus of the product and the sum of the newly added characters based on the modulus to obtain a second window hash value; Determining whether the data block to be detected is a duplicate data block according to the comparison result includes: If the characteristic value to be detected corresponding to at least one window moving operation is different from the target characteristic value, determining that the data block to be detected and the target data block are not repeated; Otherwise, it is determined that the data block to be detected is repeated with the target data block.

2. The duplicate data detection method according to claim 1, wherein: Before performing hash calculations on the hash values ​​of the data blocks to be detected based on the multiple preset hash functions and determining addresses of multiple second target element bits corresponding to the data blocks to be detected, the method further includes: Dividing the data to be detected into blocks according to the type of the data to be detected to obtain multiple data blocks to be detected; Calculate the hash value of the data block to be detected.

3. The duplicate data detection method according to claim 2, wherein: The step of dividing the data to be detected into blocks according to the type of the data to be detected to obtain a plurality of data blocks to be detected includes: If the data to be detected is unstructured data, moving a first sliding window in the data to be detected with a first preset step size, and calculating data feature values ​​within the first sliding window in real time; When the data characteristic value meets a preset condition, determining a block position according to the position of the first sliding window to obtain at least one block position; The data to be detected is divided into a plurality of data blocks to be detected according to the block positions.

4. The method according to claim 3, characterized in that Calculating the hash value of the data block to be detected includes: Determine in the data block to be detected that a data hash value corresponding to the initial position of the first sliding window is a hash value of the data block to be detected.

5. The duplicate data detection method according to claim 2, wherein: The step of dividing the data to be detected into blocks according to the type of the data to be detected to obtain a plurality of data blocks to be detected includes: If the data to be detected is structured data, the data to be detected is divided into blocks according to a preset number of bytes to obtain a plurality of data blocks to be detected, and the number of bytes of each data block to be detected is the preset number of bytes.

6. The duplicate data detection method according to claim 1, wherein: After performing hash calculations on the hash values ​​of the data blocks to be detected based on the multiple preset hash functions and determining addresses of multiple second target element bits corresponding to the data blocks to be detected, the method further includes: If there is a null value in the plurality of second target element bits, determining that the data block to be detected is not a duplicate data block; updating the preset bit array according to the plurality of second target element bits; The preset hash table is updated according to the data block to be detected and the hash value of the data block to be detected.

7. The duplicate data detection method according to claim 1, wherein: The method further comprises: If the data block to be detected is not a duplicate data block, updating the preset bit array according to the multiple second target element bits; The preset hash table is updated according to the data block to be detected and the hash value of the data block to be detected.

8. An electronic device, characterized in that: include: memory for storing computer programs; A processor, configured to implement the steps of the duplicate data detection method according to any one of claims 1 to 7 when executing the computer program.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the duplicate data detection method according to any one of claims 1 to 7 are implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the duplicate data detection method according to any one of claims 1 to 7 are implemented.

Images