Data deduplication method, electronic device, and storage medium

Abstract

The application provides a data deduplication method, an electronic device and a storage medium, and relates to the technical field of data processing. The method comprises the following steps: determining first specified data for to-be-deduplicated data in target data; taking the next data of the end data of the first specified data as a starting point, obtaining to-be-duplicated data with a data length conforming to a first length threshold in the to-be-deduplicated data; determining whether there is repeated data between the to-be-deduplicated data and the deduplicated local data of the target data according to whether there is same data between the to-be-duplicated data and the deduplicated local data; and if there is repeated data, performing data deduplication processing on the to-be-deduplicated data. The application can more accurately determine whether there is repeated data between the to-be-deduplicated data and the deduplicated local data, thereby improving the deduplication effect of data deduplication.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data deduplication method, electronic device, and storage medium. Background Technology

[0002] Data deduplication technology, also known as duplicate data deletion technology, identifies and removes duplicate data from target data. Deduplication not only reduces the storage space required for the target data but also reduces the amount of data transmitted during data transfer. Therefore, data deduplication technology has been widely used in data storage and transmission.

[0003] With the widespread application of data deduplication technology, how to deduplicate target data has become a technical problem that relevant technical personnel have to face. Summary of the Invention

[0004] This application provides a data deduplication method, electronic device, and storage medium to improve the deduplication effect of data deduplication.

[0005] In a first aspect, embodiments of this application provide a data deduplication method, the method comprising:

[0006] For the duplicate data to be removed from the target data, determine the first specified data;

[0007] Starting from the next data point after the last data point of the first specified data, retrieve the data to be deduplicated from the data to be deduplicated that has a length that meets the first length threshold.

[0008] Determine whether there is duplicate data between the data to be deduplicated and the deduplicated partial data of the target data, based on whether there is any identical data between them.

[0009] If duplicate data exists, perform deduplication on the data to be deduplicated.

[0010] Secondly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor implements the method provided in any embodiment of this application when executing the computer program.

[0011] Thirdly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method provided in any embodiment of this application.

[0012] Compared with the prior art, this application has the following advantages:

[0013] The data deduplication method provided in this application first takes the next element of the last element of the specified first data as the starting point, and obtains the data to be deduplicated from the data to be deduplicated that meets the first length threshold. Then, it determines whether there is duplicate data between the data to be deduplicated and the already deduplicated partial data of the target data. If duplicate data is found, the data to be deduplicated is then deduplicated. By determining whether there is duplicate data between the data to be deduplicated that meets the first length threshold and the already deduplicated partial data, the method can more precisely determine whether there is duplicate data between the data to be deduplicated and the already deduplicated partial data. Because it can more precisely determine whether there is duplicate data between the data to be deduplicated and the already deduplicated partial data, the method of determining whether there is duplicate data between the data to be deduplicated and the already deduplicated partial data can improve the deduplication effect.

[0014] The above overview is for illustrative purposes only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments, and features described above, further aspects, embodiments, and features of this application will become readily apparent from the accompanying drawings and the following detailed description. Attached Figure Description

[0015] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the various drawings denote the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings depict only some embodiments disclosed in this application and should not be construed as limiting the scope of this application.

[0016] Figure 1 This is a first schematic diagram of the data deduplication method provided in the embodiments of this application;

[0017] Figure 2 This is a second schematic diagram of the data deduplication method provided in the embodiments of this application;

[0018] Figure 3 This is a third schematic diagram of the data deduplication method provided in the embodiments of this application;

[0019] Figure 4 This is a fourth schematic diagram of the data deduplication method provided in the embodiments of this application;

[0020] Figure 5 This is a flowchart of the data deduplication method provided in the embodiments of this application;

[0021] Figure 6This is a structural block diagram of the data deduplication device provided in the embodiments of this application;

[0022] Figure 7 This is a block diagram of an electronic device used to implement embodiments of this application. Detailed Implementation

[0023] Many specific details are set forth in the following description to provide a full understanding of this application. However, this application can be implemented in many other forms than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this application; therefore, this application is not limited to the specific embodiments disclosed below.

[0024] To more clearly demonstrate the data deduplication method provided in the embodiments of this application, we first introduce the application scenarios of the data deduplication method provided in the embodiments of this application. The data deduplication method provided in the embodiments of this application can be applied to scenarios where data to be stored is deduplicated. For example, the data deduplication method provided in the embodiments of this application can be applied to scenarios where data to be stored is deduplicated in a file management system. Specifically, before storing the data to be stored into a pre-configured storage medium, in order to reduce the storage space required for the data to be stored, data deduplication can be performed on the data to be stored first. At this time, the data to be stored is the target data.

[0025] The following specific combinations Figure 1-4 The figure illustrates the application scenarios of the data deduplication method provided in the embodiments of this application.

[0026] In practical applications, after obtaining data to be stored with a length not less than a preset threshold, it is necessary to first determine whether the data to be stored contains data that has already undergone deduplication. That is, first determine whether there is deduplicated partial data. If there is no deduplicated data in the data to be stored, then it is further determined whether there is compressed data corresponding to the deduplicated partial data, and whether this compressed data is not stored in the pre-constructed storage medium for storing the data to be stored. Since the data to be stored has not yet undergone deduplication at this point, there is no deduplicated data in the data to be stored, and at the same time, there is no compressed data corresponding to the deduplicated partial data.

[0027] If it is determined that there is no compressed data corresponding to the deduplicated local data, it is necessary to further determine the data to be deduplicated. At this time, all data in the data to be stored is the data to be deduplicated. After determining the data to be deduplicated, data compression is performed starting from the data starting point of the data to be deduplicated to obtain the first compressed data whose data length meets the preset length threshold, and the first compressed data is determined as the compressed block (0). Then, the local data corresponding to the first compressed data (compressed block (0)) in the data to be deduplicated is determined as the first specified data, and the first specified data is determined as the original data (0). At this time, the data length of the first specified data (original data (0)) reaches the preset length threshold. Specifically, the data length of the first specified data (original data (0)) is greater than or equal to the preset length threshold.

[0028] It should be noted that the preset length threshold is determined by the storage space corresponding to the storage medium. For example, the storage medium can be a disk block, and when the size of a disk block is 4KB (kilobyte), the preset length threshold is 4KB. Of course, the preset length threshold can also be other length values, including but not limited to 2KB, 24KB, and 128KB. In this embodiment, the preset length threshold is not specifically limited. The following uses a preset length threshold of 4KB as an example to illustrate the application scenario of the data deduplication method provided in this embodiment.

[0029] like Figure 1 As shown, after obtaining the compressed block (0), the first 4KB of data in the original data (0) is used to calculate the hash value, and this hash value is determined as hash(0). After obtaining hash(0), the hash value is determined as hash(0) and stored in the pre-built deduplication list.

[0030] It should be noted that the length of data used to calculate the hash value is determined by a first length threshold. Specifically, when the first length threshold is 4KB, 4KB of data is used to calculate the hash value; and when the first length threshold is 3KB, 3KB of data is used. For ease of distinction, in this embodiment, the preset length threshold corresponding to the data to be deduplicated is defined as the second length threshold.

[0031] Furthermore, this application embodiment does not specifically limit the first length threshold. However, it must be ensured that the first length threshold is not higher than the second length threshold. In the detailed explanation of the application scenario of the data deduplication method provided in this application embodiment, the second length threshold of 4KB is also used for illustration.

[0032] After determining the first specified data (original data (0)), it is necessary to determine whether there is duplicate data between the data to be deduplicated and the deduplicated local data corresponding to the target data, and to perform deduplication processing on the data to be deduplicated based on the determination result.

[0033] The specific implementation process for determining whether there is duplicate data between the data to be deduplicated and the already deduplicated partial data is as follows:

[0034] First, determine whether the length of the second specified data in the data to be deduplicated is lower than the first length threshold, and whether there is the same data to be compared in the first local data in the first specified data; wherein, the starting point of the second specified data is located after the ending point of the first specified data, the first local data is obtained based on the starting point of the first specified data, the starting point of the first local data is the same as the starting point of the first specified data, and the length of the first local data meets the first length threshold.

[0035] Specifically, first, it is determined whether the length of the second specified data is less than a second length threshold (4KB). If it is less than 4KB, the following methods can be used to determine whether there is duplicate data between the data to be deduplicated and the already deduplicated partial data: First, based on whether there is identical data to be compared in the first partial data, it is determined whether there is duplicate data between the data to be deduplicated and the already deduplicated partial data. Second, based on whether there is identical data to be compared in the first partial data, it is determined whether there is duplicate data between the data to be deduplicated and the already deduplicated partial data. The data to be compared is obtained based on the starting data of the already deduplicated partial data. Specifically, the data to be compared is the first 4KB of data in the already deduplicated partial data.

[0036] In the application scenario of the data deduplication method provided in this application embodiment, it is assumed that the data length of the second specified data is not less than the first length threshold. If the data length of the second specified data in the data to be deduplicated is not less than the first length threshold, it is further determined whether there is identical data to be compared in the first local data of the first specified data. If so, it is directly determined that there is duplicate data between the data to be deduplicated and the already deduplicated local data.

[0037] Specifically, the first 4KB of data in the first specified data is determined as the first local data, and then it is determined whether there is any identical data to be compared in the first local data. Accordingly, the specific implementation of determining whether there is identical data to be compared in the first local data is as follows: determine whether there is any deduplicated local data whose first 4KB data is the same as the first 4KB data in the first specified data (original data (0)). If so, it is determined that there is duplicate data between the data to be deduplicated and the deduplicated local data.

[0038] In practical applications, the existence of duplicate local data whose first 4KB of data is identical to the first 4KB of data in the first specified data (original data (0)) can be determined by checking if hash(0) has any other identical hash values. Here, "other hash values" refers to hash values ​​calculated from the first 4KB of data in the duplicate local data. If hash(0) does not have any other identical hash values, it can be determined that there is no duplicate local data whose first 4KB of data is identical to the first 4KB of data in the first specified data (original data (0)). That is, it can be determined that at least the local data does not contain identical data to be compared.

[0039] Only when the length of the second specified data in the data to be deduplicated is not less than the first length threshold, and the first local data in the first specified data does not contain the same data to be compared, will the following iterative process be further adopted to determine whether there is duplicate data between the data to be deduplicated and the deduplicated local data.

[0040] Specifically, the iterative process is as follows: First, starting from the next element after the last element of the specified first data, obtain duplicate data from the data to be deduplicated that meets the first length threshold. Second, determine whether there is duplicate data between the data to be deduplicated and the already deduplicated partial data, based on whether there is identical data to be compared in the data to be deduplicated; the data to be compared is obtained based on the starting data of the already deduplicated partial data. Third, if there is no identical data to be compared in the data to be deduplicated, update the data to be deduplicated, with the starting data of the updated data preceding the starting data of the data to be deduplicated.

[0041] It should be noted that the so-called "end data" refers to the data located at the end of the specified data. In this embodiment, the specified data includes, but is not limited to: target data, deduplicated partial data, first specified data, and data to be deduplicated. For target data, the last data in the target data is the end data of the target data; for data to be deduplicated, the last data in the data to be deduplicated is the end data of the data to be deduplicated. Similarly, for first specified data, the last data in the first specified data is the end data of the first specified data. Taking the data next to the end data of the first specified data as the starting point means taking the data located at the next position after the end data of the first specified data as the starting data.

[0042] The above iterative process can be implemented using a rolling hashing method. First, the window length of the rolling hashing is configured to 4KB, and the starting point is the data following the last data of the first specified data. In this case, the data corresponding to the current window is the data to be deduplicated. Then, the hash value corresponding to the current window needs to be calculated to obtain the first hash value corresponding to the data to be deduplicated, and the first hash value corresponding to the data to be deduplicated is determined as hash(01). Then, it is determined whether there are other hash values ​​that are the same as hash(01) in the deduplication list. For ease of distinction, other hash values ​​are collectively referred to as the second hash value below. That is to say, the so-called second hash value is the hash value corresponding to the data to be compared, specifically the hash value corresponding to the first 4KB of data in the deduplicated local data. At this time, the process of determining whether there are other hash values ​​that are the same as hash(01) in the deduplication list is the process of determining whether there are second hash values ​​that are the same as hash(01) in the deduplication list.

[0043] If a second hash value exists that is identical to hash(01), the iteration stops. This determines if there is duplicate data between the data to be deduplicated and the already deduplicated partial data.

[0044] If hash(01) does not have a matching second hash value, then if hash(0) does not have a matching second hash value, the hash is moved forward by the target number of bytes as the unit of window movement, and the data corresponding to the current window is redefined as the data to be deduplicated. Then, the hash value corresponding to the current window is recalculated and defined as hash(02). That is, the first hash value corresponding to the data to be deduplicated is redefined as hash(02). Then, it is determined in the deduplication list whether a second hash value identical to hash(02) exists.

[0045] The hashing process proceeds sequentially in the manner described above, checking the deduplication list to see if a second hash value exists that matches the hash value corresponding to the current window. The rolling hash stops when either a second hash value matching the hash value corresponding to the current window is found in the deduplication list, or the starting data corresponding to the rolling hash window is not after the last data in the first local data set. Specifically, iteration stops when the starting data corresponding to the rolling hash window is the last data in the first local data set.

[0046] If it is determined that there is no duplicate data between the data to be deduplicated and the deduplicated partial data, the first specified data (original data (0)) is identified as the deduplicated partial data to complete the data deduplication process of the data to be deduplicated. After that, the compressed block (0) needs to be stored in a predetermined disk block. And the following information is recorded in the pre-built deduplication table: the original data (0) corresponds to hash (0), hash (0) corresponds to compressed block (0), and compressed block (0) is stored in the specified disk block.

[0047] After storing the compressed block (0) into a predetermined disk block, it is necessary to further redetermine the data to be deduplicated. Then, based on the redetermined data to be deduplicated, the first specified data is updated. The starting point of the redetermined data to be deduplicated is located after the ending point of the first specified data, and the starting point of the updated first specified data is located after the ending point of the first specified data.

[0048] Specifically, data compression is performed starting from the newly determined deduplicated data to be compressed, resulting in first compressed data whose length meets the second length threshold. This first compressed data is then designated as a compression block (1). The local data corresponding to the first compressed data (compression block (1)) in the newly determined deduplicated data is then designated as the updated first specified data, and this first specified data is then designated as the original data (1). At this point, the data length of the first specified data reaches the second length threshold.

[0049] like Figure 2 As shown, after obtaining the compressed block (1), the first 4KB of data in the original data (1) is used to calculate the hash value, and the hash value is determined as hash(1). After obtaining hash(1), hash(1) is stored in a pre-built deduplication list.

[0050] After determining the first specified data (original data (1)), it is necessary to determine whether there is duplicate data between the data to be deduplicated and the deduplicated local data corresponding to the target data. The specific implementation process is as follows:

[0051] Determine whether the length of the second specified data in the data to be deduplicated is lower than the first length threshold, and whether the first local data in the first specified data does not have the same data to be compared; wherein, the starting point of the second specified data is located after the ending point of the first specified data (original data (1)), and the first local data is obtained based on the starting data of the first specified data.

[0052] Specifically, first, it is determined whether the length of the second specified data is less than the first length threshold (4KB). If it is less than 4KB, the following methods can only be used to determine whether there is duplicate data between the data to be deduplicated and the already deduplicated partial data: First, based on whether there is identical data to be compared in the first partial data, it is determined whether there is duplicate data between the data to be deduplicated and the already deduplicated partial data. Second, based on whether there is identical data to be compared in the first partial data, it is determined whether there is duplicate data between the data to be deduplicated and the already deduplicated partial data. Specifically, the data to be compared is the first 4KB of data in the already deduplicated partial data.

[0053] In the application scenario of the data deduplication method provided in this application embodiment, it is assumed that the data length of the second specified data is not less than the first length threshold. If the data length of the second specified data in the data to be deduplicated is not less than the first length threshold, it is further determined whether there is identical data to be compared in the first local data of the first specified data. If so, it is directly determined that there is duplicate data between the data to be deduplicated and the already deduplicated local data.

[0054] Specifically, the first 4KB of data in the first specified data is determined as the first local data, and then it is determined whether there is any identical data to be compared in the first local data. The data to be compared is obtained based on the starting data of the deduplicated local data. Accordingly, the specific implementation of determining whether there is identical data to be compared in the first local data is as follows: determine whether there is any data in the deduplicated local data that is the same as the first 4KB of data in the first specified data (original data (1)). If so, it is determined that there is duplicate data between the data to be deduplicated and the deduplicated local data.

[0055] In practical applications, it can be determined whether there is deduplicated local data that is the same as the first 4KB of the first specified data (original data (1)) by determining whether hash(1) has the same second hash value. If there is no same second hash value in hash(1), it can be determined that there is no identical data to be compared in the first local data.

[0056] If the length of the second specified data in the data to be deduplicated is not less than the first length threshold, and there is no identical data to be compared in the first local data of the first specified data, the following iterative process is further adopted to determine whether there is duplicate data between the data to be deduplicated and the deduplicated local data. Specifically, the iterative process is as follows: First, starting from the next data after the last data of the first specified data, obtain the data to be deduplicated in the data to be deduplicated whose data length meets the first length threshold. Second, determine whether there is duplicate data between the data to be deduplicated and the deduplicated local data based on whether there is identical data to be compared in the data to be deduplicated; the data to be compared is obtained based on the starting data of the deduplicated local data. Third, if there is no identical data to be compared in the data to be deduplicated, update the data to be deduplicated, and the starting data of the updated data to be deduplicated is placed before the starting data of the data to be deduplicated.

[0057] The above iterative process can be implemented using a rolling hash. First, the window length of the rolling hash is configured to 4KB, and the starting point of the rolling hash is the next data after the last data of the first specified data. That is, the starting position of the rolling hash is the next position after the last data of the previous first specified data (original data (0)). The data corresponding to the current window is the data to be deduplicated. Then, the hash value corresponding to the current window needs to be calculated and determined as hash(11). That is, the first hash value corresponding to the data to be deduplicated is determined as hash(11). Then, it is determined whether there is a second hash value that is the same as hash(11) in the deduplication list.

[0058] If hash(11) has the same second hash value, the iteration stops. It is then determined that there is duplicate data between the data to be deduplicated and the already deduplicated local data.

[0059] If hash(11) does not have a second hash value, then the hash is moved forward by the target number of bytes as the unit of window movement, and the data corresponding to the current window is re-determined as the data to be deduplicated. The hash value corresponding to the current window is recalculated and determined as hash(12). That is, the first hash value corresponding to the data to be deduplicated is determined as hash(12). Then, it is determined whether there is a second hash value that is the same as hash(12) in the deduplication list.

[0060] The hash is rolled sequentially as described above, and a second hash value with the same hash value as the current window is checked in the deduplication list. The rolling hash stops rolling when: a second hash value with the same hash value as the current window is found in the deduplication list, or the starting data corresponding to the rolling hash window is not after the last data of the first local data. Specifically, the data corresponding to the rolling hash window is the last data of the first local data.

[0061] Assuming hash(12) is the same as hash(0), it is determined that there is duplicate data between the data to be deduplicated (original data(1)) and the original data(0). At this time, starting from the next bit of the data at the end of the current window, the duplicate data between the data to be deduplicated (original data(1)) and the original data(0) is determined in byte order from front to back, until the original data(1) contains data different from the original data(0). At this time, the duplicate data between the two is determined as the target duplicate data, and the target duplicate data is determined as the original data(11) to complete the data deduplication process of the data to be deduplicated.

[0062] If it is determined that there is a target duplicate data (original data (11)) between the data to be deduplicated (original data (1)) and the original data (0), the compression block (1) needs to be updated. The data endpoint of the updated compression block (1) in the first specified data is located before the starting data of the target duplicate data (original data (11)). The data corresponding to the updated compression block (1) in the first specified data is the original data (12).

[0063] After obtaining the updated compressed block (1), it is necessary to store the compressed block (1) to a predetermined disk block. The following information should be recorded in the pre-built deduplication table: hash (1) corresponds to compressed block (1), compressed block (1) corresponds to original data (11), compressed block (1) is stored in a certain disk block, and the original data (11) is data that overlaps with the original data (0) corresponding to compressed block (0). The overlapping data is the data from the Nth data to the Mth data in the original data (0). Here, M and N are positive integers, and M > N.

[0064] like Figure 3 As shown, after storing the compressed block (1) into a predetermined disk block, it is necessary to first redetermine the deduplicated data based on the target duplicate data (original data (11)). In the redetermined deduplicated data, the target duplicate data is updated to the first specified data to obtain the updated first specified data, and the first specified data is determined as the original data (2). That is, the original data (11) is updated to the original data (2).

[0065] After determining the first specified data (original data (2)), it is necessary to determine whether there is duplicate data between the data to be deduplicated and the deduplicated local data.

[0066] Assuming it's determined that there are no duplicate data between the data to be deduplicated and the already deduplicated partial data, it's necessary to further determine if there are any compressed blocks that need to be stored on disk. Since no compressed blocks need to be stored in this process, the following steps are required: first, redetermine the data to be deduplicated, and then update the first specified data based on the redetermined data. Specifically, the starting point of the redetermined data to be deduplicated is located after the ending point of the first specified data, and the starting point of the updated first specified data is also located after the ending point of the first specified data.

[0067] Specifically, data compression is performed starting from the newly determined deduplicated data to be compressed, resulting in first compressed data whose length meets the second length threshold. This first compressed data is then designated as a compression block (2). The local data corresponding to the first compressed data (compression block (2)) in the newly determined deduplicated data is then designated as the updated first specified data, and this first specified data is designated as the original data (3). At this point, the data length of the first specified data reaches the first length threshold.

[0068] like Figure 4 As shown, after obtaining the compressed block (2), the first 4KB of data in the original data (3) is used to calculate the hash value, and the hash value is determined as hash (2). After obtaining hash (2), hash (2) is stored in a pre-built deduplication list.

[0069] After storing hash(2) into the pre-built deduplication list, based on the updated first specified data, it is re-determined whether there is duplicate data between the data to be deduplicated and the deduplicated partial data.

[0070] The process of determining whether duplicate data exists between the data to be deduplicated and the already deduplicated partial data is executed iteratively until the corresponding configured second iteration stopping condition is met. The second iteration stopping condition includes completing the deduplication of the data to be stored.

[0071] In the application scenario of the data deduplication method provided in this application embodiment, starting from the next digit of the last digit of the first specified data, data to be deduplicated that meets the first length threshold is obtained from the data to be deduplicated. Then, based on whether there is any identical data between the data to be deduplicated and the already deduplicated partial data of the data to be stored, it is determined whether there is duplicate data between the data to be deduplicated and the already deduplicated partial data. If duplicate data exists, data deduplication processing is performed on the data to be deduplicated. By determining whether there is any identical data between the data to be deduplicated that meets the first length threshold and the already deduplicated partial data, the method of determining whether there is duplicate data between the data to be deduplicated and the already deduplicated partial data can more precisely determine whether there is duplicate data between the data to be deduplicated and the already deduplicated partial data. Since it can more precisely determine whether there is duplicate data between the data to be deduplicated and the already deduplicated partial data, the deduplication effect of data deduplication can be improved.

[0072] In determining whether duplicate data exists between the data to be deduplicated and the already deduplicated partial data, the following iterative process is required: If there is identical data between the data to be deduplicated and the already deduplicated partial data, then it is determined that there is duplicate data between the data to be deduplicated and the already deduplicated partial data; if there is no identical data between the data to be deduplicated and the already deduplicated partial data, then the data to be deduplicated is updated, and based on whether there is identical data between the updated data to be deduplicated and the already deduplicated partial data, it is determined whether there is duplicate data between the updated data to be deduplicated and the already deduplicated partial data.

[0073] The iterative process described above is used to determine whether duplicate data exists between the data to be deduplicated and the already deduplicated partial data. This allows for a more precise assessment of whether duplicate data exists between these two sets of data. Therefore, using this iterative process to determine whether duplicate data exists between the data to be deduplicated and the already deduplicated partial data can improve the deduplication effect.

[0074] In addition, performing deduplication on the data to be stored before storing it on the storage medium can reduce the storage space required for the data.

[0075] Furthermore, if there is no duplicate data between the data to be deduplicated and the partially deduplicated data, the compressed data corresponding to the first specified data is stored in the storage medium. However, if there is duplicate data between the data to be deduplicated and the partially deduplicated data, the compressed data corresponding to the first specified data needs to be updated first, and the updated compressed data is stored in the storage medium. Storing the compressed data corresponding to the partially deduplicated data in the storage medium after deduplication reduces the storage space required to store the data to be stored.

[0076] It should be noted that the above-described application scenarios of the data deduplication method provided in the embodiments of this application are for the purpose of facilitating understanding of the data deduplication method provided in the embodiments of this application, and are not intended to limit the data deduplication method provided in the embodiments of this application. The data deduplication method provided in the embodiments of this application can also be used in other scenarios, such as in the scenario of deduplicating data to be transmitted, where the data to be transmitted is the target data. Specifically, the application scenarios of the data deduplication method provided in the embodiments of this application are not specifically limited.

[0077] This application provides a data deduplication method, which is applied to the process of deduplicating target data. Figure 5 This is a flowchart of the data deduplication method provided in the embodiments of this application. Figure 5 As shown, the method includes:

[0078] Step S501: For the data to be deduplicated in the target data, determine the first specified data.

[0079] In this embodiment, the target data includes, but is not limited to, data to be stored and data to be transmitted. Specifically, the target data can be image data, text data, or audio data. In this embodiment, the data type of the target data is not specifically limited.

[0080] The so-called deduplicated data refers to the data in the target data that needs to be deduplicated. The so-called first specified data generally refers to the data in the deduplicated data whose length reaches the second length threshold, starting from the data starting point of the deduplicated data.

[0081] The second length threshold is determined by the storage space corresponding to the storage medium. For example, the storage medium can be a disk block, and when the size of the disk block is 16KB, the second length threshold is 16KB. Of course, the second length threshold can also be other length values, including but not limited to 2KB, 4KB, and 8KB. In this embodiment, the second length threshold is not specifically limited. The following uses a second length threshold of 16KB as an example to illustrate the application scenario of the data deduplication method provided in this embodiment. After determining the first local data, step S502 needs to be further executed.

[0082] Step S502: Starting from the next data point after the last data point of the first specified data, obtain the data to be deduplicated from the data to be deduplicated that has a data length that meets the first length threshold.

[0083] The term "end data" refers to the data located at the end of a specified set of data. In this embodiment, the specified data includes, but is not limited to, the target data, deduplicated partial data, the first specified data, and the data to be deduplicated. For example, for the target data, the last data element is the end data of the target data; similarly, for the data to be deduplicated, the last data element is the end data of the data to be deduplicated. Likewise, for the first specified data, the last data element is the end data of the first specified data. "Taking the next data element of the end data of the first specified data as the starting point" means taking the data element located at the next position after the end data of the first specified data as the starting data.

[0084] In this embodiment, the data to be compared refers to data whose length meets a first length threshold from the deduplicated local data. The first length threshold is not higher than a second length threshold. That is, the first length threshold can be equal to or lower than the second length threshold. For example, when the second length threshold is 16KB, the first length threshold is less than or equal to 16KB.

[0085] The data to be checked for plagiarism is data whose length is equal to the first length threshold. Taking the first length threshold as 16KB as an example, when the first length threshold is 16KB, the data to be checked for plagiarism is the data in the first local data whose length is equal to 16KB.

[0086] After determining the data to be checked for plagiarism, step S503 needs to be performed.

[0087] Step S503: Determine whether there is duplicate data between the data to be deduplicated and the deduplicated partial data based on whether there is any identical data between the data to be deduplicated and the target data.

[0088] The term "deduplicated local data" refers to the local data within the target data that has already undergone deduplication.

[0089] The determination of whether duplicate data exists between the data to be deduplicated and the already deduplicated portions of the target data, based on whether identical data exists between these portions, includes: if identical data exists between the data to be deduplicated and the already deduplicated portions of the target data, then duplicate data exists between the data to be deduplicated and the already deduplicated portions of the target data; if no identical data exists between the data to be deduplicated and the already deduplicated portions of the target data, then duplicate data exists between the data to be deduplicated and the already deduplicated portions of the target data.

[0090] Step S504: If duplicate data exists, perform data deduplication on the data to be deduplicated.

[0091] If it is determined that there is duplicate data between the data to be deduplicated and the partially deduplicated data, it is necessary to first determine the target duplicate data between the data to be deduplicated and the partially deduplicated data, and then perform data deduplication processing on the data to be deduplicated based on the target duplicate data.

[0092] In this embodiment, the specific implementation method for deduplicating the data to be deduplicated is as follows: First, the deduplicated local data that contains the same data as the data to be deduplicated is identified as the target local data. Then, it is determined that there is target duplicate data between the data to be deduplicated and the target local data, starting from the data starting point of the data to be deduplicated. Finally, deduplication is performed on the data to be deduplicated based on the target duplicate data.

[0093] Specifically, first identify the target local data that overlaps with the data to be deduplicated. For example, there are two deduplicated local data sets: deduplicated local data set 1 and deduplicated local data set 2. The data to be deduplicated and deduplicated local data set 1 contain duplicate data. In this case, the target duplicate data can be obtained using the following method:

[0094] First, the data to be checked for duplicates is identified as duplicate data. Then, starting from the next element after the last element in the data to be checked, and from the corresponding element in the already deduplicated section 1, the data is compared sequentially from front to back to see if any duplicates exist between the data to be checked and the already deduplicated section 1, until a distinct difference is found. Finally, if no further duplicates exist between the data to be checked and the already deduplicated section 1, the data to be checked is identified as the target duplicate data; otherwise, the data to be checked and any subsequent duplicates with the already deduplicated section are also identified as target duplicate data.

[0095] The data deduplication method provided in this application first takes the next element of the last element of the specified first data as the starting point, and obtains the data to be deduplicated from the data to be deduplicated that meets the first length threshold. Then, it determines whether there is duplicate data between the data to be deduplicated and the already deduplicated partial data of the target data. If duplicate data is found, the data to be deduplicated is then deduplicated. By determining whether there is duplicate data between the data to be deduplicated that meets the first length threshold and the already deduplicated partial data, the method can more precisely determine whether there is duplicate data between the data to be deduplicated and the already deduplicated partial data. Because it can more precisely determine whether there is duplicate data between the data to be deduplicated and the already deduplicated partial data, the deduplication effect can be improved.

[0096] In this embodiment, if there is no identical data between the data to be checked and the deduplicated partial data, the data to be checked needs to be updated. It is ensured that the starting data of the updated data to be checked is located before the starting data of the data to be deduplicated, and a specified number of bytes are spaced between the starting data of the updated data to be checked and the starting data of the data to be deduplicated. After updating the data to be checked, it is necessary to further determine whether there is duplicate data between the updated data to be deduplicated and the deduplicated partial data based on whether there is identical data between them.

[0097] In practical applications, the process of determining whether there is duplicate data between the data to be deduplicated and the already deduplicated local data is an iterative process as follows, until the corresponding configured iteration stop condition is met.

[0098] The first step involves using the next element after the last element of the specified first data as the starting point to retrieve duplicate data from the data to be deduplicated, ensuring the data length meets the first length threshold. The second step determines whether duplicate data exists between the data to be deduplicated and the already deduplicated portions of the target data. The fourth step confirms this by identifying if identical data exists between the data to be deduplicated and the already deduplicated portions of the target data. The fifth step updates the data to be deduplicated and repeats steps two through five if no identical data exists between the data to be deduplicated and the already deduplicated portions.

[0099] The corresponding iteration stopping conditions include, but are not limited to: duplicate data exists between the data to be deduplicated and the partially deduplicated data.

[0100] The iterative process described above is used to determine whether duplicate data exists between the data to be deduplicated and the already deduplicated partial data. This allows for a more precise assessment of whether duplicate data exists between these two sets of data. Therefore, using this iterative process to determine whether duplicate data exists between the data to be deduplicated and the already deduplicated partial data can improve the deduplication effect.

[0101] In one possible implementation, the specified number of bytes can be 1 byte. By specifying the number of bytes, the level of detail in the duplicate data search can be controlled. The smaller the specified number of bytes, the more detailed the duplicate data search. In this case, the probability of finding duplicate data is greater, and the deduplication effect will be correspondingly better.

[0102] When a specified number of bytes is 1 byte, it can ensure that during the data deduplication process, even if 1 byte of data is added to the data to be deduplicated, the data deduplication process can still continue on the data to be deduplicated after the addition of 1 byte.

[0103] One possible implementation involves taking the next element of the last element of the first specified data as the starting point and retrieving duplicate data from the target data that meets the first length threshold. The implementation method is as follows:

[0104] First, determine whether the length of the second specified data in the data to be deduplicated is not less than the first length threshold; the starting point of the second specified data is located after the last data of the first specified data. Then, when the length of the second specified data is not less than the first length threshold, take the next data after the last data of the first specified data as the starting point, and obtain the data to be deduplicated from the data to be deduplicated.

[0105] In other words, in this embodiment, only when the length of the second specified data is not less than the first length threshold can the data to be deduplicated be further determined, and whether there is duplicate data between the data to be deduplicated and the already deduplicated partial data is determined based on whether there is the same data between the data to be deduplicated and the already deduplicated partial data. However, when the length of the second specified data is less than the first length threshold, it is necessary to first determine whether the first partial data in the first specified data is the same as the second partial data in the already deduplicated partial data, and then determine whether there is duplicate data between the data to be deduplicated and the already deduplicated partial data based on whether the first partial data and the second partial data are the same.

[0106] It should be noted that the starting point of the first local data is the same as the starting point of the first specified data, and the starting point of the second local data is the same as the starting point of the deduplicated local data. The data lengths of the first local data and the deduplicated local data conform to the first length threshold. Taking a first length threshold of 4KB as an example, the first local data is the first 4KB of data in the first specified data, and the second local data is the first 4KB of data in the deduplicated local data.

[0107] In other words, when the length of the second specified data is lower than the first length threshold, the system directly determines whether there is duplicate data between the data to be deduplicated and the deduplicated local data by determining whether the first 4KB of data in the first specified data is the same as the first 4KB of data in the deduplicated local data.

[0108] In one possible implementation, before determining whether duplicate data exists between the data to be deduplicated and the deduplicated portions of the target data, it is necessary to first determine whether there is any common data between the data to be deduplicated and the deduplicated portions. The process for determining whether there is common data between the data to be deduplicated and the deduplicated portions is as follows:

[0109] First, data to be compared that meets the first length threshold is obtained from the deduplicated partial data. Then, based on whether there is identical data to be compared in the data to be deduplicated, it is determined whether there is any identical data between the data to be deduplicated and the deduplicated partial data.

[0110] The data to be compared is generally taken as the starting point of the deduplicated local data. That is, the starting point of the data to be compared is often the same as the starting point of the deduplicated local data. The deduplicated local data refers to the local data obtained after deduplicating local data in the target data.

[0111] In practical applications, the method for determining whether there is identical data between the data to be checked and the deduplicated partial data, based on whether there is identical data to be compared, is as follows: First, determine the first hash value corresponding to the data to be checked. Then, determine the second hash value corresponding to the data to be compared. Finally, determine whether there is identical data between the data to be checked and the deduplicated partial data based on whether the first hash value is the same as the second hash value.

[0112] Specifically, the process of determining whether there are identical data between the data to be checked and the deduplicated partial data can be transformed into the process of determining whether there are identical second hash values ​​for the first hash value. If there are identical second hash values ​​for the first hash value, then it is determined that there are identical data between the data to be checked and the deduplicated partial data. Conversely, if there are no identical second hash values ​​for the first hash value, then it is determined that there are no identical data between the data to be checked and the deduplicated partial data.

[0113] In this application, when it is determined that there is duplicate data between the data to be deduplicated and the partially deduplicated data, after deduplicating the data to be deduplicated, the following steps are further performed: First, based on the target duplicate data, update the data to be deduplicated; the updated data to be deduplicated starts from the data starting point of the target duplicate data. Then, update the target duplicate data with the first specified data to obtain the updated first specified data.

[0114] If it is determined that there is no duplicate data between the data to be deduplicated and the already deduplicated partial data, the following steps are further required: First, if there is no duplicate data between the data to be deduplicated and the already deduplicated partial data, update the first specified data to the already deduplicated partial data. Then, redetermine the data to be deduplicated; the starting point of the redetermined data to be deduplicated is after the last data of the first specified data. Finally, for the redetermined data to be deduplicated, update the first specified data; the starting point of the updated first specified data is after the last data of the first specified data.

[0115] Regardless of whether there is duplicate data between the data to be deduplicated and the already deduplicated partial data, the following steps must be performed after updating the first specified data:

[0116] First, starting from the next data point after the last data point of the updated first specified data, extract the data to be deduplicated from the newly determined data to be deduplicated, where the data length meets the first length threshold.

[0117] Secondly, based on whether there are identical data between the data to be deduplicated and the deduplicated partial data, it is determined whether there are duplicate data between the newly determined data to be deduplicated and the deduplicated partial data.

[0118] Next, based on whether there is any identical data between the newly determined data to be deduplicated and the already deduplicated partial data, it is determined whether there is duplicate data between the newly determined data to be deduplicated and the already deduplicated partial data.

[0119] Finally, if duplicate data exists, the newly determined data to be deduplicated is then processed for deduplication.

[0120] In other words, in this embodiment of the application, the process of deduplicating the data to be deduplicated is also an iterative process, and each iteration performs the following steps:

[0121] The first step is to identify the data to be deduplicated in the target data. The second step is to determine the first specified data for the data to be deduplicated in the target data. The third step is to start from the next element after the last element of the first specified data and obtain the data to be deduplicated from the data to be deduplicated that meets the first length threshold. The fourth step is to determine whether there is duplicate data between the data to be deduplicated and the already deduplicated parts of the target data. The fifth step is to perform deduplication on the data to be deduplicated if duplicate data exists, and then update the data to be deduplicated based on the target duplicate data; if no duplicate data exists, the first specified data is updated to the already deduplicated parts, and the data to be deduplicated is re-determined. After re-determining the data to be deduplicated, steps two through five are repeated.

[0122] The corresponding iteration stopping conditions include, but are not limited to: completing the deduplication of the target data.

[0123] In one possible implementation, updating the first specified data can be done as follows: First, data compression is performed starting from the newly determined starting point of the data to be deduplicated, resulting in first compressed data whose length meets the second length threshold. Then, the data corresponding to the first compressed data in the target data is determined as the updated first specified data.

[0124] In one possible implementation, after deduplication of the data to be deduplicated, the following steps can be further performed: First, determine the second compressed data corresponding to the first specified data. Then, based on the target duplicate data, update the second compressed data; the end data of the updated second compressed data in the target data is located before the start data of the target duplicate data. Finally, determine the data corresponding to the updated second compressed data in the target data as the deduplicated local data.

[0125] It should be noted that the so-called second compressed data also refers to the compressed data corresponding to the first specified data. If there is duplicate data between the data to be deduplicated and the already deduplicated partial data, the second compressed data can be updated first, and the updated second compressed data stored on the storage medium.

[0126] Whether the compressed data corresponding to the first specified data is stored directly to the storage medium, or the updated compressed data corresponding to the first specified data is stored to the storage medium, the storage space required for the target data can be reduced.

[0127] Corresponding to the application scenarios and data deduplication methods provided in the embodiments of this application, the embodiments of this application also provide a data deduplication device, which is applied in the process of deduplicating target data. Figure 6 As shown, Figure 6 This is a structural block diagram of the data deduplication device provided in the embodiments of this application. The data deduplication device may include:

[0128] The specified data determination module 601 is used to determine the first specified data for the deduplicated data in the target data;

[0129] The data to be deduplicated acquisition module 602 is used to acquire data to be deduplicated from the data to be deduplicated, with the data length conforming to the first length threshold, starting from the next data after the last data of the first specified data;

[0130] The duplicate data determination module 603 is used to determine whether there is duplicate data between the data to be deduplicated and the deduplicated partial data of the target data based on whether there is any identical data between the data to be deduplicated and the deduplicated partial data of the target data.

[0131] The deduplication module 604 is used to perform deduplication on the data to be deduplicated when duplicate data exists.

[0132] In one possible implementation, the duplicate data determination module 603 is further configured to update the data to be checked if there is no identical data between the data to be checked and the deduplicated partial data; the starting data of the updated data to be checked is located before the starting data of the data to be checked, and there is a specified number of bytes between the starting data of the updated data to be checked and the starting data of the data to be checked; based on whether there is identical data between the updated data to be checked and the deduplicated partial data, it is determined whether there is duplicate data between the updated data to be checked and the deduplicated partial data; the process of determining whether there is duplicate data between the data to be checked and the deduplicated partial data is iteratively executed until the corresponding configured iteration stop condition is met.

[0133] In one possible implementation, the device further includes:

[0134] The comparison data acquisition module is used to acquire comparison data whose length meets the first length threshold from the deduplicated local data;

[0135] The identical data determination module is used to determine whether there is identical data between the data to be checked and the deduplicated local data, based on whether there is identical data to be compared.

[0136] In one possible implementation, the same data determination module includes:

[0137] The first hash value determination submodule is used to determine the first hash value corresponding to the data to be deduplicated.

[0138] The second hash value determination submodule is used to determine the second hash value corresponding to the data to be compared.

[0139] The identical data determination submodule is used to determine whether there is identical data between the data to be deduplicated and the deduplicated local data, based on whether the first hash value has the same second hash value.

[0140] In one possible implementation, the data acquisition module 602 is specifically used to determine whether the data length of the second specified data in the data to be deduplicated is not less than the first length threshold; the starting point of the second specified data is located after the last data of the first specified data; when the data length of the second specified data is not less than the first length threshold, the data to be deduplicated is acquired from the data to be deduplicated, starting from the next data of the last data of the first specified data.

[0141] In one possible implementation, the device further includes:

[0142] The local data comparison module is used to determine whether the first local data in the first specified data is the same as the second local data in the deduplicated local data when the data length of the second specified data is lower than the first length threshold.

[0143] The data determination module is used to determine whether there is duplicate data between the data to be deduplicated and the deduplicated local data based on whether the first local data and the second local data are the same.

[0144] The starting point of the first local data is the same as the starting point of the first specified data, the starting point of the second local data is the same as the starting data of the deduplicated local data, and the data length of the first local data and the deduplicated local data meets the first length threshold.

[0145] In one possible implementation, the deduplication module 604 includes:

[0146] The target local data determination submodule is used to determine the deduplicated local data that contains the same data as the data to be deduplicated as the target local data.

[0147] The target duplicate data determination submodule is used to determine whether there is target duplicate data between the data to be deduplicated and the target local data, with the starting point of the data to be deduplicated as the data starting point of the data to be deduplicated.

[0148] The first deduplication submodule is used to perform deduplication processing on the target duplicate data.

[0149] In one possible implementation, the specified data determination module 601 is further configured to, after performing data deduplication on the data to be deduplicated, update the data to be deduplicated based on the target duplicate data; the updated data to be deduplicated takes the starting point of the target duplicate data as the starting point; and update the target duplicate data to the first specified data to obtain the updated first specified data.

[0150] In one possible implementation, the specified data determination module 601 is further configured to update the first specified data to the deduplicated partial data when there is no duplicate data between the data to be deduplicated and the deduplicated partial data; redetermine the data to be deduplicated; the starting point of the redetermined data to be deduplicated is located after the end data of the first specified data; update the first specified data for the redetermined data to be deduplicated; the starting point of the updated first specified data is located after the end data of the first specified data.

[0151] In one possible implementation, the data to be deduplicated acquisition module 602 is further configured to take the next data of the last data of the updated first specified data as the starting point, and acquire the data to be deduplicated from the newly determined data to be deduplicated that has a data length that meets the first length threshold.

[0152] The duplicate data determination module 603 is also used to determine whether there is duplicate data between the newly determined data to be deduplicated and the already deduplicated partial data, based on whether there is any identical data between the newly determined data to be deduplicated and the already deduplicated partial data.

[0153] The deduplication module 604 is also used to perform deduplication processing on the newly determined data to be deduplicated when duplicate data exists.

[0154] In one possible implementation, the specified data determination module 601 includes:

[0155] The first compressed data acquisition submodule is used to compress data starting from the newly determined starting point of the deduplicated data to obtain first compressed data whose data length meets the second length threshold.

[0156] The first specified data determination submodule is used to determine the data corresponding to the first compressed data in the target data as the updated first specified data.

[0157] In one possible implementation, the device further includes:

[0158] The second compressed data determination module is used to determine the second compressed data corresponding to the first specified data;

[0159] The second compressed data update module is used to update the second compressed data based on the target duplicate data; the updated second compressed data has the last data in the target data before the first data in the target duplicate data.

[0160] The module for determining deduplicated local data is used to identify the data corresponding to the updated second compressed data in the target data as deduplicated local data.

[0161] Figure 7 This is a block diagram of an electronic device used to implement embodiments of this application. Figure 7 As shown, the electronic device includes a memory 710 and a processor 720. The memory 710 stores a computer program that can run on the processor 720. When the processor 720 executes the computer program, it implements the method described in the above embodiments. The number of memories 710 and processors 720 can be one or more.

[0162] The electronic device also includes:

[0163] The communication interface 730 is used to communicate with external devices and perform data exchange and transmission.

[0164] If the memory 710, processor 720, and communication interface 730 are implemented independently, they can be interconnected via a bus to communicate with each other. This bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 7 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0165] Optionally, in a specific implementation, if the memory 710, processor 720, and communication interface 730 are integrated on a single chip, then the memory 710, processor 720, and communication interface 730 can communicate with each other through an internal interface.

[0166] This application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method provided in this application.

[0167] This application also provides a chip, which includes a processor for calling and executing instructions stored in a memory, causing a communication device on which the chip is installed to perform the method provided in this application.

[0168] This application also provides a chip, including: an input interface, an output interface, a processor, and a memory. The input interface, output interface, processor, and memory are connected through an internal connection path. The processor is used to execute code in the memory. When the code is executed, the processor is used to execute the method provided in the application embodiment.

[0169] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. General-purpose processors can be microprocessors or any conventional processor. It is worth noting that the processor can be a processor supporting Advanced Reduced Instruction Set Machines (ARM) architecture.

[0170] Further, optionally, the aforementioned memory may include read-only memory and random access memory, and may also include non-volatile random access memory. The memory may be volatile or non-volatile, or may include both. Non-volatile memory may include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which serves as an external cache. Many forms of RAM are available by way of example, but not limitation. Examples include Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0171] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.

[0172] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0173] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0174] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process. Furthermore, the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functionality involved.

[0175] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0176] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. All or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware, the program being stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiments.

[0177] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. This storage medium can be a read-only memory, a disk, or an optical disk, etc.

[0178] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various variations or substitutions within the technical scope disclosed in this application, and these should all be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data deduplication method, the method comprising: determining first specified data for to-be-deduplicated data in target data; acquiring, in the to-be-deduplicated data, to-be-duplicated data with a data length conforming to a first length threshold, starting from next data of terminal data of the first specified data; acquiring, in deduplicated local data of the target data, to-be-compared data with the data length conforming to the first length threshold, determining whether there is identical data between the to-be-duplicated data and the deduplicated local data according to whether the to-be-duplicated data has identical to-be-compared data, and determining whether there is repeated data between the to-be-deduplicated data and the deduplicated local data according to whether there is identical data between the to-be-duplicated data and the deduplicated local data; if there is repeated data, performing data deduplication processing on the to-be-deduplicated data.

2. The method of claim 1, wherein, The method further comprises: if there is no identical data between the to-be-duplicated data and the deduplicated local data, updating the to-be-duplicated data, wherein starting data of the updated to-be-duplicated data is located before starting data of the to-be-duplicated data, and the starting data of the updated to-be-duplicated data is spaced apart from the starting data of the to-be-duplicated data by a specified number of bytes; determining whether there is repeated data between the updated to-be-deduplicated data and the deduplicated local data according to whether there is identical data between the updated to-be-duplicated data and the deduplicated local data; iteratively performing the above process of determining whether there is repeated data between the to-be-deduplicated data and the deduplicated local data until an iteration stop condition corresponding to a configuration is met.

3. The method of claim 1, wherein, The determining whether there is identical data between the to-be-duplicated data and the deduplicated local data according to whether the to-be-duplicated data has identical to-be-compared data comprises: determining a first hash value corresponding to the to-be-duplicated data; determining a second hash value corresponding to the to-be-compared data; determining whether there is identical data between the to-be-duplicated data and the deduplicated local data according to whether there is identical second hash value.

4. The method of claim 1, wherein, The acquiring, in the to-be-deduplicated data, to-be-duplicated data with a data length conforming to a first length threshold, starting from next data of terminal data of the first specified data comprises: determining whether data length of second specified data in the to-be-deduplicated data is not lower than the first length threshold, wherein a data starting point of the second specified data is located after the terminal data of the first specified data; when the data length of the second specified data is not lower than the first length threshold, acquiring the to-be-duplicated data in the to-be-deduplicated data, starting from next data of the terminal data of the first specified data.

5. The method of claim 4, wherein, The method further comprises: when the data length of the second specified data is lower than the first length threshold, determining whether first local data in the first specified data is identical to second local data in the deduplicated local data; determining whether there is repeated data between the to-be-deduplicated data and the deduplicated local data based on whether the first local data is identical to the second local data; The data start point of the first partial data is the same as the data start point of the first specified data, the data start point of the second partial data is the same as the start data of the deduplicated partial data, and the data length of the first partial data and the deduplicated partial data meets the first length threshold.

6. The method of claim 3, wherein, The data deduplication processing of the to-be-deduplicated data includes: The deduplicated partial data having the same data as the to-be-duplicated data is determined as target partial data; It is determined that there is target repeated data between the to-be-deduplicated data and the target partial data, which starts from the data start point of the to-be-duplicated data; The data deduplication processing of the to-be-deduplicated data is performed on the target repeated data.

7. The method of claim 6, wherein, After the data deduplication processing of the to-be-deduplicated data, the method further includes: Based on the target repeated data, the to-be-deduplicated data is updated; the updated to-be-deduplicated data starts from the data start point of the target repeated data; The target repeated data is updated as the first specified data to obtain updated first specified data.

8. The method of claim 7, wherein, The method further includes: When there is no repeated data between the to-be-duplicated data and the deduplicated partial data, the first specified data is updated as the deduplicated partial data; The to-be-deduplicated data is re-determined; the data start point of the re-determined to-be-deduplicated data is located after the end data of the first specified data; The first specified data is updated for the re-determined to-be-deduplicated data; the data start point of the updated first specified data is located after the end data of the first specified data.

9. The method of claim 8, wherein, The method further includes: The to-be-duplicated data having a data length meeting the first length threshold is obtained in the re-determined to-be-deduplicated data, starting from the next data of the end data of the updated first specified data; It is determined whether there is repeated data between the re-determined to-be-deduplicated data and the deduplicated partial data according to whether there is the same data between the re-determined to-be-deduplicated data and the deduplicated partial data; If there is repeated data, the data deduplication processing is performed on the re-determined to-be-deduplicated data.

10. The method of claim 8, wherein, The updating of the first specified data includes: Data compression is performed starting from the data start point of the re-determined to-be-deduplicated data to obtain first compressed data having a data length meeting a second length threshold; The data corresponding to the first compressed data in the target data is determined as the updated first specified data.

11. The method of claim 7, wherein, In the data deduplication processing of the to-be-deduplicated data, the method further includes: The second compressed data corresponding to the first specified data is determined; Based on the target repeated data, the second compressed data is updated; the end data of the updated second compressed data in the target data is located before the start data of the target repeated data; The data corresponding to the updated second compressed data in the target data is determined as the deduplicated partial data.

12. An electronic device comprising a memory, a processor, and a computer program stored on the memory, the processor, when executing the computer program, implementing the method of any one of claims 1-11.

13. A computer readable storage medium having stored therein a computer program, the computer program, when executed by a processor, implementing the method of any one of claims 1-11.

Images