A method for deduplication of data

Abstract

The present application discloses a method for data deduplication. The present application can read the data to be detected line by line in sequence; calculate the MD5 value of the target row data currently read and use the hash algorithm to calculate the target block number of the index file block of the target row data. Determine whether there is an index file block with the target block number; if so, determine whether there is a secondary jump table index file block corresponding to the index file block with the target block number; if not, retrieve whether the index file block with the target block number has the MD5 value of the target row data; if the index file block with the target block number has the MD5 value of the target row data, delete the target row data. Because the storage space of the MD5 value of each row of data is smaller than the storage space of each row of data, when deduplication is performed, only the MD5 value of each row of data is read for data comparison to determine whether the target data exists, without reading each row of data, which reduces the operating pressure of the server, saves server resources, and improves the speed of data deduplication.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and more specifically, to a method for deduplication of data. Background Art

[0002] In the information age, big data has become an important strategic resource that enterprises and society pay attention to, and data management has become the core competitiveness of enterprises or individuals. In order to fully utilize the opportunities brought by big data, it is often necessary to do big data analysis to explore the value of big data. In order to efficiently explore the value of big data, sometimes it is necessary to perform data deduplication processing on some redundant big data to obtain high-quality big data for big data analysis. For example, sometimes it is necessary to deduplicate the line data of large files with a capacity of hundreds of GB or several TB, in which the line data in these large files is very long. If it is based on manual identification and comparison one by one, it will be a huge and tedious project. Therefore, people generally use some data deduplication technical means to achieve rapid deduplication.

[0003] The commonly used deduplication technology is to store all the large file row data in the processor's memory for data deduplication processing. This data deduplication processing method consumes a lot of server resources and has low processing efficiency.

[0004] Therefore, there is an urgent need to provide a data processing method that can quickly deduplicate data to solve the problem of occupying a large number of server resources during data deduplication processing. Summary of the invention

[0005] In view of this, the present application provides a method for deduplication of data, which is used to solve the problem that data deduplication processing occupies a lot of server resources.

[0006] A method for deduplicating data, comprising:

[0007] Read the data to be tested line by line in sequence;

[0008] Calculate the MD5 value of the target row data currently being read;

[0009] Calculate the target block number of the index file block of the target row data by using a hash algorithm according to the MD5 value of the target row data;

[0010] Determine whether there is an index file block with the target block number, the index file block is used to store the MD5 value of each row of data, wherein the MD5 values ​​in each index file block are sorted in ascending or descending order;

[0011] If yes, determine whether there is a secondary skip table index file block corresponding to the index file block of the target block number, wherein the secondary skip table index file block contains at least one secondary skip table index, and the secondary skip table index records the row number where the partial MD5 value in the corresponding index file block is located;

[0012] If not, search the index file block of the target block number to see if the MD5 value of the target row data exists;

[0013] If the index file block of the target row data contains the MD5 value of the target row data, the target row data is deleted.

[0014] Preferably, the method further comprises:

[0015] If there is a secondary skip table index file block corresponding to the index file block of the target block number, determining whether there is a secondary skip table index corresponding to the MD5 value of the target row data in the secondary skip table index file block;

[0016] If there is a secondary skip table index corresponding to the MD5 value of the target row data, delete the target row data;

[0017] If there is no secondary skip table index corresponding to the MD5 value of the target row data, searching whether there is the MD5 value of the target row data in the index file block of the target block number based on the secondary skip table index in the secondary skip table index file block;

[0018] If the MD5 value of the target row data exists in the index file block of the target block number, the target row data is deleted.

[0019] Preferably, the method further comprises:

[0020] If there is no index file block corresponding to the target block number, create an index file block corresponding to the target block number;

[0021] The MD5 value of the target row data is stored in the created index file block of the target block number.

[0022] Preferably, the method further comprises:

[0023] If the MD5 value of the target row data does not exist in the index file block of the target block number, insert the MD5 value of the target row data into the index file block of the target block number;

[0024] Update the order of the MD5 values ​​in the index file block of the target block number.

[0025] Preferably, after inserting the MD5 value of the target row data into the index file block of the target block number, the method further comprises:

[0026] Determine whether the size of the index file block of the target block number exceeds a preset first threshold;

[0027] If it exceeds, a secondary skip list index file block is created for the index file block of the target block number;

[0028] In the secondary skip list file block of the index file block of the target block number, a secondary skip list index of a part of the MD5 value stored in the index file block of the target block number is created.

[0029] Preferably, the step of creating a secondary skip table index of a portion of the MD5 values ​​stored in the index file block of the target block number includes:

[0030] For each MD5 value in the index file block of the target block number, refer to a preset extraction rule to extract the MD5 value and the row number corresponding to the extracted MD5 value in the index file block of the target block number;

[0031] Based on the extracted MD5 value and its corresponding row number, a secondary skip list index about the extracted MD5 value is created.

[0032] Preferably, the retrieving whether the MD5 value of the target row data exists in the index file block with the target block number based on the secondary skip table index in the secondary skip table index file block comprises:

[0033] In the secondary skip table index of the secondary skip table index file block, determine the secondary skip table index corresponding to the MD5 value closest to the MD5 value of the target row data and smaller than the MD5 value of the target row data as the first target secondary skip table index, and the secondary skip table index corresponding to the MD5 value closest to the MD5 value of the target row data and larger than the MD5 value of the target row data as the second target secondary skip table index;

[0034] According to the row numbers corresponding to the MD5 values ​​in the first and second target secondary skip table indexes, the MD5 values ​​between the row numbers corresponding to the MD5 values ​​in the first and second target secondary skip table indexes in the index file block of the target block number are retrieved to determine whether the MD5 value of the target row data exists.

[0035] Preferably, the method further comprises:

[0036] If the MD5 value of the target row data does not exist in the index file block of the target block number, insert the MD5 value of the target row data into the index file block of the target block number;

[0037] Update the sorting of each MD5 value in the index file block of the target block number;

[0038] Based on the row numbers of each updated MD5 value in the index file block of the target block number, the secondary skip list index in the secondary skip list index file block corresponding to the index file block of the target block number is updated.

[0039] Preferably, after the updating of the secondary skip table index in the secondary skip table index file block corresponding to the index file block of the target block number, the method further includes:

[0040] Determining whether the size of the secondary skip list index file block exceeds a preset second threshold;

[0041] If yes, the secondary skip list index file block is stored in the local disk.

[0042] Preferably, after obtaining the target block number of the index file block of the target row data, the method further comprises:

[0043] Determine whether there is a data file block corresponding to the target block number;

[0044] If not, create a data file block, define the block number of the data file block as the target block number, and store the target row data into the data file block corresponding to the target block number;

[0045] If so, the target row data is stored in the data file block corresponding to the target block number.

[0046] It can be seen from the above technical scheme that the data deduplication method of the present application can read the data to be detected line by line in sequence; can calculate the MD5 value of the target line data currently read; and use the hash algorithm to calculate the target block number of the index file block of the target line data. After obtaining the target block number, it can be determined whether there is an index file block of the target block number, the index file block is used to store the MD5 value of each line of data, wherein the MD5 values ​​in each index file block are sorted in ascending or descending order; if so, it is determined whether there is a secondary skip table index file block corresponding to the index file block of the target block number, the secondary skip table index block contains at least one secondary skip table index, and the secondary skip table index records the row number where the MD5 value of part of the corresponding index file block is located; if not, then search whether the index file block of the target block number has the MD5 value of the target line data; if the index file block of the target block number has the MD5 value of the target line data, then delete the target line data.

[0047] Because the storage space of the MD5 value of each row of data is smaller than the storage space of each row of data, when deduplicating data, only the MD5 value of each row of data needs to be read for data comparison to determine whether the target data exists. Compared with the prior art, the present application does not need to read each row of data to be detected for data comparison, which reduces the operating pressure of the server, saves server resources, and improves the speed of data deduplication. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0049] Figure 1-Figure 8 A flowchart of a method for implementing data deduplication provided in an embodiment of the present application;

[0050] Fig. 9 A schematic diagram of the structure of a data deduplication device exemplified in an embodiment of the present application;

[0051] Fig.10 A hardware structure block diagram of a data deduplication device disclosed in an embodiment of the present application. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0053] In the era of big data, it is often necessary to deduplicate some data, but many existing deduplication methods occupy more server resources when deduplicating, resulting in high operating pressure on the server. In order to solve this problem, the present application provides a method for deduplicating data, which can be used to solve the problem of occupying more server resources when deduplicating data.

[0054] The method can be applied to any device that can implement data deduplication. The device that reads the data to be detected can be a terminal with data processing capability, such as a tablet computer, a mobile phone, a digital television, etc.

[0055] Combine the following Figure 1 , introduces the process of the data deduplication method provided in the embodiment of the present application, which may include the following steps:

[0056] Step S101, reading the data to be detected line by line in sequence.

[0057] Specifically, the data to be detected can be data in any format. The data to be detected can be read line by line in units of lines. The purpose of reading the data to be detected line by line in units of lines is to divide the data to be detected into multiple lines of data so as to calculate the MD5 value of each line of data.

[0058] Step S102, calculating the MD5 value of the target row data currently being read.

[0059] Specifically, after reading a row of data, the information digest algorithm may be used to calculate the MD5 value of the target row of data currently being read.

[0060] Among them, MD5 stands for Message-Digest Algorithm 5, which is a widely used cryptographic hash function. Through the information digest algorithm calculation, a 128-bit (16-byte) hash value can be generated, commonly known as the MD5 value. The MD5 value can be used to ensure the integrity and consistency of information transmission. The MD5 value is equivalent to the ID of a file or a row of data. If a file or a row of data has been modified, the MD5 value corresponding to the file or data will also change.

[0061] Therefore, the MD5 value of each row of data in the data to be detected can be calculated as the ID of each row of data, so as to be used as a basis for comparing whether the data is repeated when deduplicating the data.

[0062] Step S103, calculating by using a hash algorithm according to the MD5 value of the target row data to obtain a target block number of the index file block of the target row data.

[0063] Specifically, after obtaining the MD5 value of the target row data, the number of index file blocks related to the data to be detected can be determined by using a hash algorithm to calculate based on the size of the data to be detected as a parameter. Therefore, a hash value can be obtained based on the MD5 value of the target row data using a hash algorithm. The obtained hash value can be defined as the target block number of the index file block of the target row data, and the target block number can be used to mark the index file block storing the MD5 value of the target row data.

[0064] Step S104, determining whether there is an index file block corresponding to the target block number, if so, executing step S105.

[0065] Specifically, after obtaining the target block number, it can be determined whether there is an index file block corresponding to the target block number in the local disk or server. The index file block can be used to store the MD5 value of each row of data, wherein the MD5 values ​​in each index file block are sorted in ascending or descending order. If there is an index file block corresponding to the target block number in the local disk or server, step S105 is executed.

[0066] Step S105, determining whether there is a secondary skip list index file block corresponding to the index file block of the target block number, if not, executing step S106.

[0067] Specifically, if there is an index file block of the target block number, it can be further determined whether there is a secondary skip table index file block corresponding to the index file block. The secondary skip table index file block contains at least one secondary skip table index, and the secondary skip table index records the row number where the partial MD5 value in the corresponding index file block is located. If there is no secondary skip table index file block corresponding to the index file block of the target block number, it means that there is no secondary skip table index corresponding to the MD5 value of the target row data in the secondary skip table index file block corresponding to the index file block of the target block number, and the index file block of the target block number should be further retrieved to determine whether there is an MD5 value of the target row data. Therefore, if there is no secondary skip table index file block corresponding to the index file block of the target block number, step S106 is executed.

[0068] Step S106, determining whether the index file block of the target block number has the MD5 value of the target row data, if so, executing step S107.

[0069] Specifically, it is determined whether the index file block of the target block number has the MD5 value of the target row data, so as to quickly determine whether the target row data already exists. If the index file block of the target block number has the MD5 value of the target row data, step S107 is executed.

[0070] Step S107, deleting the target row data.

[0071] Specifically, if the MD5 value of the target row data exists in the index file block of the target block number, it means that the row data already exists in the index file block of the target block number and the target row data is duplicate data, then the target row data is deleted.

[0072] It can be seen from the above technical solution that the present application can determine whether the target row data exists by querying whether the MD5 value of the target row data exists in the index file block of the target block number. Since the storage space of the MD5 value of a row of data is smaller than the storage space of a row of data. When deduplicating data, the embodiment of the present application only reads the MD5 values ​​in the index file block where the MD5 value of the target row data is located to perform data comparison. Compared with the prior art, the embodiment of the present application does not need to read each row of data in the data to be detected for data comparison, which reduces the operating pressure of the server, saves server resources, and improves the speed of data deduplication.

[0073] Considering that in actual application, there may be a secondary skip list index file block corresponding to the index file block of the target block number, the process of data deduplication processing will also be different. In some embodiments of the present application, the process of data deduplication when there is a secondary skip list index file block corresponding to the index file block of the target block number is introduced.

[0074] Next, combine Figure 2 The data deduplication process provided in the embodiment of the present application is introduced, and the process may include the following steps:

[0075] Step S101, reading the data to be detected line by line in sequence.

[0076] Step S102, calculating the MD5 value of the target row data currently being read.

[0077] Step S103, calculating by using a hash algorithm according to the MD5 value of the target row data to obtain a target block number of the index file block of the target row data.

[0078] Step S104, determining whether there is an index file block corresponding to the target block number, if so, executing step S105.

[0079] Step S105, determining whether there is a secondary skip list index file block corresponding to the index file block of the target block number, if not, executing step S106; if yes, executing step S108.

[0080] Step S106, determining whether the index file block of the target block number has the MD5 value of the target row data, if so, executing step S107.

[0081] Step S107, deleting the target row data.

[0082] Among them, steps S101-S107 correspond one-to-one to steps S101-S107 in the above embodiment. Please refer to the above introduction for details, which will not be repeated here.

[0083] Step S108, determining whether there is a secondary skip table index corresponding to the MD5 value of the target row data in the secondary skip table index file block corresponding to the index file block of the target block number, if so, executing step S107; if not, executing step S109.

[0084] Specifically, if there is a secondary skip table index file block corresponding to the index file block of the target block number, it can be further determined whether there is a secondary skip table index corresponding to the MD5 value of the target row data in the secondary skip table file block. If so, it means that the target row data already exists, and step S107 is executed. If not, it means that there is no secondary skip table index corresponding to the MD5 value of the target row data in the secondary skip table index file block corresponding to the index file block of the target block number, and the following step S109 process can be executed.

[0085] Step S109, based on the secondary skip list index in the secondary skip list index file block, search whether the MD5 value of the target row data exists in the index file block of the target block number, if so, execute step S107.

[0086] Specifically, if there is no secondary skip table index corresponding to the MD5 value of the target row data in the secondary skip table index file block of the index file block of the target block number, it is possible to retrieve whether there is the MD5 value of the target row data in the index file block of the target block number based on the secondary skip table index of the secondary skip table index file block. If so, it means that the target row data already exists, and step S107 is executed.

[0087] In the embodiment of the present application, when there is an index file block of the target block number, it is possible to further determine whether there is a secondary skip table index file block of the index file block of the target block number to quickly determine whether there is the target row data. If the secondary skip table index of the MD5 value of the target data just exists in the secondary skip table index file block corresponding to the index file block of the target block number, it is not necessary to retrieve the index file block of the target block number, and it is also possible to determine that the target row data exists. If the secondary skip table index of the MD5 value of the target data does not exist in the secondary skip table index file block of the index file block of the target block number, the index file block of the target block number is retrieved according to the secondary skip table index in the secondary skip table index file block. Through the role of the secondary skip table index, retrieval time can be saved, the operating pressure of the server is reduced, and the efficiency of data deduplication is improved.

[0088] In some embodiments of the present application, the process of searching for the MD5 value of the target row data in the index file block of the target block number based on the secondary skip table index in the secondary skip table index file block in step S109 is introduced. The process may include the following steps:

[0089] Step S1, determining a first target secondary skip table index and a second target secondary skip table index.

[0090] Specifically, in the secondary skip table index of the secondary skip table index file block, the secondary skip table index corresponding to the MD5 value that is closest to the MD5 value of the target row data and smaller than the MD5 value of the target row data can be determined as the first target secondary skip table index, and the secondary skip table index corresponding to the MD5 value that is closest to the MD5 value of the target row data and larger than the MD5 value of the target row data can be determined as the second target secondary skip table index.

[0091] Step S2: According to the first target secondary skip table index and the second target secondary skip table index, the index file block of the target block number is retrieved to determine whether the MD5 value of the target row data exists.

[0092] Specifically, after determining the first target secondary skip table index and the second target secondary skip table index, the MD5 value between the row numbers corresponding to the MD5 values ​​in the first and second target secondary skip table indexes in the index file block of the target block number can be retrieved according to the row numbers corresponding to the MD5 values ​​in the first and second target secondary skip table indexes to determine whether the MD5 value of the target row data exists.

[0093] In an embodiment of the present application, when there is no secondary skip table index corresponding to the MD5 value of the target row data in the secondary skip table index file block of the index file block of the target block number, the index file block of the target block number can be retrieved according to the first target secondary skip table index and the second target secondary skip table index to determine whether there is an MD5 value of the target row data. According to the sorting rules of each MD5 value in the index file block of the target block number, if the index file block of the target block number has the MD5 value of the target row data, it will only exist between the row number of the first target secondary index and the target secondary index. Therefore, based on the first and second target secondary skip table indexes, the scope of retrieval can be narrowed, the operating pressure of the server can be reduced, and the retrieval time can also be shortened, thereby improving the efficiency of data deduplication.

[0094] Taking into account that in actual application, after obtaining the target block number, the local disk or server may not have the index file block of the target block number, resulting in the inability to store the MD5 value of the target row data, in order to solve this problem, some embodiments of the present application may, after the above-mentioned step S103, add a process of creating an index file block corresponding to the target block number to store the MD5 value of the target row data.

[0095] Next, combine Figure 3The process of creating the index file block of the target block number and storing the MD5 value of the target row data is introduced. The process may include the following steps:

[0096] Step S101, reading the data to be detected line by line in sequence.

[0097] Step S102, calculating the MD5 value of the target row data currently being read.

[0098] Step S103, calculating by using a hash algorithm according to the MD5 value of the target row data to obtain a target block number of the index file block of the target row data.

[0099] Step S104, determine whether there is an index file block corresponding to the target block number. If so, execute step S105; if not, it means that there is no index file block with the target block number to store the MD5 value of the target row data, and an index file block corresponding to the target block number must be created, so execute step S110.

[0100] Step S105, determining whether there is a secondary skip list index file block corresponding to the index file block of the target block number, if not, executing step S106; if yes, executing step S108.

[0101] Step S106, determining whether the index file block of the target block number has the MD5 value of the target row data, if so, executing step S107.

[0102] Step S107, deleting the target row data.

[0103] Among them, steps S101-S107 correspond one-to-one to steps S101-S107 in the above embodiment. Please refer to the above introduction for details, which will not be repeated here.

[0104] Step S110: creating an index file block corresponding to the target block number.

[0105] Specifically, if there is no index file block corresponding to the target block number, an index file block may be created, and the target block number may be defined as the block number of the created index file block, so as to be used to store the MD5 value of the target row data.

[0106] Step S111, storing the MD5 value of the target row data into the index file block of the target block number.

[0107] Specifically, after the index file block corresponding to the target block number is created, the MD5 value of the target row data may be stored in the created index file block of the target block number.

[0108] In the embodiment of the present application, when there is no index file block corresponding to the target block number, a new index file block can be created to store the MD5 value of the target row data.

[0109] Taking into account that in actual application, although there is an index file block of the target block number, the MD5 value of the target row data may not exist in the index file block. Therefore, the embodiment of the present application adds a process of storing the MD5 value of the target data into the index file block of the target block number after the above-mentioned step S106.

[0110] Next, combine Figure 4 The following describes a process of storing the MD5 value of the target data into the index file block of the target block number when the index file block of the target block number already exists but the index file block does not have the MD5 value of the target row data. The process may include the following steps:

[0111] Step S101, reading the data to be detected line by line in sequence.

[0112] Step S102, calculating the MD5 value of the target row data currently being read.

[0113] Step S103, calculating by using a hash algorithm according to the MD5 value of the target row data to obtain a target block number of the index file block of the target row data.

[0114] Step S104, determine whether there is an index file block corresponding to the target block number. If so, execute step S105; if not, it means that there is no index file block with the target block number to store the MD5 value of the target row data, and an index file block corresponding to the target block number must be created, so execute step S110.

[0115] Step S105, determining whether there is a secondary skip list index file block corresponding to the index file block of the target block number, if not, executing step S106; if yes, executing step S108.

[0116] Step S106, determining whether the index file block of the target block number has the MD5 value of the target row data, if yes, executing step S107, if no, executing step S112.

[0117] Step S107, deleting the target row data.

[0118] Among them, steps S101-S107 correspond one-to-one to steps S101-S107 in the above embodiment. Please refer to the above introduction for details, which will not be repeated here.

[0119] Step S112: insert the MD5 value of the target row data into the index file block of the target block number.

[0120] Specifically, if the MD5 value of the target row data does not exist in the index file block of the target block number, the MD5 value of the target row data can be inserted into the index file block of the target block number. After the MD5 value of the target row data is inserted, the sorting of each MD5 value in the index file block of the target block number is updated according to the sorting rules of each MD5 value in the index file block of the target block number.

[0121] In an embodiment of the present application, when an index file block with the target block number already exists and the index file block does not contain the MD5 value of the target row data, the MD5 value of the target row data can be stored in the index file block with the target block number, and the order of the MD5 values ​​in the index file block can be updated for further processing.

[0122] Taking into account that in actual application, as the number of stored MD5 values ​​gradually increases, the size of the index file block storing the MD5 values ​​gradually increases. When performing data comparison, the time for traversing each MD5 value of an index file block will also increase accordingly. In order to quickly confirm whether a certain MD5 value exists in an index file block, the embodiment of the present application can add a process after the above-mentioned step S112, in which a secondary skip table index file block is created for the index file block when the size of the index file block exceeds a preset first threshold, and a secondary skip table index is created in the secondary skip table index file block regarding some MD5 values ​​in the index file block.

[0123] Next, combine Figure 5 The process of creating a secondary skip table index file block for the index file block of the target block number and creating a secondary skip table index of part of the MD5 values ​​in the index file of the target block number in the secondary skip table index file block may include the following steps:

[0124] Step S101, reading the data to be detected line by line in sequence.

[0125] Step S102, calculating the MD5 value of the target row data currently being read.

[0126] Step S103, calculating by using a hash algorithm according to the MD5 value of the target row data to obtain a target block number of the index file block of the target row data.

[0127] Step S104, determine whether there is an index file block corresponding to the target block number. If so, execute step S105; if not, it means that there is no index file block with the target block number to store the MD5 value of the target row data, and an index file block corresponding to the target block number must be created, so execute step S110.

[0128] Step S105, determining whether there is a secondary skip list index file block corresponding to the index file block of the target block number, if not, executing step S106; if yes, executing step S108.

[0129] Step S106, determining whether the index file block of the target block number has the MD5 value of the target row data, if yes, executing step S107, if not, executing step S112.

[0130] Step S107, deleting the target row data.

[0131] Step S112: insert the MD5 value of the target row data into the index file block of the target block number.

[0132] Among them, steps S101-S107 and step S112 correspond one-to-one to steps S101-S107 and step S112 in the above embodiment. Please refer to the above introduction for details, which will not be repeated here.

[0133] Step S113, determining whether the index file block size of the target block number exceeds a preset first threshold, if so, executing step S114.

[0134] Specifically, when the size of the index file of the target block number reaches a certain value, it is time-consuming to traverse the MD5 values ​​of the index file block when performing data deduplication, and it also increases the operating pressure of the server. Therefore, a first threshold value can be set in advance. After the MD5 value of the target row data is stored in the index file block of the target block number, it is determined whether the size of the index file block of the target block number exceeds the preset first threshold value. If so, execute step S114.

[0135] For example, the first threshold may be set to 10M, and when the size of the index file block of the target block number exceeds 10M, step S114 may be executed.

[0136] Step S114, creating a secondary skip list index file block for the index file block of the target block number, and creating a secondary skip list index of a part of the MD5 values ​​in the index file block of the target block number in the secondary skip list index file block.

[0137] Specifically, when the size of the index file block of the target block number exceeds a preset first threshold, a secondary skip table index file block may be created for the index file block of the target block number; and a secondary skip table index of a part of the MD5 values ​​stored in the index file block of the target block number may be created in the secondary skip table file block of the index file block of the target block number, so as to be used to quickly determine whether the target row data exists.

[0138] In an embodiment of the present application, after storing the MD5 value of the target row data into the index file block of the target block number, it is possible to determine whether to create a secondary skip list index file block and a secondary skip list index for the index file block of the target block number according to a preset first threshold, so as to be used for fast data deduplication. Based on the secondary skip list index file block and the secondary skip list index, when performing data deduplication, it is possible to preferentially query the secondary skip list index file block and the secondary skip list index to quickly determine whether the data is repeated, which can reduce the retrieval time and improve the data deduplication efficiency.

[0139] In some embodiments of the present application, for step S114 above, a process of creating a secondary skip list index file block for the index file block of the target block number and creating a secondary skip list index for some MD5 values in the index file block of the target block number is introduced. This process may include the following steps:

[0140] Step S1, for each MD5 value in the index file block of the target block number, referring to a preset extraction rule, extract the MD5 value and the row number corresponding to the extracted MD5 value in the index file block of the target block number.

[0141] Specifically, when the index file block of the target block number exceeds a preset first threshold, for each MD5 value in the index file block of the target block number, it is possible to refer to a preset extraction rule to extract the MD5 value and the row number corresponding to the extracted MD5 value in the index file block of the target block number.

[0142] For example, in each MD5 value of the index file block of the target block number, it is possible to set to extract one MD5 value and its corresponding row number every 100,000 rows, so as to create a secondary skip list index for the extracted MD5 value.

[0143] Step S2, based on the extracted MD5 value and its corresponding row number, create a secondary skip list index for the extracted MD5 value.

[0144] Specifically, after extracting the corresponding MD5 value and its corresponding row number according to the preset extraction rule, it is possible to create a secondary skip list index for the extracted MD5 value based on the extracted MD5 value and its corresponding row number. The secondary skip list index can be used for fast data deduplication.

[0145] Among them, the storage structure of the secondary skip list index may be <MD5 value, row number corresponding to the MD5 value in the index file block of the target block number>.

[0146] The embodiment of the present application introduces a process of creating a secondary skip table file block and a secondary skip table index for the index file block of the target block number. The created secondary skip table index can be used for fast data deduplication. The secondary skip table index records the MD5 value and its specific row number stored in the index file block of the target block number. Therefore, when deduplicating data, it is possible to quickly determine whether the target row data exists by querying the secondary skip table index. Even if it is impossible to know whether the target row data exists, the retrieval range of the index file block at the target block number can be further narrowed according to the secondary skip table index, thereby reducing the server operating pressure and retrieval time and improving the efficiency of data deduplication.

[0147] Considering that in actual application, when searching for the MD5 value of the target row data in the index file block of the target block number based on the secondary skip table index in the secondary skip table index file block, the index file block of the target block number may not have the MD5 value of the target row data. In view of this situation, in some embodiments of the present application, when searching for the MD5 value of the target row data in the index file block of the target block number based on the secondary skip table index in the secondary skip table index file block, if it is found that the index file block of the target block number does not have the MD5 value of the target row data, a process of storing the MD5 value of the target row data in the index file block corresponding to the target block number and updating the secondary skip table index in the secondary skip table index file block corresponding to the index file block of the target block number can be added.

[0148] Next, combine Figure 6 The process is introduced, which may include the following steps:

[0149] Step S101, reading the data to be detected line by line in sequence.

[0150] Step S102, calculating the MD5 value of the target row data currently being read.

[0151] Step S103, calculating by using a hash algorithm according to the MD5 value of the target row data to obtain a target block number of the index file block of the target row data.

[0152] Step S104, determining whether there is an index file block corresponding to the target block number, if so, executing step S105.

[0153] Step S105, determining whether there is a secondary skip list index file block corresponding to the index file block of the target block number, if not, executing step S106; if yes, executing step S108.

[0154] Step S106, determining whether the index file block of the target block number has the MD5 value of the target row data, if so, executing step S107.

[0155] Step S107, deleting the target row data.

[0156] Step S108, determining whether there is a secondary skip table index corresponding to the MD5 value of the target row data in the secondary skip table index file block corresponding to the index file block of the target block number, if so, executing step S107; if not, executing step S109.

[0157] Step S109, based on the secondary skip list index in the secondary skip list index file block, search whether the MD5 value of the target row data exists in the index file block of the target block number, if yes, execute step S107. If not, execute step S115.

[0158] Among them, steps S101-S109 correspond one-to-one to steps S101-S109 in the above embodiment. Please refer to the above introduction for details, which will not be repeated here.

[0159] Step S115, storing the MD5 value of the target row data into the index file block corresponding to the target block number, and updating the order of each MD5 value in the index file block of the target block number.

[0160] Specifically, if the MD5 value of the target row data does not exist in the index file block of the target block number, it means that the target row data is unique, then the MD5 value of the target row data is inserted into the index file block of the target block number, and according to the sorting rule of each MD5 value in the index file block of the target block number, the sorting of each MD5 value in the index file block of the target block number is updated.

[0161] Step S116, updating the secondary skip list index in the secondary skip list index file block corresponding to the index file block of the target block number.

[0162] Specifically, after the sorting of the MD5 values ​​of the index file block of the target block number is updated, the row numbers of the MD5 values ​​in the index file block will also change accordingly. The secondary skip table index in the secondary skip table index file block is created based on some MD5 values ​​and their corresponding row numbers in the index file block of the target block number. Therefore, after the sorting of the MD5 values ​​in the index file block of the index file block of the target block number is updated, it is also necessary to update the secondary skip table index corresponding to some MD5 values ​​in the index file block of the index file block of the target block number. Therefore, the secondary skip table index in the secondary skip table index file block corresponding to the index file block of the target block number can be updated based on the row numbers of the updated MD5 values ​​in the index file block of the target block number.

[0163] After inserting the MD5 value of the target row data into the corresponding index file block, the embodiment of the present application can update the secondary skip table index of each MD5 value of the index file block of the target block number based on the row number of each updated MD5 value of the index file block of the target block number, so that data confusion will not occur when further data deduplication is performed.

[0164] Taking into account that in actual application, the capacity of the commonly used CPU running memory buffer is limited. When performing data deduplication, the size of the data read exceeds the capacity of the CPU running memory buffer, and a CPU interrupt failure is likely to occur. When a CPU interrupt failure occurs, it will be time-consuming to rebuild the data reading process and read the secondary skip table index established before the data deduplication. Therefore, the embodiment of the present application can add a process of storing the secondary skip table index file block in the local disk after the above-mentioned step S116 when the size of the secondary skip table index file block is less than a preset second threshold.

[0165] Next, combine Figure 7 Introduce the process, which may include the following steps:

[0166] Step S101, reading the data to be detected line by line in sequence.

[0167] Step S102, calculating the MD5 value of the target row data currently being read.

[0168] Step S103, calculating by using a hash algorithm according to the MD5 value of the target row data to obtain a target block number of the index file block of the target row data.

[0169] Step S104, determining whether there is an index file block corresponding to the target block number, if so, executing step S105.

[0170] Step S105, determining whether there is a secondary skip list index file block corresponding to the index file block of the target block number, if not, executing step S106; if yes, executing step S108.

[0171] Step S106, determining whether the index file block of the target block number has the MD5 value of the target row data, if so, executing step S107.

[0172] Step S107, deleting the target row data.

[0173] Step S108, determining whether there is a secondary skip table index corresponding to the MD5 value of the target row data in the secondary skip table index file block corresponding to the index file block of the target block number, if so, executing step S107; if not, executing step S109.

[0174] Step S109, based on the secondary skip list index in the secondary skip list index file block, search whether the MD5 value of the target row data exists in the index file block of the target block number, if yes, execute step S107. If not, execute step S115.

[0175] Step S115, storing the MD5 value of the target row data into the index file block corresponding to the target block number, and updating the order of each MD5 value in the index file block of the target block number.

[0176] Step S116, updating the secondary skip list index in the secondary skip list index file block corresponding to the index file block of the target block number.

[0177] Among them, steps S101-S109, S115-S116 correspond one-to-one to steps S101-S109, S115-S116 in the above embodiment. Please refer to the above introduction for details, which will not be repeated here.

[0178] Step S117, determining whether the size of the secondary skip list index file block exceeds a preset second threshold, if so, executing step S118.

[0179] Specifically, a second threshold may be preset to determine whether the size of the secondary skip list index file block is less than the preset second threshold. When the size of the secondary skip list index file block exceeds the preset second threshold, step S118 may be executed.

[0180] For example, the second threshold may be set to 1M, and when the size of the secondary skip list index file block exceeds 1M, step S118 is executed.

[0181] Step S118, storing the secondary skip list index file block in a local disk.

[0182] Specifically, when the size of the secondary skip list index file block exceeds a preset second threshold, the secondary skip list index file block may be stored in a local disk.

[0183] In the embodiment of the present application, the secondary skip list index file block can be stored in a local disk so that when a CPU interruption failure occurs and the secondary skip list index of the secondary skip list index file block needs to be re-read, the secondary skip list index file block can be quickly obtained.

[0184] Taking into account that after data deduplication, some application scenarios require the output of deduplicated data. Therefore, the embodiment of the present application can add a process of storing the target row data into the data file block after obtaining the target block number of the index file block of the target row data. Subsequently, for confirmed duplicate data, it will be deleted from the corresponding data file block.

[0185] Next, combine Figure 8 Introducing the process, which can include the following steps:

[0186] Step S101, reading the data to be detected line by line in sequence.

[0187] Step S102, calculating the MD5 value of the target row data currently being read.

[0188] Step S103, calculating by using a hash algorithm according to the MD5 value of the target row data to obtain a target block number of the index file block of the target row data.

[0189] Step S104, determining whether there is an index file block corresponding to the target block number, if so, executing step S105.

[0190] Step S105, determining whether there is a secondary skip list index file block corresponding to the index file block of the target block number, if not, executing step S106; if yes, executing step S108.

[0191] Step S106, determining whether the index file block of the target block number has the MD5 value of the target row data, if so, executing step S107.

[0192] Step S107, deleting the target row data.

[0193] Specifically, in this step, the target row data that is confirmed to be duplicated is deleted from the corresponding data file block.

[0194] Step S108, determining whether there is a secondary skip table index corresponding to the MD5 value of the target row data in the secondary skip table index file block corresponding to the index file block of the target block number, if so, executing step S107; if not, executing step S109.

[0195] Step S109, based on the secondary skip list index in the secondary skip list index file block, search whether the MD5 value of the target row data exists in the index file block of the target block number, if yes, execute step S107. If not, execute step S115.

[0196] Step S110: creating an index file block corresponding to the target block number.

[0197] Step S111, storing the MD5 value of the target row data into the index file block of the target block number. Step S112, inserting the MD5 value of the target row data into the index file block of the target block number.

[0198] Step S113, determining whether the index file block size of the target block number exceeds a preset first threshold, if so, executing step S114.

[0199] Step S114, creating a secondary skip list index file block for the index file block of the target block number, and creating a secondary skip list index of a part of the MD5 values ​​in the index file block of the target block number in the secondary skip list index file block.

[0200] Step S115, storing the MD5 value of the target row data into the index file block corresponding to the target block number, and updating the order of each MD5 value in the index file block of the target block number.

[0201] Step S116, updating the secondary skip list index in the secondary skip list index file block corresponding to the index file block of the target block number.

[0202] Step S117, determining whether the size of the secondary skip list index file block exceeds a preset second threshold, if so, executing step S118.

[0203] Step S118, storing the secondary skip list index file block in a local disk.

[0204] Among them, steps S101-S1118 correspond one-to-one to steps S101-S118 in the above embodiment. Please refer to the above introduction for details, which will not be repeated here.

[0205] Step S119, determining whether there is a data file block corresponding to the target block number, if so, executing step S120; if not, executing step S121.

[0206] Step S120, creating a data file block, and defining the block number of the data file block as the target block number.

[0207] Step S121, storing the target row data into the data file block corresponding to the target block number.

[0208] In the embodiment of the present application, when there is no data file block that can be used to store the target row data, a data file block can be created based on the target block number to store the target row data. It is also possible to delete duplicate target row data from the corresponding data file block. Furthermore, when it is necessary to output deduplicated data, the deduplicated data is output for further processing.

[0209] The following is an introduction to the data deduplication device provided in an embodiment of the present application. The data deduplication device described below and the data deduplication method described above can be referenced to each other.

[0210] See also Fig. 9 , Fig. 9 A schematic diagram of the structure of a data deduplication processing device disclosed in an embodiment of the present application.

[0211] like Fig. 9 As shown, the data deduplication device may include:

[0212] The reading unit 110 is used to read the data to be detected line by line in sequence;

[0213] An MD5 value calculation unit 120 is used to calculate the MD5 value of the target row data currently being read;

[0214] The block number calculation unit 130 is used to calculate the target block number of the index file block of the target row data by using a hash algorithm according to the MD5 value of the target row data;

[0215] A first judgment unit 140 is used to judge whether there is an index file block with the target block number, wherein the index file block is used to store the MD5 value of each row of data, wherein the MD5 values ​​in each index file block are sorted in ascending or descending order;

[0216] A second judgment unit 150 is used to judge whether there is a secondary skip table index file block corresponding to the index file block of the target block number when there is an index file block of the target block number, wherein the secondary skip table index file block includes at least one secondary skip table index, and the secondary skip table index records the row number where a part of the MD5 value in the corresponding index file block is located;

[0217] The third judgment unit 160 is used to retrieve whether the index file block of the target block number has the MD5 value of the target row data when there is no index file block of the target block number;

[0218] The deleting unit 170 is configured to delete the target row data when the index file block of the target row data contains the MD5 value of the target row data.

[0219] Among them, the specific processing flow of each unit included in the above-mentioned data deduplication device can refer to the relevant introduction in the method part above, and will not be repeated here.

[0220] The data deduplication device provided in the embodiment of the present application can be applied to data deduplication devices, such as terminals: mobile phones, computers, etc. Optionally, Fig.10 The hardware structure diagram of the data deduplication processing device is shown in FIG. Fig.10 The hardware structure of the data deduplication device may include: at least one processor 1, at least one communication interface 2, at least one memory 3 and at least one communication bus 4.

[0221] In the embodiment of the present application, the number of the processor 1 , the communication interface 2 , the memory 3 , and the communication bus 4 is at least one, and the processor 1 , the communication interface 2 , and the memory 3 communicate with each other through the communication bus 4 .

[0222] The processor 1 may be a central processing unit CPU, or an application-specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present invention, etc.;

[0223] The memory 3 may include a high-speed RAM memory, and may also include a non-volatile memory, such as at least one disk memory;

[0224] The memory stores a program, and the processor can call the program stored in the memory, and the program is used to: implement each processing flow in the aforementioned terminal data deduplication solution.

[0225] An embodiment of the present application further provides a storage medium, which can store a program suitable for execution by a processor, wherein the program is used to: implement various processing flows of the aforementioned terminal in the data deduplication solution.

[0226] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0227] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0228] The above description of the disclosed embodiments enables professionals and technicians in the field to implement or use the present application. Various modifications to these embodiments will be apparent to professionals and technicians in the field, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. The various embodiments can be combined with each other. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for deduplication of data, characterized in that: include: Read the data to be tested line by line in sequence; Calculate the MD5 value of the target row data currently being read; Calculate the target block number of the index file block of the target row data by using a hash algorithm according to the MD5 value of the target row data; Determine whether there is an index file block with the target block number, the index file block is used to store the MD5 value of each row of data, wherein the MD5 values ​​in each index file block are sorted in ascending or descending order; If yes, determine whether there is a secondary skip table index file block corresponding to the index file block of the target block number, wherein the secondary skip table index file block contains at least one secondary skip table index, and the secondary skip table index records the row number where the partial MD5 value in the corresponding index file block is located; If there is no secondary skip list index file block corresponding to the index file block of the target block number, searching whether the index file block of the target block number has the MD5 value of the target row data; If the index file block of the target row data contains the MD5 value of the target row data, then the target row data is deleted; If there is a secondary skip table index file block corresponding to the index file block of the target block number, determining whether there is a secondary skip table index corresponding to the MD5 value of the target row data in the secondary skip table index file block; If there is a secondary skip table index corresponding to the MD5 value of the target row data, delete the target row data; If there is no secondary skip table index corresponding to the MD5 value of the target row data, searching whether there is the MD5 value of the target row data in the index file block of the target block number based on the secondary skip table index in the secondary skip table index file block; If the MD5 value of the target row data exists in the index file block of the target block number, the target row data is deleted; The retrieving, based on the secondary skip table index in the secondary skip table index file block, whether the MD5 value of the target row data exists in the index file block with the target block number, comprises: In the secondary skip table index of the secondary skip table index file block, determine the secondary skip table index corresponding to the MD5 value closest to the MD5 value of the target row data and smaller than the MD5 value of the target row data as the first target secondary skip table index, and the secondary skip table index corresponding to the MD5 value closest to the MD5 value of the target row data and larger than the MD5 value of the target row data as the second target secondary skip table index; According to the row numbers corresponding to the MD5 values ​​in the first and second target secondary skip table indexes, the MD5 values ​​between the row numbers corresponding to the MD5 values ​​in the first and second target secondary skip table indexes in the index file block of the target block number are retrieved to determine whether the MD5 value of the target row data exists.

2. The method according to claim 1, characterized in that The method further includes: If there is no index file block corresponding to the target block number, create an index file block corresponding to the target block number; The MD5 value of the target row data is stored in the created index file block of the target block number.

3. The method according to claim 1, characterized in that The method further includes: If the MD5 value of the target row data does not exist in the index file block of the target block number, insert the MD5 value of the target row data into the index file block of the target block number; Update the order of the MD5 values ​​in the index file block of the target block number.

4. The method according to claim 3, characterized in that: After inserting the MD5 value of the target row data into the index file block of the target block number, the method further includes: Determine whether the size of the index file block of the target block number exceeds a preset first threshold; If it exceeds, a secondary skip list index file block is created for the index file block of the target block number; In the secondary skip list file block of the index file block of the target block number, a secondary skip list index of a part of the MD5 value stored in the index file block of the target block number is created.

5. The method according to claim 4, characterized in that The step of creating a secondary skip table index of a portion of the MD5 values ​​stored in the index file block of the target block number includes: For each MD5 value in the index file block of the target block number, refer to a preset extraction rule to extract the MD5 value and the row number corresponding to the extracted MD5 value in the index file block of the target block number; Based on the extracted MD5 value and its corresponding row number, a secondary skip list index about the extracted MD5 value is created.

6. The method according to claim 1, characterized in that The method further includes: If the MD5 value of the target row data does not exist in the index file block of the target block number, insert the MD5 value of the target row data into the index file block of the target block number; Update the sorting of each MD5 value in the index file block of the target block number; Based on the row numbers of each updated MD5 value in the index file block of the target block number, the secondary skip list index in the secondary skip list index file block corresponding to the index file block of the target block number is updated.

7. The method according to claim 6, characterized in that After the updating of the secondary skip table index in the secondary skip table index file block corresponding to the index file block of the target block number, the method further includes: Determining whether the size of the secondary skip list index file block exceeds a preset second threshold; If yes, the secondary skip list index file block is stored in the local disk.

8. The method according to any one of claims 1 to 7, characterized in that After obtaining the target block number of the index file block of the target row data, the method further includes: Determine whether there is a data file block corresponding to the target block number; If not, create a data file block, define the block number of the data file block as the target block number, and store the target row data into the data file block corresponding to the target block number; If so, the target row data is stored in the data file block corresponding to the target block number.

Images