A cloud data deduplication method and device based on Reed-Solomon encoding

Abstract

The application discloses a cloud data decomposition deduplication method and device based on Reed-Solomon coding, wherein the method comprises the following steps: acquiring base data and offset data of to-be-uploaded data, then generating standard base data by a Reed-Solomon coding method according to the base data, generating label information according to the standard base data, generating a data deduplication request based on the offset data and the label information, sending the data deduplication request to a cloud server, and finally receiving a data deduplication response fed back by the cloud server based on the data deduplication request. The application can reduce data redundancy and improve the generalization ability of the base data.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of big data, and particularly relates to a cloud data decomposition deduplication method and device based on Reed-Solomon coding. BACKGROUND

[0002] With the advent of the era of big data, users use various terminals to generate massive and highly redundant data on the Internet. The emergence of cloud storage provides an effective solution for the efficient storage of these data. However, a large amount of redundant and repetitive data brings heavy storage and management overhead to cloud service providers. According to statistics, the current data duplication rate in the cloud is as high as about 60%. In order to upload these data to the cloud, users also need to pay a huge redundant communication overhead.

[0003] Cloud data deduplication technology is an effective means to solve the storage of redundant data, and the efficiency of the deduplication technology is affected by factors such as the location of deduplication occurrence and the size of the data processing unit. Data deduplication technology can be divided into file-level data deduplication and block-level data deduplication according to the size of the data processing unit. File-level data deduplication is an early popular service type, in which only one copy of each file is stored. If two or more files have the same hash value, they are considered to be the same. This method is simple to implement, but the deduplication granularity is relatively coarse and the performance is poor. Block-level data deduplication refers to the operation of dividing files into fixed-size blocks or variable-size blocks, and only one copy of each block is stored. Compared with file-level data deduplication, block-level data deduplication has a finer granularity, so it can achieve better deduplication efficiency.

[0004] In cross-user data deduplication technology, users only need to upload copies that do not exist in the cloud to improve communication and storage efficiency, but the deterministic response returned by this technology provides an edge channel for attackers. The latter can observe the existence of the requested data in the cloud by launching an edge channel attack. Specifically, an attacker can guess the content of the target data and initiate a deduplication request to the cloud. If the cloud service provider does not require further data upload, the attacker can know that the guessed data exists in the cloud, and the existence privacy is exposed. This method of privacy theft is generally feasible in real-world scenarios.

[0005] In order to solve the side channel attack problem, at present, a large number of research work has been carried out at home and abroad. One method is that the cloud service provider limits the attacker to obtain the existence privacy by setting the deduplication response threshold. When the number of copies of the data requested by the user for deduplication does not reach the threshold in the cloud, the cloud service provider requires the user to upload the data, so that the attacker cannot probe the data existence; on the contrary, when the number of copies of the data requested by the user for deduplication reaches the threshold in the cloud, the cloud service provider returns the existence response to block the further upload of the data. At this time, even if the attacker knows the existence of the data, it will not cause harm because the data has become popular data in the cloud. Although this kind of method can realize the data existence privacy protection to some extent, when the number of copies in the cloud does not reach the threshold, it still requires the user to upload a large amount of redundant data. On the one hand, it brings a large storage overhead to the cloud service provider, and on the other hand, it also makes the user pay additional communication overhead. Another method is to use confusion strategy to confuse the attacker. The basic idea of this method is to divide the file into several blocks, and assume that the user's name, ID number and other sensitive information exist in one or more blocks. For the cloud service provider, whether the sensitive information exists in the cloud or not, it needs to generate a confused response for the requester, so that the latter cannot distinguish the existence state through the deduplication response. However, in the real scene, the cloud service provider is difficult to distinguish the position of the sensitive block, and in the case of sensitive block hit, there is still a risk of privacy leakage. Although subsequent researchers have proposed some improved methods, these methods still have defects and are difficult to achieve complete confusion. In recent years, some researchers have proposed a generalized deduplication method and developed a secure deduplication framework. Under this framework, the original data is decomposed into a base and an offset. In order to achieve confusion, only the base is cross-user deduplicated, and the offset is deduplicated in the cloud. Since similar data can extract the same base with a certain probability, the attacker cannot judge the cloud existence of the target data according to the deduplication response of the base, and the side channel attack can be effectively resisted.

[0006] Based on the above analysis, it is found that the existing cloud data deduplication technology has the problem of side channel attack, which is easy to cause the leakage of user data existence privacy; the existing cloud data deduplication technology based on generalized deduplication can effectively resist the side channel attack, but the extracted base and offset have a lot of redundancy, which brings a large storage overhead to the cloud service provider on the one hand, and also makes the user pay additional communication overhead on the other hand. SUMMARY

[0007] The present application aims to at least solve one of the technical problems in the related art to some extent.

[0008] To this end, the purpose of the present application is to solve the base and offset redundancy problem in the existing cloud data deduplication technology based on generalized deduplication, and to propose a cloud data decomposition deduplication method based on Reed-Solomon coding.

[0009] Another object of the present application is to provide a cloud data decomposition deduplication device based on Reed-Solomon encoding.

[0010] To achieve the above object, the present application provides a cloud data decomposition deduplication method based on Reed-Solomon encoding, comprising:

[0011] obtaining base data and offset data of data to be uploaded;

[0012] generating standard base data by Reed-Solomon encoding method according to the base data;

[0013] generating tag information according to the standard base data;

[0014] generating a data deduplication request based on the offset data and the tag information, and sending the data deduplication request to a cloud server;

[0015] receiving a data deduplication response feedback by the cloud server based on the data deduplication request.

[0016] In a possible implementation, the generating standard base data by Reed-Solomon encoding method according to the base data comprises:

[0017] grouping the base data according to a preset length;

[0018] matching the base data in the index dictionary group by group to obtain a matching result, wherein the index dictionary is constructed in advance according to the Reed-Solomon encoding method;

[0019] generating the standard base data according to the matching result.

[0020] In a possible implementation, the generating tag information according to the standard base data comprises:

[0021] obtaining a serial number of each group of the standard base data based on the index dictionary;

[0022] chunking the standard base data according to a preset number of groups;

[0023] obtaining the tag information by a hash algorithm according to the serial number corresponding to each block of the standard base data.

[0024] In a possible implementation, the generating a data deduplication request based on the offset data and the tag information comprises:

[0025] chunking the offset data according to the preset number of groups;

[0026] compressing each block of the offset data according to a ZSTD compression algorithm to obtain compressed offset data;

[0027] generating a data deduplication request according to the compressed offset data and the tag information.

[0028] In a possible implementation, after receiving the data deduplication response fed back by the cloud server based on the data deduplication request, the method further includes:

[0029] sending target standard base data to the cloud server according to the data deduplication response.

[0030] To achieve the above object, another aspect of the present application provides a cloud data decomposition deduplication device based on Reed-Solomon encoding, comprising:

[0031] an obtaining module, configured to obtain base data and offset data of to-be-uploaded data;

[0032] a first generating module, configured to generate standard base data by a Reed-Solomon encoding method according to the base data;

[0033] a second generating module, configured to generate tag information according to the standard base data;

[0034] a third generating module, configured to generate a data deduplication request based on the offset data and the tag information, and send the data deduplication request to a cloud server;

[0035] a receiving module, configured to receive a data deduplication response fed back by the cloud server based on the data deduplication request.

[0036] In a possible implementation, the first generating module comprises:

[0037] a grouping unit, configured to group the base data according to a preset length;

[0038] a matching unit, configured to match the base data in an index dictionary by group to obtain a matching result, wherein the index dictionary is constructed in advance according to the Reed-Solomon encoding method;

[0039] a first generating unit, configured to generate the standard base data according to the matching result.

[0040] In a possible implementation, the second generating module comprises:

[0041] a first obtaining unit, configured to obtain a serial number of each group of the standard base data based on the index dictionary;

[0042] The first block unit is configured to block the standard base data according to a preset group number.

[0043] The second acquisition unit is configured to acquire the tag information through a hash algorithm according to the serial number corresponding to each block of the standard base data.

[0044] In a possible implementation, the third generation module comprises:

[0045] The second block unit is configured to block the offset data according to the preset group number.

[0046] The compression unit is configured to compress each block of the offset data according to a ZSTD compression algorithm to obtain compressed offset data.

[0047] The second generation unit is configured to generate a data deduplication request according to the compressed offset data and the tag information.

[0048] In a possible implementation, the cloud data decomposition deduplication device based on Reed-Solomon encoding further comprises:

[0049] The sending module is configured to send target standard base data to the cloud server according to the data deduplication response.

[0050] The beneficial effects of the present application are as follows:

[0051] In the embodiments of the present application, the base data and the offset data of the data to be uploaded are acquired, then the standard base data is generated according to the base data through the Reed-Solomon encoding method, the tag information is generated according to the standard base data, the data deduplication request is generated based on the offset data and the tag information, and the data deduplication request is sent to the cloud server, and finally the data deduplication response fed back by the cloud server based on the data deduplication request is received. The present application can reduce data redundancy and improve the generalization ability of the base data.

[0052] Additional aspects and advantages of the present application will be in part apparent and in part pointed out hereinafter. BRIEF DESCRIPTION OF DRAWINGS

[0053] The above and / or additional aspects and advantages of the present application will become apparent and be readily appreciated from the following description, including the accompanying drawings.

[0054] Figure 1 A flowchart of the cloud data decomposition deduplication method based on Reed-Solomon encoding according to the embodiments of the present application;

[0055] Figure 2A schematic diagram of base data extraction and offset data calculation according to an embodiment of the present application;

[0056] Figure 3 A schematic diagram of a framework of a cloud data decomposition deduplication method based on Reed-Solomon encoding according to an embodiment of the present application;

[0057] Figure 4 A schematic diagram of a structure of a cloud data decomposition deduplication device based on Reed-Solomon encoding according to an embodiment of the present application. DETAILED DESCRIPTION

[0058] It should be noted that the embodiments and features in the present application can be combined with each other without conflict. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0059] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work should belong to the scope of protection of the present application.

[0060] The present application is applied to a data hosting scenario, involving two entities of a user and a cloud service provider. In the data hosting scenario, the cloud service provider needs to provide data uploading and downloading services to the user, and bear data maintenance and security guarantee services. The user needs to pay data storage fees to the cloud service provider and bear data leakage risks. Therefore, for the cloud service provider, first of all, sufficient storage space is needed to meet the data storage needs of a certain scale of users. In addition, the cloud service provider should also have strong computing power to support redundant data management, repeated data deletion and data integrity download functions. Most importantly, it should have a perfect data security storage scheme to protect the integrity of the user's data and the privacy of sensitive data from being stolen and leaked.

[0061] To make the description more clear and understandable, before introducing the embodiments of the present application, Reed-Solomon coding is introduced first. Reed-Solomon coding (hereinafter referred to as RS coding) is a kind of error correction code defined in Galois field, which is an important theoretical basis of RS coding. In the Galois field, algebraic operation has closure, i.e. the result of algebraic operation is within the field, and there is no data overflow problem, so the Galois field is also called finite field. In the Galois field, addition is equivalent to XOR operation, multiplication is equivalent to logical AND operation, and negative number is the same as positive number. As an example, a two-dimensional code is a typical RS coding result, which uses GF(256). In the GF(256) field, there are numbers from 0 to 255, and if the calculation result exceeds this range, the modulo operation is continued on the calculation result until the result is within the range of 0 to 255. In order to reduce the amount of algebraic calculation, the generation value required for operation in GF(256) has been generated and saved, which can be directly queried in the table during use, without repeated calculation. Experiments show that when the data operation amount is very large, the time of table lookup is much less than the time of repeatedly calculating the value.

[0062] RS coding will rearrange and calculate the stream data to be coded, and its formula is as follows:

[0063] RS(x) = M(x) + P(x) (1)

[0064] Wherein, M(x) and P(x) represent the original data polynomial and the error correction data polynomial respectively, and their calculation formulas are as follows:

[0065] M(x) = m k-1 x n-1 +m k-2 x n-2 +…+m0x t (2)

[0066] P(x) = P t-1 x t-1 +…+P1x 1 +p0x 0 (3)

[0067] Wherein, k represents the length of the original data, t represents the length of the error correction data, n represents the RS coding length and satisfies n = k + t, and for any m i ,i∈[0,k-1] represents the data value of the k original data, and p i ,i∈[0,t-1] represents the data value of the t error correction data.

[0068] The process of RS encoding includes the construction of message polynomial and generator polynomial, and then the calculation of error correction data using the two polynomials. The decoding principle is relatively simple, and the error position can be found and the correct value can be recovered according to the encoding process and the principle of matrix operation. Next, the RS encoding process will be described in detail:

[0069] Message polynomial: The message polynomial uses the data codeword as the coefficient, for example, the decimal values of the data: 00100000, 01011011, 00001011 are 32, 91, 11, and then the message polynomial is:

[0070] M(x) = 32x 2 + 91x + 11

[0071] Generator polynomial: The generator polynomial g(x) is given by the formula:

[0072]

[0073] Using GF(256), where α takes the value of 2, and t represents the number of error correction positions.

[0074] Next, taking t = 2 as an example, the calculation process of the generator polynomial is introduced:

[0075] g(x) = (x - α 0 )*(x - α 1 ) (5)

[0076] 1) Convert the variable coefficient to α representation, and the table shows that α 0 = 1, so the polynomial can be written as

[0077] g(x) = (α 0 x - α 0 )*(α 0 x - α 1 ) (6)

[0078] 2) Expand the polynomial and merge the coefficients

[0079] g(x) = α 0 x 2 -(α 1 + α 0 )x 1 + α 1 x 0 (7)

[0080] 3) In the Galois field, negative numbers are equal to positive numbers, that is, subtraction is equivalent to addition, so

[0081] g(x) = α 0 x 2 +(α 1 + α 0x 1 + α 1 x 0 (8)

[0082] 4) According to the table, α 0 = 1, α 1 = 2, then

[0083] g(x) = x 2 + 3x 1 + 2x 0 (9)

[0084] 5) According to the table again, α 25 = 3, the generating polynomial when n takes 2 is expressed as

[0085] g(x) = α 0 x 2 + α 25 x 1 + α 1 x 0 (10)

[0086] According to the above process, the generating polynomial is related to the value of t and is irrelevant to the original data code word.

[0087] Generating error correction code: starting from the highest term of the generating polynomial, repeating the operation, first multiplying the generating polynomial to make it the same as the highest term coefficient of the message polynomial, then performing XOR operation on the message polynomial and the transformed generating polynomial to eliminate the highest term of the message polynomial. Repeat several times, and the remaining coefficient after the completion of the result is the error correction code. Finally, the generated error correction code is converted to be added to the original information code.

[0088] Data recovery process: according to the encoding process, if no error occurs in the transmission process, it meets the requirement of RS(x) % g(x) = 0. If the remainder is not 0, it indicates that an error has occurred.

[0089]

[0090] wherein, l represents the number of correctable data values and satisfies t = 2l, Y i , i ∈ [0, l] represents the error position, e i , i ∈ [0, l] represents the original correct value.

[0091] Therefore, in the decoding process, only the received encoded data is replaced according to the calculation result of E(x), and finally the error correction data is deleted to recover the original data content.

[0092] According to the RS encoding and data recovery principle, within a limited range, a group of numerical errors can be recovered to the correct result. In simple terms, there is a corresponding relationship between the correct numerical sequence and the error numerical sequence. Using this principle, a group of numerical sequences is corresponded to a group of numerical sequences, that is, the similar data content described above can extract the same base.

[0093] Therefore, the corresponding relationship between each group of bases is calculated in advance, which can avoid repeated calculation each time the file is uploaded. Here we select the length of RS encoding as 4, that is, every four bases are a group, and multiple base groups can be converted into the same source, thereby further improving the generalization ability of the base.

[0094] In the index construction process, first, all permutation and combination results of 0000, 0001, 0010, 0011, 0101, 0100, 0111, and 0110 are calculated, and then the source code is calculated using the data recovery principle. Figure 2 The storage form of the related data is shown, which represents the corresponding results of the base source code and the original data. In order to facilitate description and understanding, in Figure 2 , we use {a, b, c, d, e, f} to represent {0010, 0011, 0100, 0101, 0110, 0111}, for example, the base with the value {0010001101000111} is encoded from the source code with the value {0010001101000101}. Similarly, the base with the value {0010001101000100} also has the source code value {0010 001101000101}. In this way, the two different bases {00100011 01000111} and {0010001101000100} are converted into the same source code {0010001101000101}. After the base is converted into the source code through the table structure shown in the figure, it is tested that the storage space required by the table structure shown in the figure for the corresponding relationship between the base before storage decoding and the base after decoding is 184,512B. In order to facilitate base extraction and data recovery, the user end and the cloud end both need to maintain this table structure data.

[0095] Next, the cloud data decomposition deduplication method based on Reed-Solomon encoding according to an embodiment of the present application will be described with reference to the accompanying drawings.

[0096] Figure 1 is a flowchart of the cloud data decomposition deduplication method based on Reed-Solomon encoding of an embodiment of the present application.

[0097] As shown in Figure 1 , the cloud data decomposition deduplication method based on Reed-Solomon encoding comprises:

[0098] Step S110: Obtain the base data and offset data of the data to be uploaded.

[0099] In this embodiment, the data to be uploaded can be obtained, converted byte by byte into standard ASCII binary form, and then padded with zeros. Then, each byte can be divided into high four bits and low four bits. The high four bits can be used as the base data corresponding to the data to be uploaded, and the low four bits can be used as the offset data corresponding to the data to be uploaded. For example, file F consists of {x1, x2, x3, ..., x...} n}, consisting of n bytes, for each byte All are based on base data b i and offset data d i Composition. Among them, the base data b i It consists of 4 bits, the offset data d i The remaining 4 bits then constitute the file F, which can be represented as:

[0100] F={(b1,d1),(b2,d2),(b3,d3),…,(b n ,d n )}

[0101] We can use B to represent the base data set of file F, and D to represent the offset data set of file F, that is:

[0102] B = {b1, b2, b3, ..., b} n}

[0103] D = {d1, d2, d3, ..., d} n}

[0104] like Figure 2 As shown, the base data set of file F is B = {0100,0110,0111,…,0010,…}, and the offset data set is D = {1101,0101,0011,…,0000,…}.

[0105] Understandably, according to ASCII encoding rules, the high four bits of a byte can be 0000, 0001, 0010, 0011, 0101, 0100, 0111, or 0110, representing eight possibilities. 0000 and 0001 correspond to less frequently used control characters and communication-specific characters, respectively, while the others correspond to common uppercase letters, lowercase letters, symbols, numbers, etc.

[0106] Step S120: Generate standard base data based on the base data using the Reed-Solomon encoding method.

[0107] In this embodiment, after obtaining the base data and offset data of the data to be uploaded, standard base data can be generated based on the base data using the Reed-Solomon encoding method. For example, as shown below... Figure 2 As shown, based on the data recovery principle of RS encoding, assuming set B represents potentially erroneous data results, data recovery can correct erroneous bits and restore them to their original values, i.e., standard base data. Therefore, for B = {b1, b2, b3, ..., b...} n The result of its data recovery can be represented as B. rs ={b1,b2 correct ,b3,…,b n}, where b2 correct This is the correct value after correction.

[0108] For a similar set B′={b1,b2′,b3,…,b n After data recovery, result B may be obtained. rs ′={b1,b2 correct ,b3,…,b n Both b2 and b2' can be converted back to b2 after data recovery. correct .

[0109] Step S130: Generate label information based on standard base data.

[0110] In this embodiment of the application, tag information can be generated based on standard base data. For example, as shown below... Figure 2 As shown, the standard base data can be divided into blocks, B = {B1, B2, B3, ..., B}. c}, where variable c represents the number of blocks, and then tag information is generated for each block, using tags. B express:

[0111] Step S140: Generate a data deduplication request based on offset data and tag information, and send the data deduplication request to the cloud server.

[0112] In this embodiment, offset data and tag information can be used to generate a data deduplication request, which is then sent to a cloud server. It is understood that after receiving the data deduplication request, the cloud server can perform a data deduplication operation based on the request, generate a data deduplication response corresponding to the request, and then send the response back.

[0113] S150 receives the data deduplication response from the cloud server based on the data deduplication request.

[0114] In the embodiments of the present application, the data deduplication response fed back by the cloud server based on the data deduplication request can be received. It should be noted that the cloud server can request the user for the missed base data block through the data deduplication response, and can also notify the user of the end of data uploading through the data deduplication response.

[0115] In the embodiments of the present application, the base data and the offset data of the data to be uploaded are obtained, then the standard base data is generated according to the base data through the Reed-Solomon encoding method, the tag information is generated according to the standard base data, the data deduplication request is generated based on the offset data and the tag information, the data deduplication request is sent to the cloud server, and finally the data deduplication response fed back by the cloud server based on the data deduplication request is received. The present application can reduce data redundancy and improve the generalization ability of the base data.

[0116] In a possible implementation, the standard base data is generated according to the base data through the Reed-Solomon encoding method, comprising:

[0117] grouping the base data according to a preset length;

[0118] matching the base data in the index dictionary group by group to obtain a matching result, wherein the index dictionary is constructed in advance according to the Reed-Solomon encoding method;

[0119] generating the standard base data according to the matching result.

[0120] The preset length is a preset base data grouping length, for example, it can be 4, that is, 4 base data are divided into a group.

[0121] In the embodiments of the present application, in order to avoid repeated calculation in each file uploading process, the base data can be grouped according to a preset length, and then the base data can be matched in the index dictionary group by group to obtain a matching result, and finally the standard base data is generated according to the matching result. The index dictionary is constructed in advance according to the Reed-Solomon encoding method. For example, first, all permutation and combination results of 0000, 0001, 0010, 0011, 0101, 0100, 0111 and 0110 are calculated, and then the source code, i.e. the standard base data, is calculated by using the data recovery principle, and finally the index dictionary is formed by the base data and the standard base data corresponding to the base data. The index dictionary can be as shown in Table 1, and {a, b, c, d, e, f} represents {0010, 0011, 0100, 0101, 0110, 0111}. For example, the base data with a value of {0010001101000111} (abcf) corresponds to the standard base data with a value of {0010001101000101} (abcd). Similarly, the base data with a value of {0010001101000100} (abcc) also corresponds to the standard base data with a value of {0010001101000101} (abcd). In this way, the two different base data {0010001101000111} and {0010001101000100} are converted into the same standard base data {0010001101000101}.

[0122]

[0123] In this way, the standard base data is generated based on the pre-constructed index dictionary, which can effectively avoid repeated calculation and reduce the calculation overhead.

[0124] It should be noted that, through testing, the required storage space of the table structure for storing the correspondence between the base data and the standard base data is 184,512 bytes. In addition, in order to facilitate base data extraction and data recovery, the user end and the cloud server both need to maintain the above-mentioned index dictionary.

[0125] It can be understood that after the index dictionary is constructed, each group of base data can be numbered. As shown in Table 1, the base data {0100, 0110, 0111, 0111} corresponds to the serial number 8, the base data {0110, 0000, 0110, 0010} corresponds to the serial number 976, and the base data {0100, 0100, 0011, 0010} corresponds to the serial number 472. Figure 2

[0126] In one possible implementation, the label information is generated according to the standard base data, including:

[0127] The serial number of each group of standard base data is obtained based on the index dictionary; ​

[0128] According to the preset group number, the standard base data is blocked;

[0129] According to the serial number corresponding to each block of standard base data, the tag information is obtained through a hash algorithm.

[0130] The preset group number can be a preset group number of standard base data contained in each block of standard base data.

[0131] In the embodiments of the present application, the serial number of each group of standard base data can be obtained based on the index dictionary, then the standard base data is blocked according to the preset group number, and finally the tag information is obtained through a hash algorithm according to the serial number corresponding to each block of standard base data. For example, as shown in Figure 2 , the serial number of each group of standard base data can be obtained based on the index dictionary, for example, {0100,0110,0111,0111} corresponds to the serial number 8, {0110,0000,0110,0010} corresponds to the serial number 976, and {0100,0100,0011,0010} corresponds to the serial number 472. Then, the standard base data is blocked according to the preset group number, and the preset group number can be 3. Then, the base data set B={B1,B2,B3,…,B c}, where the variable c represents the number of blocks, where B1={{0100,0110,0111,0111},{0110,0000,0110,0010},{0100,0100,0011,0010}} corresponds to the serial number {8,976,472}. Finally, the tag data is obtained through a hash algorithm according to the serial number corresponding to each block of standard base data, and is represented as tag B . For example, the serial number corresponding to B1 is generated as tag information through a hash algorithm, that is, tag B1 . In this way, by performing a hash operation on the serial number corresponding to each block of standard base data, the tag information can be generated, and the calculation overhead can be saved.

[0132] In one possible implementation, the data deduplication request is generated based on the offset data and the tag information, including:

[0133] According to the preset group number, the offset data is blocked;

[0134] According to the ZSTD compression algorithm, each block of offset data is compressed to obtain compressed offset data;

[0135] The data deduplication request is generated according to the compressed offset data and the tag information.

[0136] The preset group number is the same as the preset group number when the base data is blocked, and each block of offset data corresponds to each block of base data one by one.

[0137] In the embodiments of the present application, considering the problem of communication overhead, the offset data can be blocked according to the preset group number. Then each block of offset data can be compressed according to the ZSTD (Zstandard) compression algorithm, so as to obtain compressed offset data. For example, as shown in the figure, the compressed offset data set D = {D1, D2, D3, …, Dn} is obtained. Figure 2 c Finally, the data deduplication request can be generated according to the compressed offset data and the tag information, that is, the cloud server receives the compressed offset data and the tag data. In this way, by compressing each block of offset data by the ZSTD compression algorithm, the communication overhead can be effectively reduced.

[0138] It should be noted that the ZSTD compression algorithm is a lightweight dictionary compression algorithm, which is especially suitable for small-scale data samples. Its compression efficiency can be improved to about 80% after using an effective dictionary, and the compression rate is about 50% when the dictionary is invalid. In the future, machine learning algorithms can be used to introduce more efficient and adaptive compression methods.

[0139] In a possible implementation, after receiving the data deduplication response fed back by the cloud server based on the data deduplication request, the method further includes:

[0140] According to the data deduplication response, the target standard base data is sent to the cloud server.

[0141] The target standard base data is a base data block in the base data set B that is not hit by the cloud server.

[0142] In the embodiments of the present application, after receiving the data deduplication response fed back by the cloud server based on the data deduplication request, the content of the data deduplication response can be obtained. In the case where the data deduplication response is used to request the target standard base data from the user, the target standard base data can be sent to the cloud server. In this way, the target standard base data can be uploaded to the cloud server according to the data deduplication response, so as to ensure the integrity and availability of the data.

[0143] To make the description of the embodiments of the present application more complete, the following Figure 3 The cloud data decomposition deduplication method based on Reed-Solomon encoding is further described. Figure 3 The framework diagram of the cloud data decomposition deduplication method based on Reed-Solomon encoding according to the embodiments of the present application is shown in the figure. Figure 3

[0144] ​​User X wants to upload file F to the cloud server, first sends a data deduplication request to the cloud server through base extraction and offset calculation, including: obtaining the base data {B1, B2, B3, B4, B5, B6} and offset data {D1, D2, D3, D4, D5, D6} of file F, wherein {B1, B2, B3, B4, B5, B6} and {D1, D2, D3, D4, D5, D6} form file block {C1, C2, C3, C4, C5, C6} pair by pair, then generate standard base data by Reed-Solomon encoding method according to base data, and generate tag information according to standard base data Based on the offset data and the tag information, generate a data deduplication request, and send the data deduplication request to the cloud server, that is, upload {D1, D2, D3, D4, D5, D6} to the cloud server. After receiving the data deduplication request, the cloud server can create a file F storage table to store the offset data, because file F is a new file uploaded for the first time. At the same time, the cloud server can compare the tag information with the base data copy tag stored in the cloud server itself, and according to the comparison result, judge whether the user needs to upload the base data block that does not match in the base data, and then form a data deduplication response, and feed back the data deduplication response to the user terminal device. After receiving the data deduplication response fed back by the cloud server based on the data deduplication request, the user terminal device needs to upload {B1, B2, B3, B4, B5, B6} to the cloud server, because the cloud server does not have any base data of file F. After receiving all the base data of file F, the cloud server can store all the base data in its own storage, so that the cloud server has stored a complete file F.

[0145] After the user uploads file F to the cloud server, user Y uploads file F` to the cloud server. First, obtain the base data {B`1, B`2, B`3, B`4, B`5, B`6} and offset data {D`1, D`2, D`3, D`4, D`5, D`6} of file F`, and form file block {C`1, C`2, C`3, C`4, C`5, C`6}, and obtain the tag information in the same way as above And upload {D`1, D`2, D`3, D`4, D`5, D`6} to the cloud server. After receiving {D`1, D`2, D`3, D`4, D`5, D`6}, the cloud server also compares the tag information. Since file F` and file F have repeated tag of file block, it is considered that file F` and file F are similar files, and a storage table is not established for file F`, but all the offset data of file F` is stored in file F storage table. Therefore, when the following requirements are met, a storage table dicF is not established for the file, but the offset data is directly indexed in the similar file:

[0146] (1) For each block C in F′ j ′,j∈[1,c], Suppose there exists a block of basic data B. j ′≠B j Then dicF is updated to {B1, B2, B3, ..., B}. c B j ′}.

[0147] (2) For each block C in F′ j ′,j∈[1,c], suppose there exists a basis B in a block. j ′=B j Offset D j ′≠D j Then dicF[B] j The value of ] is updated to {D j D j ′}.

[0148] After the cloud server stores all offset data of file F' in the file F storage table, since D6' lacks a corresponding B6', it can generate a data deduplication response and send it back to the user's terminal device. Upon receiving the data deduplication response, the user's terminal device can upload B6' to the cloud server. Upon receiving B6', the cloud server stores the non-duplicated base data B6' and performs cloud-based deduplication on the offsets to obtain the final storage result.

[0149] To achieve the above embodiments, such as Figure 4 As shown, this embodiment also provides a cloud data decomposition and deduplication device 400 based on Reed-Solomon encoding. The device 400 includes: an acquisition module 410, a first generation module 420, a second generation module 430, a third generation module 440, and a receiving module 450.

[0150] The acquisition module 410 is used to acquire the base data and offset data of the data to be uploaded;

[0151] The first generation module 420 is used to generate standard base data based on the base data using the Reed-Solomon encoding method;

[0152] The second generation module 430 is used to generate label information based on standard base data;

[0153] The third generation module 440 is used to generate a data deduplication request based on offset data and tag information, and send the data deduplication request to the cloud server.

[0154] The receiving module 450 is used to receive the data deduplication response from the cloud server based on the data deduplication request.

[0155] In a possible implementation, the first generating module 420 includes:

[0156] a grouping unit configured to group the base data according to a preset length;

[0157] a matching unit configured to match the base data by group in an index dictionary to obtain a matching result, wherein the index dictionary is constructed in advance according to a Reed-Solomon encoding method;

[0158] a first generating unit configured to generate standard base data according to the matching result.

[0159] In a possible implementation, the second generating module 430 includes:

[0160] a first obtaining unit configured to obtain a serial number of each group of standard base data based on the index dictionary;

[0161] a first blocking unit configured to block the standard base data according to a preset number of groups;

[0162] a second obtaining unit configured to obtain label information by using a hash algorithm according to the serial number corresponding to each block of standard base data.

[0163] In a possible implementation, the third generating module 440 includes:

[0164] a second blocking unit configured to block the offset data according to the preset number of groups;

[0165] a compression unit configured to compress each block of offset data according to a ZSTD compression algorithm to obtain compressed offset data;

[0166] a second generating unit configured to generate a data deduplication request according to the compressed offset data and the label information.

[0167] In a possible implementation, the cloud data decomposition deduplication apparatus 400 based on Reed-Solomon encoding further includes:

[0168] a sending module configured to send target standard base data to a cloud server according to a data deduplication response.

[0169] According to the cloud data decomposition deduplication device based on Reed-Solomon coding, the base data and the offset data of the data to be uploaded are acquired, then the standard base data is generated by the Reed-Solomon coding method according to the base data, the label information is generated according to the standard base data, the data deduplication request is generated based on the offset data and the label information, the data deduplication request is sent to the cloud server, and finally the data deduplication response fed back by the cloud server based on the data deduplication request is received. The application can reduce data redundancy and improve the generalization ability of the base data.

[0170] It should be noted that the foregoing explanation of the cloud data decomposition deduplication method based on Reed-Solomon coding is also applicable to the cloud data decomposition deduplication device based on Reed-Solomon coding, which will not be described here.

[0171] In addition, the terms "first", "second" are only for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined as "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "a plurality of" is at least two, for example, two, three, etc., unless otherwise specifically limited.

[0172] In the description of the present application, the description of the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In the present application, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, the skilled in the art can combine and combine the different embodiments or examples described in the present application and the features of the different embodiments or examples without contradiction.

[0173] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above embodiments within the scope of the present application.

Claims

1. A cloud data decomposition and deduplication method based on Reed-Solomon encoding, characterized in that, include: Obtain the base data and offset data of the data to be uploaded; Based on the base data, standard base data is generated using the Reed-Solomon encoding method. The base data is grouped according to a preset length, and the base data is matched group by group in the index dictionary to obtain the matching results. The index dictionary is pre-constructed according to the Reed-Solomon encoding method, and the standard base data is generated based on the matching results. Generate label information based on the standard base data; A data deduplication request is generated based on the offset data and the tag information, and the data deduplication request is sent to the cloud server; Receive the data deduplication response from the cloud server based on the data deduplication request.

2. The cloud data decomposition and deduplication method based on Reed-Solomon encoding according to claim 1, characterized in that, The step of generating label information based on the standard base data includes: The sequence number of each group of standard base data is obtained based on the index dictionary; The standard base data is divided into blocks according to a preset number of groups; The tag information is obtained by using a hash algorithm based on the sequence number corresponding to each block of standard base data.

3. The cloud data decomposition and deduplication method based on Reed-Solomon encoding according to claim 2, characterized in that, The process of generating a data deduplication request based on the offset data and the tag information includes: The offset data is divided into blocks according to the preset number of groups; Each block of offset data is compressed according to the ZSTD compression algorithm to obtain the compressed offset data; A data deduplication request is generated based on the compressed offset data and the tag information.

4. The cloud data decomposition and deduplication method based on Reed-Solomon encoding according to claim 1, characterized in that, After receiving the data deduplication response from the cloud server based on the data deduplication request, the process further includes: Based on the deduplication response of the data, the target standard base data is sent to the cloud server.

5. A cloud data decomposition and deduplication device based on Reed-Solomon encoding, characterized in that, include: The acquisition module is used to obtain the base data and offset data of the data to be uploaded; The first generation module is used to generate standard base data based on the base data using the Reed-Solomon encoding method. The base data is grouped according to a preset length, and the base data is matched group by group in an index dictionary to obtain matching results. The index dictionary is pre-constructed according to the Reed-Solomon encoding method, and the standard base data is generated based on the matching results. The second generation module is used to generate label information based on the standard base data; The third generation module is used to generate a data deduplication request based on the offset data and the tag information, and send the data deduplication request to the cloud server; The receiving module is used to receive the data deduplication response from the cloud server based on the data deduplication request.

6. The cloud data decomposition and deduplication device based on Reed-Solomon encoding according to claim 5, characterized in that, The second generation module includes: The first acquisition unit is used to acquire the sequence number of each group of standard base data based on the index dictionary; The first segmentation unit is used to segment the standard base data into blocks according to a preset number of groups; The second acquisition unit is used to acquire the tag information by means of a hash algorithm based on the sequence number corresponding to each block of the standard base data.

7. The cloud data decomposition and deduplication device based on Reed-Solomon encoding according to claim 6, characterized in that, The third generation module includes: The second segmentation unit is used to segment the offset data into blocks according to the preset number of groups; A compression unit is used to compress each block of offset data according to the ZSTD compression algorithm to obtain compressed offset data; The second generation unit is used to generate a data deduplication request based on the compressed offset data and the tag information.

8. The cloud data decomposition and deduplication device based on Reed-Solomon encoding according to claim 5, characterized in that, Also includes: The sending module is used to send the target standard base data to the cloud server based on the deduplication response of the data.

Images