Data processing method, device and equipment and computer readable storage medium

Abstract

The invention discloses a data processing method, device and equipment and a computer readable storage medium, and belongs to the technical field of computers. In the method, a characteristic value of first data is obtained, and the characteristic value represents a compression rate or a deduplication rate of the first data. And then, determining the range of the slice length according to the characteristic value, and slicing the first data according to the range of the slice length to obtain the sliced data. And then, performing at least one of deduplication or compression on the sliced data to obtain processed data. Wherein the characteristic value reflects the characteristics of the first data, the range of the slice length determined on the basis of the characteristic value is relatively accurate, and after slicing is executed according to the range of the slice length, the sliced data can have a relatively appropriate length, so that after at least one of deduplication or compression is executed on the sliced data, the data can be subjected to re-deletion and compression, and the data can be subjected to re-deletion and compression. And the obtained processed data has a relatively high reduction rate.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a method and device for processing data, equipment and computer readable storage medium. BACKGROUND

[0002] With the development of computer technology, there are more and more ways to process data. One of the processing methods is to perform operations such as deduplication or compression on data to reduce data.

[0003] In the related art, a fixed length is determined according to the access frequency or modification frequency of data, the data is sliced according to the fixed length, the sliced data is obtained, and the sliced data is deduplicated and compressed. For example, the fixed length is determined as 4 kilobytes (KB), and the total length of the data is 20 KB. The sliced data includes five 4 KB data slices, and the five 4 KB data slices are deduplicated or compressed.

[0004] However, the fixed length determined according to the access frequency or modification frequency of data is not accurate enough, resulting in a low data reduction rate. SUMMARY

[0005] The present application provides a method and device for processing data, equipment and computer readable storage medium to improve the problem of low data reduction rate. The technical solution provided by the present application includes the following aspects.

[0006] In a first aspect, a method for processing data is provided. In the method, a characteristic value of first data is obtained, and the characteristic value represents the compression rate or deduplication rate of the first data. Then, a range of slice length is determined according to the characteristic value, the first data is sliced according to the range of slice length, and the sliced data is obtained. Next, at least one of deduplication or compression is performed on the sliced data to obtain processed data.

[0007] The characteristic value of the first data represents the compression rate or deduplication rate of the first data, which reflects the characteristics of the first data itself. Then, the range of slice length is determined according to the characteristic value, the first data is sliced according to the range of slice length, and the sliced data is obtained. Since the characteristic value reflects the characteristics of the first data itself and has a strong correlation with the data itself, the range of slice length determined according to the characteristic value is more accurate, and the sliced data obtained according to the range of slice length has a more appropriate length. Therefore, the processed data obtained by performing at least one of deduplication or compression on the sliced data has a high data reduction rate.

[0008] In a possible implementation, the characteristic value of the first data is obtained, including: obtaining a characteristic value of the second data, the characteristic value representing a compression rate or a deduplication rate of the second data. Accordingly, the slicing of the first data according to the range of the slice length is performed to obtain the sliced data, including: performing the slicing of the first data according to the range of the slice length and the position of the second data in the first data to obtain the sliced data.

[0009] Since the second data is a part of the first data, the data amount of the second data is less than that of the first data, and the calculation amount of the characteristic value of the second data is less, and the obtaining efficiency is higher, which is beneficial to improve the efficiency of processing data. Moreover, since the range of the slice length is determined according to the characteristic value of the second data, the slicing is performed according to the range of the slice length and the position of the second data in the first data, which can ensure the accuracy of the slicing, and the length of the sliced data is more appropriate.

[0010] In a possible implementation, the range of the slice length is determined according to the characteristic value, including: determining an average value of the range of the slice length according to the characteristic value; and determining the range of the slice length according to the average value.

[0011] This way can simply, quickly and accurately determine the range of the slice length according to the characteristic value, which ensures the accuracy and efficiency of determining the range of the slice length, and is beneficial to improve the accuracy and efficiency of processing data.

[0012] In a possible implementation, the average value of the range of the slice length is determined according to the characteristic value, including: determining the average value of the range of the slice length according to the characteristic value according to a first determination manner; and determining the range of the slice length according to the average value, including: determining the range of the slice length according to the average value according to a second determination manner. At least one of the first determination manner or the second determination manner is determined according to a reduction rate of third data before the first data.

[0013] The first determination manner and the second determination manner are both determination manners that can be used in the process of determining the range of the slice length. The determination manner is updated according to the reduction rate of the third data, which can ensure the accuracy of the determination manner, and thus ensure the accuracy of the range of the slice length determined by using the determination manner.

[0014] In a possible implementation, the compression rate represented by the characteristic value is positively correlated with the average value of the range of the slice length.

[0015] When the compression rate represented by the characteristic value is small, it can be considered that even if the average value of the range of the slice length is increased to increase the length of the data block, it can be difficult to greatly increase the compression rate of the first data, and thus it is better to decrease the average value of the range of the slice length to decrease the length of the data block, so as to greatly increase the deduplication rate of the first data, and then ensure that the reduction rate of the first data is high. Therefore, the compression rate represented by the characteristic value can be positively correlated with the average value of the range of the slice length.

[0016] In a possible implementation, the characteristic value is an information entropy, and the information entropy represents the compression rate.

[0017] The information entropy can accurately represent the compression rate, and thus the slice length range determined based on the characteristic value can have high accuracy when the information entropy is used as the characteristic value.

[0018] In a possible implementation, the deduplication rate represented by the characteristic value is negatively correlated with the average value of the range of the slice length.

[0019] When the deduplication rate represented by the characteristic value is small, it can be considered that even if the average value of the range of the slice length is decreased to decrease the length of the data block, it can be difficult to greatly increase the deduplication rate of the first data, and thus it is better to increase the average value of the range of the slice length to increase the length of the data block, so as to greatly increase the compression rate of the first data, and then ensure that the reduction rate of the first data is high. Therefore, the deduplication rate represented by the characteristic value can be negatively correlated with the average value of the range of the slice length.

[0020] In a possible implementation, the method further includes storing the processed data.

[0021] The processed data has a high reduction rate, and thus the storage space occupied by the processed data is small, the utilization rate of the storage space is improved, and the storage cost is reduced.

[0022] In a possible implementation, the first data is obtained by decompressing compressed data.

[0023] In this implementation, the first data is obtained by decompressing the compressed data, and then the processing of the first data is performed, so that the reduction rate of the first data is ensured, and the application range is wide.

[0024] In a possible implementation, the method further includes sending the processed data.

[0025] The processed data has a high reduction rate, and thus the bandwidth occupied by the processed data is small, and the sending efficiency is improved.

[0026] In a second aspect, a device for processing data is provided, and the device includes:

[0027] The acquisition module is configured to acquire a feature value of the first data, the feature value representing a compression rate or a deduplication rate of the first data.

[0028] The determination module is configured to determine a range of the slice length according to the feature value.

[0029] The slice module is configured to perform slicing on the first data according to the range of the slice length, to obtain the sliced data.

[0030] The processing module is configured to perform at least one of deduplication or compression on the sliced data, to obtain the processed data.

[0031] In a possible implementation, the acquisition module is configured to acquire a feature value of the second data, the feature value representing a compression rate or a deduplication rate of the second data; and the slice module is configured to perform slicing on the first data according to the range of the slice length and a position of the second data in the first data, to obtain the sliced data.

[0032] In a possible implementation, the range of the slice length is determined according to an average value of the feature value.

[0033] In a possible implementation, the determination module is configured to determine an average value of the range of the slice length according to the feature value, in a first determination manner; and determine the range of the slice length according to the average value, in a second determination manner. At least one of the first determination manner or the second determination manner is determined according to a reduction rate of third data before the first data.

[0034] In a possible implementation, the feature value representing the compression rate is positively correlated with the average value of the range of the slice length.

[0035] In a possible implementation, the feature value is an information entropy, and the information entropy represents the compression rate.

[0036] In a possible implementation, the feature value representing the deduplication rate is negatively correlated with the average value of the range of the slice length.

[0037] In a possible implementation, the apparatus further includes a storage module configured to store the processed data.

[0038] In a possible implementation, the first data is obtained by decompressing compressed data.

[0039] In a possible implementation, the apparatus further includes a sending module configured to send the processed data.

[0040] In a third aspect, a storage system is provided, which includes a data processing unit and a storage unit. The data processing unit is configured to execute the method for processing data provided in the first aspect or any possible implementation of the first aspect to obtain processed data. The storage unit is configured to store the processed data.

[0041] In a fourth aspect, a device for processing data is provided, which includes a memory and a processor. The memory stores at least one computer instruction, which is loaded and executed by the processor to enable the device to implement the method for processing data provided in the first aspect or any possible implementation of the first aspect.

[0042] Optionally, the processor is one or more, and the memory is one or more.

[0043] Optionally, the memory can be integrated with the processor, or the memory and the processor are separately arranged.

[0044] In a fifth aspect, a computer readable storage medium is provided, which stores at least one computer instruction. The at least one computer instruction is loaded and executed by a processor to enable a computer including the processor to implement the method for processing data provided in the first aspect or any possible implementation of the first aspect.

[0045] In a sixth aspect, a computer program product is provided, which includes computer instructions executed by a processor to enable a computer including the processor to implement the method for processing data provided in the first aspect or any possible implementation of the first aspect.

[0046] In a seventh aspect, a chip is provided, which includes a processor configured to call and run instructions stored in a memory to enable a computer installed with the chip to execute the method for processing data provided in the first aspect or any possible implementation of the first aspect.

[0047] In an eighth aspect, another chip is provided, which includes an input interface, an output interface, a processor and a memory. The input interface, the output interface, the processor and the memory are connected through internal connection paths. The processor is configured to execute code in the memory. When the code is executed, a computer installed with the chip executes the method for processing data provided in the first aspect or any possible implementation of the first aspect.

[0048] It should be understood that the technical effects achieved by the technical solutions provided in the second aspect to the eighth aspect and the corresponding possible implementations of the present application can be referred to the description of the technical effects achieved by the technical solutions provided in the first aspect and the corresponding possible implementations, which will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0049] Figure 1 A schematic diagram of a system architecture provided for an embodiment of the present application;

[0050] Figure 2 A schematic diagram of another system architecture provided for an embodiment of the present application;

[0051] Figure 3 A flowchart of a method for processing data provided for an embodiment of the present application;

[0052] Figure 4 A schematic diagram of determining an average of a range of slice length provided for an embodiment of the present application;

[0053] Figure 5 A schematic diagram of determining an average of a range of slice length provided for another embodiment of the present application;

[0054] Figure 6 A schematic diagram of determining a range of slice length provided for an embodiment of the present application;

[0055] Figure 7 A flowchart of processing data in a storage system architecture provided for an embodiment of the present application;

[0056] Figure 8 A structural schematic diagram of an apparatus for processing data provided for an embodiment of the present application;

[0057] Figure 9 A structural schematic diagram of an apparatus for processing data provided for an embodiment of the present application. DETAILED DESCRIPTION

[0058] The terms used in the embodiments section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0059] Data reduction is a way of processing data, and is used to reduce the data amount of data. For example, data reduction is used to reduce the byte (B) number of data.

[0060] In one example, the data reduction includes deduplication of the data, i.e., deduplication. In the deduplication process, the data is sliced to obtain sliced data, and the sliced data includes a plurality of data blocks (or data slices). In the plurality of data blocks, only one data block is kept, or only one data block and a copy of the data block (equivalent to keeping two data blocks) are kept, and the remaining data blocks are deleted and referenced to any of the kept data blocks. As a result of the deletion of the data blocks, the byte size of the data is reduced. In this case, different data blocks are duplicate data blocks when the contents of the different data blocks are the same. Alternatively, different data blocks are duplicate data blocks when the unique identifiers of the different data blocks are the same, which can be determined based on the contents or determined in other feasible manners. For the deduplication process, the larger the length of the data blocks, i.e., the more the byte size of the data blocks, the lower the possibility of different data blocks being duplicate data blocks, and the fewer the number of duplicate data blocks, and thus the lower the deduplication rate.

[0061] In another example, the data reduction includes compression of the data. In the compression process, a redundant part of the data is analyzed by a compression algorithm (or compression technique), or a characteristic of the data is analyzed, to express the data in a more concise manner, so as to reduce the byte size of the data. In the case where the data is sliced, the compression process can be performed on the data blocks included in the sliced data. For the compression process, the larger the length of the data blocks, i.e., the more the byte size of the data blocks, the easier it is to analyze the redundant part of the data and the characteristic of the data, and thus the higher the compression rate. The compression rate is relative to the deduplication rate.

[0062] For the case where the data needs to be sliced, the related art regards the data as a whole file, determines a fixed length according to a file-level usage (such as the access frequency or modification frequency of the file), slices the data according to the fixed length to obtain a plurality of data blocks of the same length, and performs deduplication or compression on the plurality of data blocks. However, the file-level usage only reflects the frequency of use of the file by the user, and has less relevance to the data itself. Therefore, the fixed length determined according to the file-level usage is not accurate enough, and the length of the data blocks obtained by slicing the data according to the fixed length is not appropriate, resulting in a low data reduction rate.

[0063] To this end, an embodiment of the present application provides a method for processing data. The method is from the perspective of the data itself. After obtaining eigenvalues of the data to be processed, the method realizes accurate slicing of the data according to the eigenvalues, obtains sliced data, and the length of the sliced data is appropriate. Then, the method performs at least one of re-deletion or compression on the sliced data, obtains processed data, and the number of bytes of the processed data is smaller, so that the reduction rate of the data is higher.

[0064] Before describing the method provided by an embodiment of the present application, a system architecture to which the method can be applied is described by way of example, so as to facilitate understanding.

[0065] The first system architecture is a storage system. As shown in Figure 1 , the storage system includes a data processing unit and a storage unit. The data processing unit processes data by using the method provided by an embodiment of the present application, so as to realize data reduction, reduce the data amount of the data, and obtain processed data. The storage unit stores the processed data. The processed data occupies less storage space in the storage unit, saves the storage space of the storage unit, enables the storage unit to store more other data, thereby improving the utilization rate of the storage space and reducing the storage cost.

[0066] By way of example, the data processing unit is implemented by using at least one of software or hardware. The storage unit can be a persistent storage or a non-persistent storage. An embodiment of the present application does not limit the form of the data processing unit and the storage unit.

[0067] Optionally, the storage system described above can be located in one computer device. Alternatively, the storage system described above can be located in multiple computer devices to form a distributed storage system. For example, the storage system includes multiple data processing units and multiple storage units. The multiple computer devices respectively include at least one of the multiple data processing units and at least one of the multiple storage units. In one example, at least one of the multiple computer devices is a primary storage device, and the other computer devices are backup storage devices.

[0068] The second system architecture is a communication system. As shown in Figure 2 , the communication system includes a data processing unit and a transmission unit. The data processing unit processes data by using the method provided by an embodiment of the present application, so as to realize data reduction, reduce the data amount of the data, and obtain processed data. The transmission unit transmits the processed data. The processed data occupies less bandwidth in the transmission unit, not only saves the bandwidth of the transmission unit, but also requires less time to transmit the processed data, and has high transmission efficiency.

[0069] Exemplarily, the data processing unit is implemented by at least one of software or hardware, and the transmission unit can be a communication interface. The embodiments of the present application do not limit the forms of the data processing unit and the transmission unit.

[0070] The third system architecture is a combination of a storage system and a communication system. For example, the combination system includes a data processing unit, a storage unit and a transmission unit. The data processing unit processes data to obtain processed data. The storage unit stores the processed data. The transmission unit transmits the processed data. The descriptions of the units can be referred to the above, and will not be repeated here.

[0071] The above three system architectures are only examples. The method provided by the embodiments of the present application can also be applied to other feasible system architectures, which are not limited herein. Next, the method provided by the embodiments of the present application is applied to Figure 1 or Figure 2 The method is described by taking the data processing unit shown in Figure 3 The method includes the following steps 301 to 304.

[0072] Step 301: Obtain a characteristic value of first data. The characteristic value represents a compression ratio or a deduplication ratio of the first data.

[0073] The first data is data to be processed, or in other words, the first data is data to be reduced. The embodiments of the present application can obtain the first data by at least one of receiving, generating or decompressing. The decompression means performing decompression on the compressed data. After obtaining the first data, the characteristic value of the first data can be obtained. The characteristic value of the first data reflects the characteristics of the first data itself. The characteristics of the first data itself affect the compression ratio or the deduplication ratio of the first data. Therefore, the characteristic value of the first data can represent the compression ratio or the deduplication ratio of the first data. The compression ratio represented by the characteristic value of the first data is equivalent to the compression ratio predicted for the first data. The deduplication ratio represented by the characteristic value of the first data is equivalent to the deduplication ratio predicted for the first data.

[0074] In an exemplary embodiment, the characteristic value of the first data includes the following two kinds.

[0075] The first characteristic value represents the compression ratio of the first data.

[0076] The first characteristic value can be any index used to represent compression rate, and embodiments of the present application do not limit the same. In one example, the first characteristic value is information entropy. The information entropy is defined as an average value of the number of bits of each byte of the first data, in other words, an average number of bits needed to represent each byte of the first data. The smaller the information entropy, the fewer bits needed to represent each byte of the first data on average, and thus the higher the compression rate of the first data. The larger the information entropy, the more bits needed to represent each byte of the first data on average, and thus the lower the compression rate of the first data. Since one byte includes 8 bits, one byte can be represented by at most 8 bits, and accordingly, each byte of the first data can be represented by at most 8 bits on average, in which case the upper limit of the information entropy is 8 bits and the lower limit of the compression rate is 0%. Therefore, the information entropy is negatively correlated with the compression rate represented by the information entropy. Alternatively, the information entropy of the first data can be calculated according to the following formula (1).

[0077] E = -∑p(x)·log2p(x) Formula (1)

[0078] In the formula (1), x represents a byte included in the first data and having a certain value, p(x) represents a probability of x appearing in the first data, and log2(·) represents a logarithm operation with 2 as the base. For example, the first data AACC with a length of 4B, the byte with a value of A corresponds to p(x) of 0.5 and log2p(x) of -1, and the byte with a value of C corresponds to p(x) of 0.5 and log2p(x) of -1, and thus the information entropy E of the first data AACC = -(0.5·(-1) + 0.5·(-1)) = 1, i.e., each byte with a value needs to be represented by 1 bit on average, for example, A is represented as 0 and C is represented as 1, and the first data AACC needs to be represented by 4 bits after compression.

[0079] The second characteristic value represents a deduplication rate of the first data.

[0080] The second characteristic value can be any index used to represent the deduplication rate, and embodiments of the present application do not limit this. In an example, the second characteristic value is a deduplication rate prediction value output by a deduplication rate prediction model for the first data, which is obtained through training. In the training process, sample data is obtained, and after performing deduplication on the sample data, a deduplication rate is obtained. The sample data labeled with the deduplication rate is used as a training sample, and the deduplication rate prediction model is trained to have a deduplication rate prediction function. In the inference process, the first data is input into the deduplication rate prediction model, and the deduplication rate prediction model performs prediction for the first data to obtain a deduplication rate prediction value, and outputs the deduplication rate prediction value. Therefore, the second characteristic value can be the deduplication rate prediction value. Optionally, the deduplication rate prediction value is positively correlated with the deduplication rate represented by the deduplication rate prediction value.

[0081] In exemplary embodiments, the manner of obtaining the characteristic value of the first data includes but is not limited to the following two manners.

[0082] The first obtaining manner includes obtaining a characteristic value of second data, the characteristic value representing a compression rate or a deduplication rate of the second data, and the second data being part of the first data.

[0083] Since the second data is part of the first data, the second data can be obtained from the first data after the first data is obtained. Optionally, any part of the first data can be used as the second data, or in other words, the second data can be located at any position in the first data. For example, part of the first data starting from the beginning of the first data can be used as the second data. Taking the length of the first data as 10 KB as an example, the second data can be the first 1-4 KB of the first data. The process of obtaining the second data can be performed once or multiple times (to obtain multiple second data), and embodiments of the present application do not limit the number of times of execution.

[0084] After the second data is obtained, the characteristic value of the second data can be obtained. Since the second data is part of the first data, the data amount of the second data is smaller than that of the first data, and the calculation amount of the characteristic value of the second data is smaller and the efficiency of obtaining the characteristic value of the second data is higher. When the characteristic value of the second data represents the compression rate of the second data, the characteristic value of the second data is, for example, the information entropy of the second data, and the manner of obtaining the information entropy of the second data can be referred to the description of the first characteristic value above. When the characteristic value of the second data represents the deduplication rate of the second data, the characteristic value of the second data is, for example, a deduplication rate prediction value output by a deduplication rate prediction model for the second data, and the manner of obtaining the deduplication rate prediction value can be referred to the description of the second characteristic value above.

[0085] In one example, the embodiment of the present application can take the eigenvalue of the second data as the eigenvalue of the first data, in other words, the eigenvalue of the first data is the eigenvalue of the second data. In this example, the number of the eigenvalue of the first data is one.

[0086] In another example, the first data includes multiple second data, and the embodiment of the present application can obtain the eigenvalue of the second data multiple times to obtain the eigenvalue of each of the multiple second data, and determine the eigenvalue of the first data according to the eigenvalue of each of the multiple second data. For example, the first data includes second data A and second data B, and the eigenvalue 1 of the second data A and the eigenvalue 2 of the second data B can be obtained respectively, and the eigenvalue of the first data is determined according to the eigenvalue 1 of the second data A and the eigenvalue 2 of the second data B.

[0087] The characteristics of different parts of the first data (i.e. different second data) can be different, and the eigenvalue of each of the multiple second data reflects the characteristics of each of the multiple second data, so that the eigenvalue of the first data determined according to the eigenvalue of each of the multiple second data can be more comprehensive and accurate.

[0088] For example, the eigenvalue of the first data is determined according to the eigenvalue of each of the multiple second data, which includes taking the eigenvalue of each of the multiple second data as the eigenvalue of the first data, in other words, the eigenvalue of the first data includes the eigenvalue of the multiple second data, and the number of the eigenvalue of the first data is multiple. Alternatively, the eigenvalue of the first data is determined according to the eigenvalue of each of the multiple second data, which includes performing weighted calculation on the eigenvalue of each of the multiple second data to obtain a result of the weighted calculation, and taking the result of the weighted calculation as the eigenvalue of the first data, in other words, the eigenvalue of the first data is the result of the weighted calculation, and the number of the eigenvalue of the first data is one.

[0089] The embodiments of the present application do not limit the length of each second data, and the length of the second data can be determined according to experience or actual demand. For example, the length of the second data is 4 KB. The lengths of the plurality of second data can be the same or different. For example, the length of the second data A is 4 KB, and the length of the second data B is 6 KB. Different second data in the plurality of second data can have a repeated part or can not have a repeated part. For example, the length of the first data is 10 KB. In the case that different second data has a repeated part, the second data A is 1-6 KB of 10 KB, and the second data B is 5-10 KB of 10 KB, and the repeated part is 5 and 6 KB of 10 KB. In the case that different second data does not have a repeated part, the second data A is 1-4 KB of 10 KB, and the second data B is 5-8 KB of 10 KB. In addition, different second data in the plurality of second data can have an interval part or can not have an interval part. For example, the length of the first data is 10 KB, the second data A is 1-4 KB of 10 KB, and the second data B is 7-10 KB of 10 KB, and the interval part is 5 and 6 KB of 10 KB.

[0090] In the second acquisition mode, the characteristic value of the first data is acquired, including: acquiring the characteristic value of all the first data, and the characteristic value representing the compression rate or the re-deletion rate of the first data.

[0091] In the second acquisition mode, the characteristic value of the first data is acquired, including: acquiring the characteristic value of all the first data, and the characteristic value representing the compression rate or the re-deletion rate of the first data.

[0092] According to the above description, after the characteristic value of the data to be processed (i.e. the first data) is acquired, the slicing of the data is realized according to the characteristic value, and the sliced data is obtained. The slicing process is described in detail in steps 302 and 303 below.

[0093] In step 302, the range of the slicing length is determined according to the characteristic value.

[0094] The range of the slice length includes at least one slice length. The range of the slice length can be a set (including at least one slice length) or an interval (including multiple slice lengths), or a combination of the set and the interval. According to the description of the characteristic value of the first data in step 301, the number of the characteristic value of the first data can be one or more. In the case where the number of the characteristic value of the first data is one, the range of the slice length is determined according to the one characteristic value. In the case where the number of the characteristic value of the first data is more (i.e., the characteristic value of the first data includes the characteristic value of the second data), the range of the slice length is determined according to each characteristic value, i.e., the range of the slice length is determined multiple times, and the multiple ranges of the slice length are obtained. The process of determining the range of the slice length multiple times is equivalent to multiple repetitions of the process of determining the range of the slice length according to one characteristic value. Therefore, the process of determining the range of the slice length according to one characteristic value is taken as an example for description, so as to facilitate understanding.

[0095] The range of the slice length determined according to the characteristic value is used to perform slicing on the first data, and the sliced data including multiple data blocks is obtained. The larger the average of the range of the slice length is, the larger the length of the data block can be. The smaller the average of the range of the slice length is, the smaller the length of the data block can be. Since the range of the slice length is determined according to the characteristic value, the characteristic value affects the range of the slice length, in other words, there is a correlation between the characteristic value and the range of the slice length. In the exemplary embodiment, the correlation includes the following two cases.

[0096] The first case: the compression rate represented by the characteristic value is positively correlated with the average of the range of the slice length.

[0097] In the first example, when the compression rate represented by the characteristic value is small, it is considered that the improvement space of the compression rate of the first data is small. For example, the compression rate represented by the characteristic value is small means that the compression rate represented by the characteristic value is in the first compression rate range. According to the above description, the larger the length of the data block is, the higher the compression rate is. Based on this, the small improvement space of the compression rate of the first data means that even if the average of the range of the slice length is increased to increase the length of the data block, it is difficult to greatly improve the compression rate of the first data, and it is better to reduce the average of the range of the slice length to reduce the length of the data block, which may cause the compression rate of the first data to decrease slightly, but can greatly improve the deduplication rate of the first data. Since both deduplication and compression can achieve data reduction, the deduplication rate and the compression rate of the first data both affect the reduction rate of the first data, so that the deduplication rate of the first data is greatly improved, the compression rate of the first data is slightly decreased, and the reduction rate of the first data is still high.

[0098] In the embodiment of the present application, the smaller the value of the first compression rate range, the smaller the compression rate represented by the characteristic value. According to the analysis in the above paragraph, the smaller the average value of the range of the slice length, and the greater the improvement of the deduplication rate of the first data, thereby ensuring a higher reduction rate of the first data. Therefore, the compression rate represented by the characteristic value is positively correlated with the average value of the range of the slice length.

[0099] Referring to Figure 4 For example, taking the information entropy as the characteristic value representing the compression rate, the greater the information entropy, the smaller the compression rate represented by the information entropy, the smaller the average value of the range of the slice length, and the smaller the length of the data block, thereby ensuring a greater improvement of the deduplication rate of the first data, and further ensuring a higher reduction rate of the first data.

[0100] In the second example, when the compression rate represented by the characteristic value is greater, it is considered that the improvement space of the compression rate of the first data is greater. For example, the compression rate represented by the characteristic value is greater, which means that the compression rate represented by the characteristic value is in the second compression rate range, and the second compression rate range is greater than the first compression rate range. The greater improvement space of the compression rate of the first data means that the average value of the range of the slice length is increased, so that the length of the data block is increased, which can greatly improve the compression rate of the first data. Therefore, the average value of the range of the slice length can be increased, so that the length of the data block is increased, although it may slightly reduce the deduplication rate of the first data, but it can greatly improve the compression rate of the first data, thereby ensuring a higher reduction rate of the first data.

[0101] In the embodiment of the present application, the greater the value of the second compression rate range, the greater the compression rate represented by the characteristic value. According to the analysis in the above paragraph, the greater the average value of the range of the slice length, and the greater the improvement of the compression rate of the first data, thereby ensuring a higher reduction rate of the first data. Therefore, the compression rate represented by the characteristic value is still positively correlated with the average value of the range of the slice length.

[0102] Referring to Figure 5 For example, taking the information entropy as the characteristic value representing the compression rate, the smaller the information entropy, the greater the compression rate represented by the information entropy, the greater the average value of the range of the slice length, and the greater the length of the data block, thereby ensuring a greater improvement of the compression rate of the first data, and further ensuring a higher reduction rate of the first data.

[0103] In the second case, the deduplication rate represented by the characteristic value is negatively correlated with the average value of the range of the slice length.

[0104] In the third example, when the feature value represents a small deduplication rate, it is considered that the first data has a small improvement space for the deduplication rate. For example, the feature value represents a small deduplication rate means that the feature value represents a deduplication rate in the first deduplication rate range. As described above, the larger the length of the data block is, the lower the deduplication rate is. Correspondingly, the smaller the length of the data block is, the higher the deduplication rate is. Based on this, the small improvement space for the deduplication rate of the first data means that even if the average of the range of the slice length is reduced to reduce the length of the data block, it is difficult to greatly improve the deduplication rate of the first data, and thus it is better to increase the average of the range of the slice length to increase the length of the data block, although it may slightly reduce the deduplication rate of the first data, it can greatly improve the compression rate of the first data, thereby ensuring a high reduction rate of the first data.

[0105] In the embodiments of the present application, the smaller the first deduplication rate range is, the smaller the feature value represents a deduplication rate, and according to the analysis in the above paragraph, the larger the average of the range of the slice length is, thereby the larger the improvement amplitude of the compression rate of the first data is, and thus a high reduction rate of the first data is ensured. Therefore, the feature value represents a deduplication rate and the average of the range of the slice length are negatively correlated.

[0106] For example, the feature value represents a deduplication rate is taken as a deduplication rate prediction value, and the deduplication rate prediction value is taken as an example for illustration. Since the deduplication rate prediction value is positively correlated with the deduplication rate represented by the deduplication rate prediction value, the smaller the deduplication rate prediction value is, the smaller the deduplication rate represented by the deduplication rate prediction value is, and the larger the average of the range of the slice length is, the larger the length of the data block is, thereby the larger the improvement amplitude of the compression rate of the first data is (i.e., ensuring the compression rate), and thus a high reduction rate of the first data is ensured.

[0107] In the fourth example, when the feature value represents a large deduplication rate, it is considered that the first data has a large improvement space for the deduplication rate. For example, the feature value represents a large deduplication rate means that the feature value represents a deduplication rate in the second deduplication rate range, and the second deduplication rate range is larger than the first deduplication rate range described above. The large improvement space for the deduplication rate of the first data means that the average of the range of the slice length is reduced to reduce the length of the data block, thereby the deduplication rate of the first data can be greatly improved, and thus the average of the range of the slice length can be reduced to reduce the length of the data block, although it may slightly reduce the compression rate of the first data, it can greatly improve the deduplication rate of the first data, thereby ensuring a high reduction rate of the first data.

[0108] In the embodiments of the present application, the greater the value of the second deduplication rate range, the greater the deduplication rate represented by the characteristic value. According to the analysis in the above paragraph, the average value of the range of the slice length can be made smaller, so that the first data deduplication rate can be increased more greatly, thereby ensuring that the first data reduction rate is higher. Therefore, the deduplication rate represented by the characteristic value is still negatively correlated with the average value of the range of the slice length.

[0109] The characteristic value representing the deduplication rate is taken as the deduplication rate prediction value as an example for illustration. Since the deduplication rate prediction value is positively correlated with the deduplication rate represented by the deduplication rate prediction value, the greater the deduplication rate prediction value, the greater the deduplication rate represented by the deduplication rate prediction value, the smaller the average value of the range of the slice length, and the smaller the length of the data block, so that the first data deduplication rate can be increased more greatly (i.e., the deduplication rate is ensured), thereby ensuring that the first data reduction rate is higher.

[0110] The first example to the fourth example described above can ensure that the first data reduction rate is higher. Among them, the first example (the characteristic value represents the compression rate) and the fourth example (the characteristic value represents the deduplication rate) both ensure the first data reduction rate by ensuring the deduplication rate, and the second example (the characteristic value represents the compression rate) and the third example (the characteristic value represents the deduplication rate) both ensure the first data reduction rate by ensuring the compression rate.

[0111] In the example embodiments, determining the range of the slice length according to the characteristic value includes: determining the average value of the range of the slice length according to the characteristic value, and determining the range of the slice length according to the average value. This kind of way is simple and easy to implement, and is conducive to accurately and quickly determining the range of the slice length.

[0112] For the first case described above, i.e., the case that the characteristic value represents the compression rate and is positively correlated with the average value of the range of the slice length, the embodiments of the present application can determine the average value of the range of the slice length according to the characteristic value according to the following formula (2), and determine the range of the slice length according to the average value of the range of the slice length according to the following formula (3).

[0113]

[0114] L min = C, mask = average - C, L max = L min + 2·mask = average + mask Formula (3)

[0115] In formula (2) and formula (3), average represents the average value of the range of the slice length, H represents the characteristic value representing the compression rate, and f(H) represents a function related to H. A, B, C, and D are all constants, and the values of these constants can be determined according to experience or actual needs, represents the ceiling. L min represents the minimum (or lower limit) of the range of slice length, mask represents the difference between the average value and the minimum value of the range of slice length, L max represents the maximum (or upper limit) of the range of slice length.

[0116] For example, H is the information entropy E in the above formula (1). A is the minimum value of the average length of the data block, and the value of A is, for example, 8 KB. B is the influence coefficient of the information entropy on the average length of the data block, and the value of B is, for example, 3 KB. C is the minimum value of the length of the data block, and the value of C is, for example, 4 KB. D is, for example, the upper limit of the information entropy, and the value of D is 8 bits.

[0117] For the second case described above, that is, the case where the feature value represents the re-duplication rate and is negatively correlated with the average value of the range of slice length, the embodiment of the present application can determine the average value of the range of slice length according to the feature value according to the following formula (4), and determine the range of slice length according to the average value of the range of slice length according to the following formula (5).

[0118] average = f (I) = F + G · I formula (4)

[0119] L min = J, mask = average - J, L max = L min + 2 · mask = average + mask formula (5)

[0120] In formula (4) and formula (5), average represents the average value of the range of slice length, I represents the feature value representing the re-duplication rate, and f (I) represents a function related to I. F, G and J are all constants, and the values of these constants can be determined according to experience or actual demand. L min , mask and L max The contents represented by them are the same as those in formula (3), and the description in formula (3) can be referred to, which will not be repeated here.

[0121] Exemplarily, determining the average value of the range of slice length according to the feature value includes determining the average value of the range of slice length according to the feature value according to a first determination manner. Determining the range of slice length according to the average value includes determining the range of slice length according to the average value according to a second determination manner.

[0122] Optionally, the first determining manner refers to performing a first processing according to at least one first processing parameter. That is, according to the feature values, the average value of the range of the slice length is determined according to the first determining manner, including: performing the first processing on the feature values according to at least one first processing parameter to obtain the average value. The first processing includes, but is not limited to, at least one of the processing manners such as summation, difference, multiplication, division, rounding up or rounding down.

[0123] In one example, the at least one first processing parameter includes a first parameter, a second parameter and a third parameter. The first processing on the feature values to obtain the average value includes: performing difference processing on the first parameter and the feature values to obtain a first difference value, performing rounding up processing on the first difference value to obtain a rounding up result, performing multiplication processing on the rounding up result and the second parameter to obtain a first product, and performing summation processing on the first product and the third parameter to obtain the average value.

[0124] For example, the first determining manner is the above-mentioned formula (2), the first parameter corresponds to D in formula (2), the second parameter corresponds to B in formula (2), and the third parameter corresponds to A in formula (2).

[0125] In another example, the at least one first processing parameter includes a fourth parameter and a fifth parameter. The first processing on the feature values to obtain the average value includes: performing multiplication processing on the fourth parameter and the feature values to obtain a second product, and performing summation processing on the second product and the fifth parameter to obtain the average value.

[0126] For example, the first determining manner is the above-mentioned formula (4), the fourth parameter corresponds to G in formula (4), and the fifth parameter corresponds to F in formula (4).

[0127] Optionally, the second determining manner refers to performing a second processing according to at least one second processing parameter. That is, according to the average value, the range of the slice length is determined according to the second determining manner, including: performing the second processing on the average value according to at least one second processing parameter to obtain the range of the slice length. The second processing can be the same as or different from the above-mentioned first processing. The second processing parameter can be the same as or different from the above-mentioned first processing parameter.

[0128] In one example, the at least one second processing parameter is a sixth parameter. The second processing on the average value according to the at least one second processing parameter to obtain the range of the slice length includes: taking the sixth parameter as a lower limit of the range of the slice length, performing difference processing on the sixth parameter and the average value to obtain a second difference value, performing summation processing on the second difference value and the average value to obtain an upper limit of the range of the slice length, and thus obtaining the range of the slice length between the lower limit and the upper limit.

[0129] For example, the second determination manner is the above-mentioned formula (3), the sixth parameter corresponds to C in formula (3), and the second difference value corresponds to mask in formula (3). For another example, the second determination manner is the above-mentioned formula (5), the sixth parameter corresponds to J in formula (5), and the second difference value corresponds to mask in formula (5).

[0130] In another example, the at least one second processing parameter includes a seventh parameter and an eighth parameter, which can be the same or different. The second processing performed on the average value according to the at least one second processing parameter to obtain the range of the slice length includes: performing difference processing on the average value and the seventh parameter to obtain a lower limit of the range of the slice length, performing sum processing on the average value and the eighth parameter to obtain an upper limit of the range of the slice length, and thus obtaining the range of the slice length between the lower limit and the upper limit.

[0131] The above descriptions of the first determination manner and the second determination manner are examples and are not used to limit the first determination manner and the second determination manner. At least one of the first determination manner or the second determination manner is obtained according to the reduction rate of the third data before the first data. That is, the embodiments of the present application can obtain the third data and obtain a reference characteristic value of the third data. According to the reference characteristic value, a reference average value is determined according to a first reference manner, and a reference range of the slice length is determined according to a second reference manner according to the reference average value. After the third data is sliced according to the reference range of the slice length, at least one of the re-deletion or the compression is performed on the sliced third data to obtain reduced data. Then, a first reduction rate of the reduced data relative to the third data is determined, the first reference manner is updated according to the first reduction rate to obtain the first determination manner, and the second reference manner is updated according to the reduction rate of the third data to obtain the second determination manner. Since at least one of the first determination manner or the second determination manner is obtained according to the reduction rate of the third data, the at least one determination manner is more accurate, and a more accurate range of the slice length can be determined according to the at least one determination manner.

[0132] The embodiments of the present application do not limit the manner of updating according to the reduction rate of the third data. For example, the first determination manner is the above-mentioned formula (2), and the updating according to the reduction rate of the third data can mean that the values of A, B and D in formula (2) are updated according to the reduction rate of the third data, or can mean that the values of A, B and D in formula (2) are updated according to the reduction rate of the third data. This processing manner.

[0133] Optionally, embodiments of this application may also update at least one of the first or second determination methods. For example, after determining the range of slice length according to the first and second determination methods, steps 303 and 304 below are executed to obtain processed data. A second reduction rate of the processed data relative to the first data is determined, and at least one of the first or second determination methods is updated according to the second reduction rate to obtain an updated determination method. The updated determination method is used to determine the range of slice length for the fourth data after the first data. In other words, the method of determining the range of slice length based on feature values ​​in embodiments of this application can be continuously updated, ensuring that the determined range of slice length is relatively accurate.

[0134] The above explains the process of determining the range of slice length based on a single feature value. When the first data has multiple feature values ​​(i.e., the first data includes the feature values ​​of multiple second data), a range of slice lengths can be determined, allowing for dynamic adjustment of the slice length range. For example, taking information entropy as the feature value, if the first data includes second data A, second data B, and second data C, see [link to relevant documentation]. Figure 6 Based on the information entropy 1 (relatively large) of the second data A, the average value X (relatively small) of the range of slice lengths is determined, resulting in a smaller data block length M. Based on the information entropy 2 (relatively small) of the second data B, the average value Y (relatively large) of the range of slice lengths is determined, resulting in a larger data block length N. Based on the information entropy 3 (moderate) of the second data C, the average value Z (moderate) of the range of slice lengths is determined, resulting in a moderate data block length Q. Wherein, information entropy 1 > information entropy 3 > information entropy 2, the average value X of the range of slice lengths < the average value Z of the range of slice lengths < the average value Y of the range of slice lengths, and the data block length M < the data block length Q < the data block length N.

[0135] Step 303: Slice the first data according to the range of slice length to obtain the sliced ​​data.

[0136] After determining the range of slice lengths, the first data can be sliced ​​according to the range of slice lengths to obtain sliced ​​data, which includes multiple data blocks. In an exemplary embodiment, the slicing methods include, but are not limited to, the following two.

[0137] The first slicing method corresponds to the case in step 302 where the number of feature values ​​of the first data is one, and the range of slice length is determined based on one feature value.

[0138] In one example, at least one slice length is determined from the range of slice lengths, and the first data is sliced according to the at least one slice length to obtain the sliced data. Since the range of slice lengths is determined according to the characteristic value of the first data, the characteristic value of the first data reflects the characteristics of the first data itself, and thus, regardless of whether the number of slice lengths is one or more, the lengths of the plurality of data blocks obtained after slicing are suitable, which is conducive to ensuring the subsequent reduction rate. For example, the length of the first data is 20 KB, and the slice length is 5 KB. Then, the first data is sliced according to 5 KB, and 4 data blocks with a length of 5 KB are obtained. For another example, the length of the first data is 20 KB, and the slice length is 4 KB and 8 KB. Then, the first data is sliced according to 4 KB, and 1 data block with a length of 4 KB is obtained. Then, the first data is sliced according to 8 KB, and 2 data blocks with a length of 8 KB are obtained.

[0139] In another example, the first data is sliced according to a sliding window in the range of slice lengths to obtain the sliced data. The length of the sliding window is, for example, the lower limit of the range of slice lengths, and the sliding step of the sliding window can be determined according to experience or actual demand. The sliding step is, for example, 1B, and the embodiments of the present application do not limit the sliding step.

[0140] In the first slicing process, the lower limit of the sliding window is located at the starting position of the first data. If the data in the sliding window meets the condition, the first data is sliced according to the range of slice lengths, starting from the starting position of the first data, to obtain a first data block, and the length of the first data block is equal to the lower limit of the range of slice lengths. If the data in the sliding window does not meet the condition, the data in the sliding window is changed by performing one sliding according to the sliding step. If the changed data meets the condition, the sliding is ended, and the position from the starting position of the first data to the upper limit of the sliding window at the time of ending the sliding is taken as the first data block. If the data in the sliding window still does not meet the condition, one or more slidings are continuously performed according to the sliding step, until the data in the sliding window meets the condition after a certain sliding, and the position from the starting position of the first data to the upper limit of the sliding window at the time of ending the sliding is taken as the first data block. If the length between the position of the upper limit of the sliding window and the starting position of the first data after the sliding has reached the upper limit of the range of slice lengths, the first data is sliced according to the range of slice lengths, starting from the starting position of the first data, to obtain a first data block, and the length of the first data block is equal to the upper limit of the range of slice lengths. After obtaining the first data block, the data of the first data other than the first data block is the first remaining data.

[0141] In the second slicing process, the lower limit of the sliding window is located at the start position of the first remaining data, and the sliding window starts to slide from this position. The sliding manner can refer to the description in the first slicing process, which will not be repeated here. After the sliding is completed, the start position of the first remaining data to the position where the upper limit of the sliding window is located after the sliding is completed is taken as the second data block. After obtaining the second data block, the data in the first remaining data except the second data block is the second remaining data.

[0142] By analogy, until the remaining data (i.e., data that has not been sliced, that is, data that has not entered the sliding window) is less than or equal to the length of the sliding window after the slicing process is completed, the remaining data is taken as the last data block, and thus the slicing of the first data can be completed.

[0143] Optionally, the data part in the sliding window satisfies a condition, including but not limited to that the last 1B data in the data part in the sliding window includes 8 bits with a value of 0, or that a first value (such as a hash value) calculated according to the data part in the sliding window can be evenly divided by a second value (such as 5), and the like. The condition can be determined according to experience or actual needs, and the embodiments of the present application do not limit the condition.

[0144] For example, the length of the first data is 20 KB, the range of the slicing length is 6 KB (lower limit) to 10 KB (upper limit), and the length of the sliding window is 6 KB. In the first slicing process, the sliding window ends sliding when the upper limit is located at 8 KB, and then a first data block with a length of 8 KB is obtained, and the first data block includes data from 1 KB to 8 KB in the first data. In the second slicing process, the sliding window ends sliding when the upper limit is located at 18 KB, and then a second data block with a length of 10 KB is obtained, and the second data block includes data from 9 KB to 18 KB in the first data. The remaining data is taken as the last data block, and then a last data block with a length of 2 KB is obtained, and the last data block includes data from 19 KB to 20 KB in the first data.

[0145] The second slicing manner corresponds to the case that the number of characteristic values of the first data in step 302 is multiple (i.e., the characteristic values of the first data include the characteristic values of multiple second data), and the range of the slicing length is determined multiple times.

[0146] In one example, a slice length is determined from each slice length range respectively, to obtain a plurality of slice lengths, and the first data is sliced according to the plurality of slice lengths to obtain the sliced data. For example, the length of the first data is 20 KB, a slice length range 1 is determined according to the characteristic value 1 of the second data A, and a slice length 1 of 8 KB is determined from the slice length range 1. In addition, a slice length range 2 is determined according to the characteristic value 2 of the second data B, and a slice length 2 of 12 KB is determined from the slice length range 2. Therefore, the first data can be sliced according to 8 KB to obtain one data block with a length of 8 KB. In addition, the remaining part of the first data can also be sliced according to 12 KB to obtain one data block with a length of 12 KB.

[0147] In another example, the first data is sliced according to the slice length range to obtain the sliced data, including: the first data is sliced according to the slice length range and the position of the second data in the first data to obtain the sliced data. For example, for the second data, the first data is sliced according to the slice length range corresponding to the second data from the starting position of the second data in the first data as the starting point to obtain the sliced data, and the starting position of the sliced data is the same as the starting position of the second data in the first data. The slice length range corresponding to the second data refers to the slice length range determined according to the characteristic value of the second data.

[0148] Since the slice length range is determined according to the characteristic value of the second data, slicing according to the slice length range and the position of the second data in the first data can ensure the accuracy of slicing, so that the length of the data block obtained after slicing is more appropriate. For example, the first data is sliced according to the sliding window in each slice length range to obtain the sliced data. The way of slicing the first data according to the sliding window in the slice length range can be referred to the description of the first slicing method, which will not be repeated here.

[0149] Exemplarily, the process of acquiring the second data, acquiring the characteristic value of the second data, and determining the range of the slice length (referred to as process 1 for short), and the process of performing the slice on the first data according to the range of the slice length and the position of the second data in the first data (referred to as process 2 for short) can be alternately performed. That is, the embodiment of the present application can first perform process 1, acquire the first second data from the first data, acquire the characteristic value of the first second data, determine the range of the slice length according to the characteristic value of the first second data, and then perform process 2 according to the above description, perform the first slice on the first data according to the range of the slice length and the position of the first second data in the first data, obtain the first data block and the first remaining data, and the first remaining data is the data in the first data except the first data block. Then, process 1 and process 2 are repeatedly performed to acquire the second second data from the first remaining data and perform the second slice to obtain the second data block and the second remaining data, and the second remaining data is the data in the first remaining data except the second data block. In this way, the processing of the first data is completed.

[0150] For example, the length of the first data is 20 KB, the first second data is the first 1-4 KB (the starting position is the first KB) in the first data, the range of the slice length determined according to the first second data is 4-8 KB, the first data block obtained includes the first 1-6 KB in the first data, and the first remaining data includes the 7-20 KB in the first data. The second second data is the 7-10 KB (the starting position is the 7 KB) in the first remaining data, the range of the slice length determined according to the second second data is 12-16 KB, and the second data block obtained includes the 7-20 KB in the first data.

[0151] The above two slice methods are only examples and are not used to limit the slice method in the embodiment of the present application. Exemplarily, the slice method can be a content defined chunking (CDC) method.

[0152] In step 304, at least one of the re-deletion or compression is performed on the sliced data to obtain the processed data.

[0153] As described above, the data after slicing includes a plurality of data blocks, and at least one of deduplication or compression is performed on the plurality of data blocks to obtain processed data. That is, for the plurality of data blocks, only deduplication processing is performed, or only compression processing is performed, or both deduplication processing and compression processing are performed. In the case of performing both deduplication processing and compression processing, the deduplication processing can be performed first, and then the compression processing is performed. Taking an example in which the plurality of data blocks include data block 1, data block 2, and data block 3, data block 2 is the same as data block 1, and data block 3 is different from data block 1, so data block 2 and data block 1 are duplicate data blocks. In the process of deduplication processing, data block 1 is retained, data block 2 is deleted, a pointer is added at the position of data block 2, the pointer is used to reference data block 1, and data block 3 is retained. Therefore, the data after deduplication processing includes data block 1, the pointer, and data block 3. In the process of compression processing, the compression processing is performed on the data after deduplication processing by using a compression algorithm to obtain processed data.

[0154] Exemplarily, the manner in which the embodiment of the present application performs processing on the data after slicing can include the following six manners.

[0155] The first processing manner is that the characteristic value represents the compression rate, deduplication processing and compression processing are performed on the data after slicing to obtain processed data. The first processing manner is applicable to the first case (including the first example and the second example) in step 302, and the first case takes into account both the deduplication rate and the compression rate, and can ensure that the processed data has a high reduction rate compared with the first data.

[0156] The second processing manner is that the characteristic value represents the deduplication rate, deduplication processing and compression processing are performed on the data after slicing to obtain processed data. The second processing manner is applicable to the second case (including the third example and the fourth example) in step 302, and the second case takes into account both the deduplication rate and the compression rate, and can ensure that the processed data has a high reduction rate compared with the first data.

[0157] The third processing manner is that the characteristic value represents the compression rate, and only deduplication processing is performed on the data after slicing to obtain processed data. The third processing manner is applicable to the first example in step 302, and the first example can ensure that the processed data has a high deduplication rate.

[0158] The fourth processing manner is that the characteristic value represents the compression rate, and only compression processing is performed on the data after slicing to obtain processed data. The fourth processing manner is applicable to the second example in step 302, and the second example can ensure that the processed data has a high compression rate.

[0159] The fifth processing manner is that the characteristic value represents the deduplication rate, and only compression processing is performed on the data after slicing to obtain processed data. The fifth processing manner is applicable to the third example in step 302, and the third example can ensure that the processed data has a high compression rate.

[0160] The sixth processing manner is that the characteristic value represents the deduplication rate, and only deduplication processing is performed on the sliced data to obtain the processed data. The sixth processing manner is suitable for the fourth example in step 302, and the fourth example can ensure that the processed data has a high deduplication rate.

[0161] In the example embodiments, according to different system architectures, the embodiments of the present application can further perform other processing on the processed data, which will be described in detail as follows.

[0162] For the first system architecture, i.e., the storage system, the method further includes storing the processed data. The processed data only needs to occupy a small amount of storage space, which is beneficial to saving storage space.

[0163] In one example, the first data is uncompressed original data. For example, the first data is received original data, or the first data is generated original data. Referring to Figure 7 , after the data processing unit receives or generates the original data (i.e., the first data), (12) the processing is performed according to the description in steps 301 to 304 to obtain the processed data, and (13) the processed data is sent to the storage unit so that the storage unit stores the processed data.

[0164] In another example, the first data is obtained by decompressing compressed data. Referring to Figure 7 , (21) the data processing unit obtains the compressed data from the storage unit, (22) the compressed data is decompressed to obtain the decompressed data (i.e., the first data), and (23) the processing is performed according to the description in steps 301 to 304 to obtain the processed data, and (24) the processed data is sent to the storage unit (not shown in Figure 7 ) so that the storage unit stores the processed data.

[0165] Decompressing the compressed data to obtain the decompressed data will cause data expansion. Data expansion refers to that the number of bytes of the decompressed data is greater than the number of bytes of the compressed data. Among them, the data expansion multiple is different, and the characteristic value of the decompressed data is also different. Since the embodiments of the present application can accurately slice the data according to the characteristic value to obtain the sliced data (i.e., the plurality of data blocks), the length of the plurality of data blocks is related to the data expansion multiple, so that the number of the plurality of data blocks is also related to the data expansion multiple, and then the number of the plurality of data blocks under different data expansion multiples is relatively balanced, avoiding the situation that the number of the plurality of data blocks is too large under the condition of large data expansion multiple, and the number of the plurality of data blocks is too small under the condition of small data expansion multiple, and avoiding affecting the overall performance of the storage system.

[0166] In one example, the greater the data expansion multiple, the greater the compression rate represented by the eigenvalue of the data, and according to the foregoing description, the compression rate represented by the eigenvalue is positively correlated with the average value of the range of the slice length, so the average value of the range of the slice length is greater, the length of the data block is greater, and the number of the plurality of data blocks is less, avoiding the case that the number of the plurality of data blocks is too large when the data expansion multiple is large. Alternatively, the smaller the data expansion multiple, the smaller the compression rate represented by the eigenvalue of the data, so the average value of the range of the slice length is smaller, the length of the data block is smaller, and the number of the plurality of data blocks is greater, avoiding the case that the number of the plurality of data blocks is too small when the data expansion multiple is small.

[0167] In another example, the greater the data expansion multiple, the greater the compression rate of the data, and according to the foregoing description, the compression rate is relative to the deduplication rate, and the greater the compression rate of the data, the smaller the deduplication rate of the data, so the eigenvalue of the data represents the smaller deduplication rate, and according to the foregoing description, the deduplication rate represented by the eigenvalue is negatively correlated with the average value of the range of the slice length, so the greater the average value of the range of the slice length, the greater the length of the data block, and the number of the plurality of data blocks is less, avoiding the case that the number of the plurality of data blocks is too large when the data expansion multiple is large. Alternatively, the smaller the data expansion multiple, the greater the eigenvalue of the data represents the deduplication rate, so the average value of the range of the slice length is smaller, the length of the data block is smaller, and the number of the plurality of data blocks is greater, avoiding the case that the number of the plurality of data blocks is too small when the data expansion multiple is small.

[0168] For the second system architecture described above, i.e., the communication system, the method further comprises: transmitting the processed data. The processed data only needs to occupy less bandwidth, and the time required for transmitting the processed data is also shorter. In combination with Figure 2 For the second system architecture shown, after the data processing unit receives or generates the first data, it performs processing according to the description in steps 301 to 304 to obtain processed data, and transmits the processed data to the transmission unit, so that the transmission unit transmits the processed data.

[0169] For the third system architecture described above, i.e., the combination of the storage system and the communication system, the method further comprises: storing the processed data and transmitting the processed data. The description of the storage process and the transmission process can be referred to the foregoing description, which will not be repeated here.

[0170] In summary, the embodiment of the present application obtains the characteristic value of the first data, which represents the compression rate or the deduplication rate of the first data, and embodies the characteristics of the first data itself. Then, the range of the slice length is determined according to the characteristic value, and the first data is sliced according to the range of the slice length to obtain the sliced data. Since the characteristic value embodies the characteristics of the first data itself and has a strong correlation with the first data itself, the range of the slice length determined according to the characteristic value is relatively accurate, and after the slicing is performed according to the range of the slice length, the sliced data has a relatively appropriate length. Therefore, at least one of the deduplication and the compression is performed on the sliced data to obtain the processed data, which has a high data reduction rate.

[0171] The above introduces the method for processing data provided by the embodiment of the present application. Corresponding to the above method, the embodiment of the present application also provides a device for processing data. The device is used to obtain the characteristic value of the first data by Figure 8 The device is used to perform the above Figure 3 The device is used to perform the above Figure 8 The device for processing data provided by the embodiment of the present application includes the following modules.

[0172] The obtaining module 801 is configured to obtain the characteristic value of the first data, and the characteristic value represents the compression rate or the deduplication rate of the first data.

[0173] The determining module 802 is configured to determine the range of the slice length according to the characteristic value.

[0174] The slicing module 803 is configured to perform slicing on the first data according to the range of the slice length to obtain the sliced data.

[0175] The processing module 804 is configured to perform at least one of the deduplication and the compression on the sliced data to obtain the processed data.

[0176] In the example embodiment, the obtaining module 801 is configured to obtain the characteristic value of the second data, and the characteristic value represents the compression rate or the deduplication rate of the second data, and the second data is part of the first data; and the slicing module 803 is configured to perform slicing on the first data according to the range of the slice length and the position of the second data in the first data to obtain the sliced data.

[0177] In the example embodiment, the average value of the range of the slice length is determined according to the characteristic value; and the range of the slice length is determined according to the average value.

[0178] For example, the determining module 802 is configured to determine the average value of the range of slice lengths according to a first determining method based on feature values; and to determine the range of slice lengths according to a second determining method based on the average value. Wherein, at least one of the first or second determining methods is obtained based on the reduction rate of the third data preceding the first data.

[0179] For example, the compression ratio represented by the eigenvalues ​​is positively correlated with the average value of the range of slice lengths.

[0180] In an exemplary embodiment, the feature value is information entropy, which characterizes the compression ratio.

[0181] For example, the deduplication rate represented by the eigenvalues ​​is negatively correlated with the average value of the range of slice lengths.

[0182] In an exemplary embodiment, the apparatus further includes a storage module for storing processed data.

[0183] For example, the first data is obtained by decompressing the compressed data.

[0184] In an exemplary embodiment, the apparatus further includes a transmitting module for transmitting processed data.

[0185] It should be understood that the above Figure 8 The device shown, in performing its function, possesses beneficial effects and Figure 3 The methods shown have the same beneficial effects. Figure 8 The device illustrated here is only an example of the division of the above-described functional modules to demonstrate its functions. In practical applications, the functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. Furthermore, the device and method embodiments provided in the above examples belong to the same concept, and their specific implementation processes are detailed in the method embodiments, which will not be repeated here.

[0186] For example, embodiments of this application also provide a data processing device, which includes a memory and a processor; the memory stores at least one computer instruction, and the at least one computer instruction is loaded and executed by the processor to enable the device to perform... Figure 3 The method shown is for processing data.

[0187] See Figure 9 , Figure 9 A schematic diagram of the structure of an exemplary data processing device 900 of this application is shown. The data processing device 900 includes at least one processor 901, a memory 903, and at least one network interface 904.

[0188] The processor 901 is, for example, a general-purpose central processing unit (CPU), a digital signal processor (DSP), a network processer (NP), a graphics processing unit (GPU), a neural-network processing units (NPU), a data processing unit (DPU), a microprocessor, or one or more integrated circuits or application-specific integrated circuits (ASICs) for implementing the schemes of the present application, programmable logic devices (PLDs), other general-purpose processors, or other programmable logic devices, discrete gates or transistor logic, discrete hardware components, or any combination thereof. The PLD is, for example, a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof. The general-purpose processor can be a microprocessor or any conventional processor, etc. It is worth noting that the processor can be a processor supporting an advanced RISC machine (ARM) architecture. It can implement or execute various logical blocks, modules, and circuits described in connection with the disclosure of the present application. The processor can also be a combination implementing a computing function, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0189] Optionally, the data processing device 900 further includes a bus 902. The bus 902 is used to transmit information between the components of the data processing device 900. The bus 902 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus 902 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 9 Only one line is used in the figure, but it does not mean that there is only one bus or only one type of bus.

[0190] The memory 903 is, for example, a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable ROM (PROM), an erasable PROM (EPROM), an electrically EPROM (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM) used as an external cache.

[0191] By way of example, and not limitation, a number of forms of ROM and RAM are possible. For example, the ROM is a compact disc read-only memory (CD-ROM). The RAM includes, but is not limited to, a static RAM (SRAM), a dynamic RAM (DRAM), a synchronous DRAM (SDRAM), a double data rate SDRAM (DDR SDRAM), an enhanced SDRAM (ESDRAM), a synchlink DRAM (SLDRAM), and a direct rambus RAM (DR RAM).

[0192] The memory 903 can also be other types of storage devices capable of storing static and dynamic information and instructions. Or it can be other types of dynamic storage devices capable of storing information and instructions. Or it can be other optical disc storage, optical disc storage (including compact disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), magnetic disk storage medium or other magnetic storage device, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and capable of being accessed by a computer, but not limited to. The memory 903 is, for example, independent and connected to the processor 901 through the bus 902. The memory 903 can also be integrated with the processor 901.

[0193] The network interface 904 uses any transceiver-type device to communicate with other devices or communication networks, which can be an Ethernet, a radio access network (RAN), a wireless local area network (WLAN), etc. The network interface 904 can include a wired network interface and can also include a wireless network interface. Specifically, the network interface 904 can be an Ethernet interface, such as a fast Ethernet (FE) interface, a gigabit Ethernet (GE) interface, an asynchronous transfer mode (ATM) interface, a WLAN interface, a cellular network interface, or a combination thereof. The Ethernet interface can be an optical interface, an electrical interface, or a combination thereof. In some embodiments of the present application, the network interface 904 can be used for the data processing device 900 to communicate with other devices.

[0194] In a particular implementation, as some embodiments, the processor 901 can include one or more CPUs, such as the CPU0 and the CPU1 shown in Figure 9 Each of these processors can be a single core processor or a multiple core processor. Here, a processor can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0195] In a particular implementation, as some embodiments, the data processing device 900 can include multiple processors, such as the processor 901 and the processor 905 shown in Figure 9 Each of these processors can be a single core processor or a multiple core processor. Here, a processor can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0196] In some embodiments, the memory 903 is used to store program instructions 910 for implementing the solutions of the present application, and the processor 901 can execute the program instructions 910 stored in the memory 903. That is, the data processing device 900 can implement the method provided by the method embodiments through the processor 901 and the program instructions 910 in the memory 903, i.e., the method shown in Figure 3 The program instructions 910 can include one or more software modules. Alternatively, the processor 901 itself can also store program instructions for implementing the solutions of the present application.

[0197] In a specific implementation process, the processing data device 900 of the present application can correspond to a data processing unit for executing the above-mentioned method, and the processor 901 in the processing data device 900 reads the instructions in the storage 903, so that Figure 9 The processing data device 900 shown in the figure can execute all or part of the steps in the method embodiment.

[0198] The processing data device 900 can also correspond to the above-mentioned Figure 8 The device shown in the figure, Figure 8 Each functional module in the device shown is implemented by software of the processing data device 900. In other words, Figure 8 The functional modules included in the device shown are generated after the processor 901 of the processing data device 900 reads the program instructions 910 stored in the storage 903.

[0199] Figure 3 The steps of the method shown are completed by the integrated logic circuit of the hardware in the processor of the processing data device 900 or the instructions in the form of software. The steps of the method embodiment disclosed in the present application can be directly embodied as execution completed by a hardware processor, or executed by a combination of hardware and software modules in the processor. The software module can be located in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, register, etc. The storage medium is located in the storage, and the processor reads the information in the storage, and combines the hardware to complete the steps of the above-mentioned method embodiment. To avoid repetition, it will not be described in detail here.

[0200] Exemplarily, the embodiment of the present application also provides a storage system, which comprises a data processing unit and a storage unit. Wherein, the data processing unit is used to execute Figure 3 The processing data method shown in the figure, to obtain the processed data. The storage unit is used to store the processed data.

[0201] Exemplarily, the embodiment of the present application also provides a computer readable storage medium, which stores at least one computer instruction. The at least one computer instruction is loaded and executed by the processor, so that the computer comprising the processor implements Figure 3 The processing data method shown in the figure.

[0202] Exemplarily, the embodiment of the present application also provides a computer program product, which comprises computer instructions. The computer instructions are executed by the processor, so that the computer comprising the processor implements Figure 3 The processing data method shown in the figure.

[0203] In some embodiments, the embodiments of the present application also provide a chip comprising a processor configured to call and run instructions stored in a memory, so that a computer installed with the chip executes Figure 3 the method for processing data shown.

[0204] In some embodiments, the embodiments of the present application also provide a chip comprising a processor configured to call and run instructions stored in a memory, so that a computer installed with the chip executes Figure 3 the method for processing data shown.

[0205] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to the embodiments of the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line) or wireless (such as infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. containing one or more available media sets. The available media can be magnetic media (such as floppy disk, hard disk, magnetic tape), optical media (such as DVD), or semiconductor media (such as solid state disk), etc.

[0206] In the present application, the terms "first", "second" and the like are used to distinguish between elements or items that have substantially the same function and are similar to each other, and it should be understood that there is no logical or time sequence relationship between "first", "second", "nth", and the number and execution order are not limited. It should also be understood that although the following description uses the terms first, second, etc. to describe various elements, these elements should not be limited by the terms. These terms are only used to distinguish one element from another.

[0207] It should also be understood that, in various embodiments of the present application, the size of the sequence number of various processes does not mean the order of execution, the execution order of various processes should be determined according to its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0208] The term "at least one" in the present application means one or more, and the term "multiple" in the present application means two or more, for example, multiple data blocks refer to two or more data blocks. The terms "system" and "network" are often used interchangeably herein.

[0209] It should be understood that the terms used in the description of various examples herein are only for the purpose of describing specific examples, and are not intended to be limiting. As used in the description of various examples and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0210] It should also be understood that the term "and / or" used herein means and encompasses any and all possible combinations of one or more of the associated listed items. The term "and / or", is a description of the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent the three cases of A alone, A and B exist at the same time, and B alone. In addition, the character " / " in the present application generally represents that the associated objects before and after are in an "or" relationship.

[0211] It should also be understood that the terms "if" and "when" can be interpreted to mean "when" or "upon" or "in response to a determination" or "in response to detecting". Similarly, the phrase "if determined" or "if detecting [stated condition or event]" can be interpreted to mean "upon determining" or "in response to determining" or "upon detecting [stated condition or event]" or "in response to detecting [stated condition or event]", depending on the context.

[0212] The above is only an embodiment of the present application, and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for processing data, characterized in that, The method includes: Obtain the feature value of the first data, wherein the feature value characterizes the compression ratio or deduplication ratio of the first data; The range of slice length is determined based on the aforementioned feature values; The first data is sliced ​​according to the range of the slice length to obtain the sliced ​​data; Perform at least one of the following processes, namely deduplication or compression, on the sliced ​​data to obtain the processed data.

2. The method according to claim 1, characterized in that, The acquisition of the feature values ​​of the first data includes: Obtain feature values ​​of the second data, wherein the feature values ​​characterize the compression ratio or deduplication ratio of the second data, and the second data is a portion of the first data; The step of slicing the first data according to the range of the slice length to obtain the sliced ​​data includes: Based on the range of the slice length and the position of the second data in the first data, the first data is sliced ​​to obtain the sliced ​​data.

3. The method according to claim 1 or 2, characterized in that, Determining the range of slice length based on the feature value includes: The average value of the range of slice lengths is determined based on the characteristic values; The range of slice lengths is determined based on the average value.

4. The method according to claim 3, characterized in that, The step of determining the average value of the range of slice lengths based on the feature values ​​includes: Based on the characteristic value, the average value of the range of slice lengths is determined according to the first determination method; Determining the range of slice lengths based on the average value includes: Based on the average value, the range of the slice length is determined according to the second determination method; Wherein, at least one of the first determination method or the second determination method is obtained based on the reduction rate of the third data preceding the first data.

5. The method according to any one of claims 1-4, characterized in that, The compression ratio, characterized by the eigenvalue, is positively correlated with the average value of the range of slice lengths.

6. The method according to any one of claims 1-5, characterized in that, The feature value is information entropy, which characterizes the compression ratio.

7. The method according to any one of claims 1-4, characterized in that, The deduplication rate, characterized by the eigenvalue, is negatively correlated with the average value of the range of slice lengths.

8. A data processing apparatus, characterized in that, The device includes: The acquisition module is used to acquire feature values ​​of the first data, wherein the feature values ​​characterize the compression ratio or deduplication ratio of the first data; The determination module is used to determine the range of slice length based on the feature values; The slicing module is used to slice the first data according to the range of the slice length to obtain the sliced ​​data; The processing module is used to perform at least one of the following processes on the sliced ​​data: deduplication or compression, to obtain processed data.

9. The apparatus according to claim 8, characterized in that, The acquisition module is used to acquire feature values ​​of the second data, the feature values ​​representing the compression rate or deduplication rate of the second data, the second data being a portion of the first data; The slicing module is used to slice the first data according to the range of the slice length and the position of the second data in the first data to obtain sliced ​​data.

10. The apparatus according to claim 8 or 9, characterized in that, The determining module is used to determine the average value of the range of slice lengths based on the feature values; and to determine the range of slice lengths based on the average value.

11. The apparatus according to claim 10, characterized in that, The determining module is configured to determine the average value of the range of slice lengths according to the feature value and a first determining method; and to determine the range of slice lengths according to the average value and a second determining method. Wherein, at least one of the first determination method or the second determination method is obtained based on the reduction rate of the third data preceding the first data.

12. The apparatus according to any one of claims 8-11, characterized in that, The compression ratio, characterized by the eigenvalue, is positively correlated with the average value of the range of slice lengths.

13. The apparatus according to any one of claims 8-12, characterized in that, The feature value is information entropy, which characterizes the compression ratio.

14. The apparatus according to any one of claims 8-11, characterized in that, The deduplication rate, characterized by the eigenvalue, is negatively correlated with the average value of the range of slice lengths.

15. A storage system, characterized in that, The storage system includes a data processing unit and a storage unit; The data processing unit is used to execute the method described in any one of claims 1-7 to obtain processed data; The storage unit is used to store the processed data.

16. A data processing device, characterized in that, The device includes a memory and a processor; the memory stores at least one computer instruction, which is loaded and executed by the processor to enable the device to implement the data processing method according to any one of claims 1-7.

17. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one computer instruction, which is loaded and executed by a processor to enable the computer to implement the method of processing data as described in any one of claims 1-7.

18. A computer program product, characterized in that, The computer program product includes computer instructions that are executed by a processor to cause the computer to perform the data processing method according to any one of claims 1-7.

Images