Data processing method and apparatus, device, and computer-readable storage medium

Abstract

The present application relates to the technical field of computers, and discloses a data processing method and apparatus, a device, and a computer-readable storage medium. The method comprises: acquiring a feature value of first data, the feature value representing a compression rate or a deduplication rate of the first data; then determining a slice length range on the basis of the feature value, and performing slicing on the first data on the basis of the slice length range to obtain sliced data; and performing at least one of deduplication or compression on the sliced data to obtain processed data. A feature value reflects the characteristics of first data, so that a slice length range determined on the basis of the feature value is relatively accurate. Sliced data obtained by performing slicing on the basis of the slice length range can have a relatively proper length, so that processed data obtained by performing at least one of deduplication or compression on the sliced data has a relatively high reduction rate.

Description

Methods, apparatus, devices and computer-readable storage media for processing data

[0001] This application claims priority to Chinese Patent Application No. 202410939795.6, filed on July 12, 2024, entitled "Method, Apparatus, Device and Computer-Readable Storage Medium for Processing Data", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of computer technology, and in particular to methods, apparatus, devices and computer-readable storage media for processing data. Background Technology

[0003] With the development of computer technology, there are more and more ways to process data. Performing operations such as deduplication or compression on data to reduce its size is one such method.

[0004] In related technologies, a fixed length is determined based on the number of times the data is accessed or modified. The data is then sliced ​​according to this fixed length to obtain the sliced ​​data. Deduplication and compression are then performed on the sliced ​​data. For example, if the fixed length is determined to be 4 kilobytes (KB) and the total length of the data is 20KB, then the sliced ​​data consists of five 4KB data slices. Deduplication or compression operations can then be performed on these five 4KB data slices.

[0005] However, a fixed length determined based on the number of times the data is accessed or modified is not accurate enough, resulting in a low data reduction rate. Summary of the Invention

[0006] This application provides a method, apparatus, device, and computer-readable storage medium for processing data to improve the problem of low data reduction rates. The technical solution provided by this application includes the following aspects.

[0007] Firstly, a method for processing data is provided. In this method, feature values ​​of first data are obtained, whereby the feature values ​​characterize the compression ratio or deduplication ratio of the first data. Then, a range of slice lengths is determined based on the feature values, and the first data is sliced ​​according to this range to obtain sliced ​​data. Next, at least one of deduplication or compression processing is performed on the sliced ​​data to obtain processed data.

[0008] In this process, the feature values ​​of the first data characterize the compression ratio or deduplication ratio of the first data, reflecting the inherent characteristics of the first data. Then, the range of slice lengths is determined based on the feature values, and the first data is sliced ​​according to this range to obtain sliced ​​data. Because the feature values ​​reflect the inherent characteristics of the first data and have a strong correlation with the data itself, the range of slice lengths determined based on the feature values ​​is relatively accurate. Slicing according to this range results in sliced ​​data with a suitable length. Therefore, performing at least one of deduplication or compression on the sliced ​​data yields processed data with a high data reduction rate.

[0009] In one possible implementation, obtaining the feature values ​​of the first data includes: obtaining the feature values ​​of the second data, whereby the feature values ​​characterize the compression ratio or deduplication ratio of the second data. Correspondingly, slicing the first data according to a range of slice lengths to obtain sliced ​​data includes: slicing the first data according to the range of slice lengths and the position of the second data within the first data to obtain sliced ​​data.

[0010] Since the second data is a part of the first data, it is smaller than the first data. Therefore, obtaining the feature values ​​of the second data requires less computation and is more efficient, which improves data processing efficiency. Furthermore, since the slice length range is determined by the feature values ​​of the second data, performing slicing based on this range and the position of the second data within the first data ensures accuracy and results in a more suitable length for the sliced ​​data.

[0011] In one possible implementation, determining the range of slice lengths based on eigenvalues ​​includes: determining the average value of the range of slice lengths based on eigenvalues; and determining the range of slice lengths based on the average value.

[0012] This method can easily, quickly, and accurately determine the range of slice lengths based on feature values, ensuring the accuracy and efficiency of determining the range of slice lengths, which is beneficial to improving the accuracy and efficiency of data processing.

[0013] In one possible implementation, determining the average value of the range of slice lengths based on feature values ​​includes: determining the average value of the range of slice lengths according to a first determination method based on feature values; determining the range of slice lengths based on the average value includes: determining the range of slice lengths according to a second determination method based on the average value. Wherein, at least one of the first or second determination methods is obtained based on the reduction rate of the third data preceding the first data.

[0014] The first and second determination methods are both methods that may be used in determining the range of slice lengths. Updating the determination method based on the reduction rate of the third data ensures the accuracy of the determination method, thereby ensuring the accuracy of the range of slice lengths determined using the determination method.

[0015] In one possible implementation, the compression ratio represented by the eigenvalues ​​is positively correlated with the average value of the range of slice lengths.

[0016] When the compression ratio represented by eigenvalues ​​is low, it can be assumed that even increasing the average range of slice lengths, thus increasing the length of data blocks, may not significantly improve the compression ratio of the first data. Therefore, it is better to decrease the average range of slice lengths, thus reducing the length of data blocks and significantly improving the deduplication rate of the first data, thereby ensuring a higher reduction rate for the first data. Thus, the compression ratio represented by eigenvalues ​​can be positively correlated with the average range of slice lengths.

[0017] In one possible implementation, the characteristic value is information entropy, which characterizes the compression rate.

[0018] Information entropy can accurately characterize the compression rate. Therefore, using information entropy as a feature value can ensure that the range of slice lengths determined based on the feature value has high accuracy.

[0019] In one possible implementation, the deduplication rate represented by the eigenvalues ​​is negatively correlated with the average value of the range of slice lengths.

[0020] When the deduplication rate represented by the eigenvalue is low, it can be assumed that even reducing the average range of slice lengths, thus decreasing the data block length, may not significantly improve the deduplication rate of the first data. Therefore, it is better to increase the average range of slice lengths, thus increasing the data block length, which would significantly improve the compression rate of the first data and ensure a high reduction rate. Thus, the deduplication rate represented by the eigenvalue can be negatively correlated with the average range of slice lengths.

[0021] In one possible implementation, the method also includes storing the processed data.

[0022] The processed data has a high reduction rate, and storing the processed data only requires less storage space, which improves the utilization of storage space and reduces storage costs.

[0023] In one possible implementation, the first data is obtained by decompressing the already compressed data.

[0024] In this implementation, after decompressing the compressed data to obtain the first data, the processing of the first data is performed, which ensures the reduction rate of the first data and has a wide range of applications.

[0025] In one possible implementation, the method also includes sending the processed data.

[0026] The processed data has a high reduction rate, and sending the processed data only requires less bandwidth, thus improving sending efficiency.

[0027] Secondly, an apparatus for processing data is provided, the apparatus comprising:

[0028] The acquisition module is used to acquire the feature values ​​of the first data, where the feature values ​​represent the compression ratio or deduplication ratio of the first data.

[0029] The determination module is used to determine the range of slice lengths based on feature values;

[0030] The slicing module is used to slice the first data according to the range of slice length to obtain the sliced ​​data;

[0031] The processing module is used to perform at least one of the following processes on the sliced ​​data: deduplication or compression, to obtain the processed data.

[0032] In one possible implementation, an acquisition module is used to acquire feature values ​​of the second data, the feature values ​​representing the compression ratio or deduplication ratio of the second data; a slicing module is used to slice the first data according to the range of slice length and the position of the second data in the first data to obtain sliced ​​data.

[0033] In one possible implementation, the average value is used to determine the range of slice lengths based on the eigenvalues; the range of slice lengths is determined based on the average value.

[0034] In one possible implementation, a determining module is configured to determine, based on feature values, the average value of the range of slice lengths according to a first determining method; and, based on the average value, determine the range of slice lengths according to a second determining method. Wherein, at least one of the first or second determining methods is obtained based on the reduction rate of third data preceding the first data.

[0035] In one possible implementation, the compression ratio represented by the eigenvalues ​​is positively correlated with the average value of the range of slice lengths.

[0036] In one possible implementation, the characteristic value is information entropy, which characterizes the compression rate.

[0037] In one possible implementation, the deduplication rate represented by the eigenvalues ​​is negatively correlated with the average value of the range of slice lengths.

[0038] In one possible implementation, the device further includes a storage module for storing the processed data.

[0039] In one possible implementation, the first data is obtained by decompressing the already compressed data.

[0040] In one possible implementation, the apparatus further includes a transmitting module for transmitting the processed data.

[0041] Thirdly, a storage system is provided, comprising a data processing unit and a storage unit. The data processing unit executes a data processing method provided in the first aspect or any possible implementation thereof to obtain processed data. The storage unit stores the processed data.

[0042] Fourthly, a data processing apparatus is provided, the apparatus including a memory and a processor; the memory stores at least one computer instruction, the at least one computer instruction being loaded and executed by the processor to cause the apparatus to implement the data processing method provided in the first aspect or any possible implementation of the first aspect.

[0043] Optionally, there may be one or more processors and one or more memories.

[0044] Alternatively, the memory can be integrated with the processor, or the memory can be set up separately from the processor.

[0045] Fifthly, a computer-readable storage medium is provided, wherein at least one computer instruction is stored therein, the at least one computer instruction being loaded and executed by a processor to enable a computer including the processor to implement the method for processing data provided in the first aspect or any possible implementation thereof.

[0046] In a sixth aspect, a computer program product is provided, the computer program product including computer instructions that are executed by a processor to cause a computer including a processor to implement the method for processing data provided in the first aspect or any possible implementation of the first aspect.

[0047] In a seventh aspect, a chip is provided, including a processor for retrieving and executing instructions stored in a memory, causing a computer equipped with the chip to perform a method for processing data provided by the first aspect or any possible implementation thereof.

[0048] Eighthly, another chip is provided, comprising: an input interface, an output interface, a processor, and a memory, wherein the input interface, the output interface, the processor, and the memory are connected via an internal connection path, the processor is used to execute code in the memory, and when the code is executed, a computer with the chip installed executes the data processing method provided by the first aspect or any possible implementation thereof.

[0049] It should be understood that the technical effects achieved by the technical solutions provided by the second to eighth aspects of this application and their corresponding possible implementations can be found in the above description of the technical effects achieved by the technical solutions provided by the first aspect and their corresponding possible implementations, and will not be repeated here. Attached Figure Description

[0050] Figure 1 is a schematic diagram of a system architecture provided in an embodiment of this application;

[0051] Figure 2 is a schematic diagram of another system architecture provided in an embodiment of this application;

[0052] Figure 3 is a flowchart of a data processing method provided in an embodiment of this application;

[0053] Figure 4 is a schematic diagram of an embodiment of this application providing an average value for determining the range of slice lengths;

[0054] Figure 5 is a schematic diagram of another method for determining the average value of the range of slice lengths provided in an embodiment of this application;

[0055] Figure 6 is a schematic diagram illustrating a method for determining the range of slice length according to an embodiment of this application;

[0056] Figure 7 is a schematic diagram of the data processing flow in a storage system architecture provided in an embodiment of this application;

[0057] Figure 8 is a schematic diagram of a data processing device provided in an embodiment of this application;

[0058] Figure 9 is a schematic diagram of the structure of a data processing device provided in an embodiment of this application. Detailed Implementation

[0059] The terminology used in the implementation section of this application is for the purpose of explaining specific embodiments of this application only, and is not intended to limit this application.

[0060] Data reduction is a method of data processing used to reduce the amount of data. For example, data reduction can be used to reduce the number of bytes (B) in a data set.

[0061] In one example, data reduction includes data deduplication, or decoupling. During deduplication, the data is sliced, resulting in sliced ​​data consisting of multiple data blocks (or data slices). Among the duplicate data blocks within these slices, only one block is retained, or only one block and a copy of that block are retained (equivalent to retaining two blocks). The remaining blocks are then referenced from any of the retained blocks after deletion. Because data blocks are deleted, the number of bytes in the data is reduced. Different data blocks are considered duplicates if their content is identical, or if their unique identifiers are identical. The unique identifier can be determined based on the content or other feasible methods. For the deduplication process, the larger the data block length (i.e., the more bytes in the block), the lower the probability of duplicate data blocks, resulting in fewer duplicate blocks and a lower deduplication rate.

[0062] In another example, data reduction includes data compression. During compression, compression algorithms (or techniques) analyze redundant parts of the data or analyze the characteristics of the data to express it in a more concise way, thereby reducing the number of bytes. When data is sliced, the compression process can be performed on units of data blocks comprised of the sliced ​​data. For the compression process, the larger the data block length, i.e., the more bytes in the data block, the easier it is to analyze redundant parts and data characteristics; therefore, the higher the compression ratio. Compression ratio and deduplication rate are relative.

[0063] For situations requiring data slicing, related technologies treat the data as a single file, determining a fixed length based on file-level usage (such as file access frequency or modification frequency), and then slicing the data according to this fixed length to obtain multiple data blocks of the same length. These blocks are then subjected to deduplication or compression. However, file-level usage only reflects the frequency of user file usage and has little correlation with the data itself. Therefore, the fixed length determined based on file-level usage is not accurate enough, resulting in unsuitable data block lengths after slicing, leading to a low data reduction rate.

[0064] In response, this application provides a method for processing data. This method starts from the perspective of the data itself. After obtaining the feature values ​​of the data to be processed, the data is accurately sliced ​​based on the feature values ​​to obtain sliced ​​data. If the length of the sliced ​​data is relatively appropriate, then at least one of the deduplication or compression processes is performed on the sliced ​​data to obtain processed data. The processed data has fewer bytes, resulting in a higher data reduction rate.

[0065] Before describing the method provided in the embodiments of this application, we will first illustrate a system architecture that can be applied to this method to facilitate understanding.

[0066] The first system architecture is a storage system. As shown in Figure 1, the storage system includes a data processing unit and a storage unit. The data processing unit processes data using the method provided in the embodiments of this application to achieve data reduction, decrease the data volume, and obtain processed data. The storage unit stores the processed data. The processed data occupies less storage space in the storage unit, saving storage space and allowing the storage unit to store more other data, thereby improving the utilization rate of storage space and reducing storage costs.

[0067] For example, the data processing unit is implemented by at least one method in software or hardware, and the storage unit can be a persistent memory or a non-persistent memory. The embodiments of this application do not limit the form of the data processing unit and the storage unit.

[0068] Optionally, the storage system described above can be located in a single computer device. Alternatively, the storage system can be located in multiple computer devices, forming a distributed storage system. For example, the storage system includes multiple data processing units and multiple storage units, and the multiple computer devices each include at least one of the multiple data processing units and at least one of the multiple storage units. In one example, at least one of the multiple computer devices is a primary storage device, and the other computer devices are backup storage devices.

[0069] The second system architecture is a communication system. As shown in Figure 2, the communication system includes a data processing unit and a transmission unit. The data processing unit processes data using the method provided in the embodiments of this application to achieve data reduction, decrease the data volume, and obtain processed data. The transmission unit then sends the processed data. This processed data occupies less bandwidth in the transmission unit, which not only saves bandwidth but also reduces the time required to send the processed data, resulting in higher transmission efficiency.

[0070] For example, the data processing unit is implemented by at least one method in software or hardware, and the transmission unit can be a communication interface. The embodiments of this application do not limit the form of the data processing unit and the transmission unit.

[0071] The third system architecture is a combined storage and communication system. For example, this combined system includes a data processing unit, a storage unit, and a transmission unit. The data processing unit processes the data to obtain the processed data, the storage unit stores the processed data, and the transmission unit sends the processed data. Explanations of each unit can be found above and will not be repeated here.

[0072] The three system architectures described above are merely examples. The method provided in this application embodiment can also be applied to other feasible system architectures, and is not limited thereto. Next, the method will be described using the application of the method provided in this application embodiment to the data processing unit shown in Figure 1 or Figure 2 as an example. As shown in Figure 3, the method includes the following steps 301 to 304.

[0073] Step 301: Obtain the feature values ​​of the first data. The feature values ​​represent the compression rate or deduplication rate of the first data.

[0074] In this context, the first data refers to the data to be processed, or in other words, the data to be reduced. Embodiments of this application can obtain the first data through at least one of receiving, generating, or decompressing. Decompression refers to performing decompression on already compressed data. After obtaining the first data, its feature values ​​can be acquired. The feature values ​​of the first data reflect the characteristics of the first data itself, and these characteristics affect the compression ratio or deduplication rate of the first data. Therefore, the feature values ​​of the first data can characterize the compression ratio or deduplication rate of the first data. The compression ratio characterized by the feature values ​​of the first data is equivalent to the predicted compression ratio for the first data, and the deduplication rate characterized by the feature values ​​of the first data is equivalent to the predicted deduplication rate for the first data.

[0075] In an exemplary embodiment, the feature values ​​of the first data include the following two types.

[0076] The first eigenvalue represents the compression rate of the first data.

[0077] The first characteristic value can be any index used to characterize the compression ratio, and this application embodiment does not limit this. In one example, the first characteristic value is information entropy. Information entropy is defined as the average number of bits required to represent the bytes of each value in the first data. In other words, it is the average number of bits required to represent the bytes of each value in the first data. The smaller the information entropy, the fewer bits are required to represent the bytes of each value in the first data on average, and therefore the higher the compression ratio of the first data. The larger the information entropy, the more bits are required to represent the bytes of each value in the first data on average, and therefore the lower the compression ratio of the first data. Since 1 byte includes 8 bits, 1 byte can be represented by a maximum of 8 bits. Correspondingly, the bytes of each value in the first data need to be represented by a maximum of 8 bits on average. In this case, the upper limit of information entropy is 8 bits, and the lower limit of compression ratio is 0%. Therefore, information entropy is negatively correlated with the compression ratio represented by information entropy. Optionally, the information entropy of the first data can be calculated according to the following formula (1). E=-∑p(x)·log2p(x) Formula (1)

[0078] In formula (1), x represents the bytes with certain values ​​included in the first data, p(x) represents the probability of x appearing in the first data, and log2(·) represents the logarithmic operation with base 2. For example, if the first data of length 4B is AACC, the p(x) corresponding to the byte with value A is 0.5 and log2p(x) is -1, and the p(x) corresponding to the byte with value C is 0.5 and log2p(x) is -1, then the information entropy of the first data AACC is E=-(0.5·(-1)+0.5·(-1))=1, that is, each byte with value needs to be represented by an average of 1 bit. For example, if A is represented as 0 and C is represented as 1, the first data AACC needs to be represented by a total of 4 bits after compression.

[0079] The second feature value represents the deduplication rate of the first data.

[0080] The second feature value can be any metric used to characterize the deduplication rate, and this application does not limit this. In one example, the second feature value is the deduplication rate prediction value output by the deduplication rate prediction model for the first data. The deduplication rate prediction model is obtained through training. During training, sample data is acquired, deduplication is performed on the sample data to obtain the deduplication rate, and the sample data labeled with the deduplication rate is used as training samples to train the deduplication rate prediction model, enabling the deduplication rate prediction model to have the function of predicting the deduplication rate. During inference, the first data is input into the deduplication rate prediction model, and the deduplication rate prediction model performs prediction on the first data to obtain the deduplication rate prediction value and outputs the deduplication rate prediction value. The second feature value can be this deduplication rate prediction value. Optionally, the deduplication rate prediction value is positively correlated with the deduplication rate represented by the deduplication rate prediction value.

[0081] In an exemplary embodiment, the method for obtaining the feature values ​​of the first data includes, but is not limited to, the following two methods.

[0082] The first acquisition method involves acquiring the feature values ​​of the first data, including: acquiring the feature values ​​of the second data, where the feature values ​​represent the compression rate or deduplication rate of the second data, and the second data is a portion of the first data.

[0083] Since the second data is a portion of the first data, it can be obtained from the first data after the first data has been acquired. Optionally, in this embodiment, any portion of the first data can be used as the second data, or in other words, the second data can be located at any position within the first data. For example, a portion of the first data starting from the beginning of the first data can be used as the second data. Taking a first data length of 10KB as an example, the second data could be the first to fourth KB of the first data. The process of acquiring the second data can be performed once or multiple times (resulting in multiple sets of second data); this embodiment does not limit the number of executions.

[0084] After obtaining the second data, its feature values ​​can be obtained. Since the second data is a subset of the first data, it is smaller in volume than the first data, resulting in less computation and higher efficiency in obtaining its feature values. When the feature values ​​of the second data represent its compression rate, they can be, for example, the information entropy of the second data. The method for obtaining the information entropy of the second data can be found in the explanation of the first type of feature value above. When the feature values ​​of the second data represent its deduplication rate, they can be, for example, the predicted deduplication rate output by the deduplication rate prediction model for the second data. The method for obtaining this predicted deduplication rate can be found in the explanation of the second type of feature value above.

[0085] In one example, embodiments of this application may use the feature values ​​of the second data as the feature values ​​of the first data; in other words, the feature values ​​of the first data are the feature values ​​of the second data. In this example, the first data has only one feature value.

[0086] In another example, the first data includes multiple second data. In this embodiment, the feature values ​​of the second data can be obtained multiple times to obtain the feature values ​​of each of the multiple second data. The feature value of the first data is then determined based on the feature values ​​of each of the multiple second data. For example, if the first data includes second data A and second data B, then feature value 1 of second data A and feature value 2 of second data B can be obtained respectively, and the feature value of the first data can be determined based on feature value 1 of second data A and feature value 2 of second data B.

[0087] Different parts of the first data (i.e., different second data) may have different characteristics. The characteristic values ​​of multiple second data reflect the characteristics of each second data. Therefore, determining the characteristic value of the first data based on the characteristic values ​​of multiple second data can make the characteristic value of the first data more comprehensive and accurate.

[0088] For example, determining the feature value of the first data based on the feature values ​​of each of the multiple second data includes: using the feature values ​​of each of the multiple second data as the feature value of the first data; in other words, the feature value of the first data includes the feature values ​​of the multiple second data, and the number of feature values ​​of the first data is multiple. Alternatively, determining the feature value of the first data based on the feature values ​​of each of the multiple second data includes: performing a weighted calculation on the feature values ​​of each of the multiple second data to obtain the result of the weighted calculation, and using the result of the weighted calculation as the feature value of the first data; in other words, the feature value of the first data is the result of the weighted calculation, and the number of feature values ​​of the first data is one.

[0089] This application does not limit the length of each second data item; the length of the second data item can be determined based on experience or actual needs. For example, the length of the second data item may be 4KB. Multiple second data items can have the same length or different lengths. For example, the length of second data item A is 4KB, and the length of second data item B is 6KB. Different second data items among multiple second data items may or may not have overlapping portions. For example, the length of the first data item is 10KB. If different second data items have overlapping portions, and second data item A is the first to sixth KB of the 10KB, and second data item B is the fifth to tenth KB of the 10KB, then the overlapping portions are the fifth and sixth KB of the 10KB. If different second data items do not have overlapping portions, and second data item A is the first to fourth KB of the 10KB, and second data item B is the fifth to eighth KB of the 10KB. Furthermore, different second data items among multiple second data items may or may not have intervals. For example, if the length of the first data is 10KB, the second data A is the first to fourth KB of the 10KB, the second data B is the seventh to tenth KB of the 10KB, then the interval part is the fifth and sixth KB of the 10KB.

[0090] The second method involves obtaining the feature values ​​of the first data, including: obtaining all the feature values ​​of the first data, where the feature values ​​represent the compression rate or deduplication rate of the first data.

[0091] This method involves obtaining all feature values ​​of the first data, fully utilizing the features of different parts of the first data to make the feature values ​​of the first data more comprehensive and accurate. The number of feature values ​​for the first data is one. The method for obtaining the feature values ​​of the first data can be found in the explanations of the first and second feature values ​​above, and will not be repeated here.

[0092] As described above, in this embodiment of the application, after obtaining the feature values ​​of the data to be processed (i.e., the first data), the data is sliced ​​based on the feature values ​​to obtain the sliced ​​data. The slicing process is detailed in steps 302 and 303 below.

[0093] Step 302: Determine the range of slice length based on the feature values.

[0094] The range of slice lengths includes at least one slice length. This range can be a set (including at least one slice length), an interval (including multiple slice lengths), or a combination of a set and an interval. As explained in step 301 regarding the feature values ​​of the first data, the number of feature values ​​for the first data can be one or more. If the first data has only one feature value, the range of slice lengths can be determined once based on that feature value. If the first data has multiple feature values ​​(i.e., the feature values ​​of the first data include multiple feature values ​​of the second data), then the range of slice lengths is determined once for each feature value, resulting in multiple ranges of slice lengths. The process of determining multiple ranges of slice lengths is equivalent to repeatedly determining the range of slice lengths based on a single feature value. Therefore, the process of determining the range of slice lengths based on a single feature value will be used as an example for explanation to facilitate understanding.

[0095] The range of slice lengths determined by the eigenvalues ​​is used to slice the first data, resulting in sliced ​​data comprising multiple data blocks. A larger average slice length range allows for larger data block lengths, while a smaller average slice length range allows for smaller data block lengths. Since the slice length range is determined by the eigenvalues, the eigenvalues ​​influence the slice length range; in other words, there is a correlation between the eigenvalues ​​and the slice length range. In an exemplary embodiment, this correlation includes the following two cases.

[0096] The first case: The compression ratio represented by the eigenvalues ​​is positively correlated with the average value of the range of slice lengths.

[0097] In the first example, if the compression ratio represented by the eigenvalues ​​is low, then the room for improvement in the compression ratio of the first data is considered small. For example, a low compression ratio represented by the eigenvalues ​​means that the compression ratio represented by the eigenvalues ​​is within the range of the first compression ratio. As explained above, the larger the data block length, the higher the compression ratio. Based on this, the small room for improvement in the compression ratio of the first data means that even if the average value of the slice length range is increased, resulting in a larger data block length, it is difficult to significantly improve the compression ratio of the first data. Therefore, it is better to decrease the average value of the slice length range, resulting in a smaller data block length. Although this may slightly reduce the compression ratio of the first data, it can significantly improve the deduplication rate of the first data. Since both deduplication and compression can achieve data reduction, both the deduplication rate and the compression ratio of the first data affect the reduction rate of the first data. Therefore, a significant increase in the deduplication rate and a slight decrease in the compression ratio of the first data can still ensure a high reduction rate for the first data.

[0098] In this embodiment, the smaller the value of the first compression ratio range, the smaller the compression ratio represented by the feature value. According to the analysis in the previous paragraph, this makes the average value of the slice length range smaller, thereby increasing the deduplication rate of the first data more significantly and ensuring a higher reduction rate of the first data. Therefore, the compression ratio represented by the feature value is positively correlated with the average value of the slice length range.

[0099] Referring to Figure 4, an example is given using information entropy as the characteristic value representing the compression rate. Since information entropy is negatively correlated with the compression rate represented by information entropy, the larger the information entropy, the smaller the compression rate represented by information entropy, the smaller the average range of slice lengths, and the smaller the length of data blocks. This results in a greater increase in the deduplication rate of the first data (i.e., ensuring the deduplication rate), thereby ensuring a higher reduction rate for the first data.

[0100] In the second example, a higher compression ratio represented by eigenvalues ​​indicates greater potential for improvement in the compression ratio of the first data. For instance, a higher compression ratio means that the eigenvalue compression ratio falls within the second compression ratio range, which is larger than the first compression ratio range mentioned above. Greater potential for improvement in the compression ratio of the first data means that increasing the average value of the slice length range, thus increasing the data block length, can significantly improve the compression ratio of the first data. Therefore, increasing the average value of the slice length range, while potentially slightly reducing the deduplication rate of the first data, can significantly improve its compression ratio, thereby ensuring a high reduction rate for the first data.

[0101] In this embodiment, the larger the value of the second compression ratio range, the greater the compression ratio represented by the feature value. According to the analysis in the previous paragraph, this allows for a larger average value of the slice length range, resulting in a greater increase in the compression ratio of the first data and thus ensuring a higher reduction rate for the first data. Therefore, the compression ratio represented by the feature value is still positively correlated with the average value of the slice length range.

[0102] Referring to Figure 5, an example is given using information entropy as the characteristic value representing the compression ratio. Since information entropy is negatively correlated with the compression ratio represented by information entropy, the smaller the information entropy, the larger the compression ratio represented by information entropy, the larger the average range of slice lengths, and the larger the length of data blocks. This results in a greater increase in the compression ratio of the first data (i.e., ensuring the compression ratio), and thus ensuring a higher reduction rate for the first data.

[0103] In the second case, the deduplication rate represented by the eigenvalues ​​is negatively correlated with the average value of the range of slice lengths.

[0104] In the third example, if the deduplication rate represented by the feature values ​​is low, then the room for improvement in the deduplication rate of the first data is considered small. For example, a low deduplication rate represented by the feature values ​​means that the deduplication rate represented by the feature values ​​is within the first deduplication rate range. As explained above, the larger the data block length, the lower the deduplication rate. Correspondingly, the smaller the data block length, the higher the deduplication rate. Based on this, the room for improvement in the deduplication rate of the first data is small, meaning that even if the average value of the slice length range is reduced, resulting in a smaller data block length, it is difficult to significantly improve the deduplication rate of the first data. Therefore, it is better to increase the average value of the slice length range, resulting in a larger data block length. Although this may slightly reduce the deduplication rate of the first data, it can significantly improve the compression ratio of the first data, thus ensuring a high reduction rate for the first data.

[0105] In this embodiment, the smaller the value of the first deduplication rate range, the smaller the deduplication rate represented by the feature value. According to the analysis in the previous paragraph, this allows for a larger average value of the slice length range, resulting in a greater improvement in the compression rate of the first data and thus ensuring a higher reduction rate of the first data. Therefore, the deduplication rate represented by the feature value is negatively correlated with the average value of the slice length range.

[0106] Let's take the predicted deduplication rate as an example to illustrate the concept. Since the predicted deduplication rate is positively correlated with the deduplication rate it represents, the smaller the predicted deduplication rate, the larger the average range of slice lengths, and the larger the data block length. This results in a greater improvement in the compression rate of the first data (i.e., ensuring the compression rate), and thus ensuring a higher reduction rate for the first data.

[0107] In the fourth example, if the deduplication rate represented by the feature values ​​is relatively high, then the deduplication rate of the first data is considered to have significant room for improvement. For example, a relatively high deduplication rate represented by the feature values ​​means that the deduplication rate represented by the feature values ​​falls within the second deduplication rate range, which is larger than the first deduplication rate range mentioned above. Significant room for improvement in the deduplication rate of the first data means that reducing the average range of slice lengths, thus reducing the length of the data blocks, can substantially improve the deduplication rate of the first data. Therefore, reducing the average range of slice lengths, thus reducing the length of the data blocks, although it may slightly decrease the compression rate of the first data, can significantly improve the deduplication rate of the first data, thereby ensuring a high reduction rate for the first data.

[0108] In this embodiment, the larger the value of the second deduplication rate range, the larger the deduplication rate represented by the feature value. According to the analysis in the previous paragraph, this can make the average value of the slice length range smaller, thereby increasing the deduplication rate of the first data more significantly and ensuring a higher reduction rate for the first data. Therefore, the deduplication rate represented by the feature value is still negatively correlated with the average value of the slice length range.

[0109] Let's take the predicted deduplication rate as an example to illustrate the concept. Since the predicted deduplication rate is positively correlated with the deduplication rate it represents, the larger the predicted deduplication rate, the larger the deduplication rate it represents. This results in a smaller average range of slice lengths and smaller data block lengths, leading to a greater increase in the deduplication rate of the first data (i.e., ensuring the deduplication rate is maintained), and thus ensuring a higher reduction rate for the first data.

[0110] The first through fourth examples described above all guarantee a high reduction rate for the first data. Specifically, the first example (where the feature value represents the compression rate) and the fourth example (where the feature value represents the deduplication rate) both guarantee the reduction rate of the first data by ensuring the deduplication rate, while the second example (where the feature value represents the compression rate) and the third example (where the feature value represents the deduplication rate) both guarantee the reduction rate of the first data by ensuring the compression rate.

[0111] In an exemplary embodiment, determining the range of slice lengths based on feature values ​​includes: determining the average value of the range of slice lengths based on the feature values, and determining the range of slice lengths based on the average value. This method is simple and easy to implement, and is beneficial for accurately and quickly determining the range of slice lengths.

[0112] In response to the first case mentioned above, namely the case where the compression ratio represented by the feature value is positively correlated with the average value of the range of slice lengths, the embodiments of this application can determine the average value of the range of slice lengths based on the feature value according to the following formula (2), and determine the range of slice lengths based on the average value of the range of slice lengths according to the following formula (3). Lmin =C, mask = average - C, L max =L min +2·mask=average+mask formula (3)

[0113] In formulas (2) and (3), average represents the average value of the range of slice lengths, H represents the characteristic value representing the compression ratio, and f(H) represents a function related to H. A, B, C, and D are all constants, and their values ​​can be determined based on experience or actual needs. Represents rounding up. L min L represents the minimum (or lower limit) of the range of slice lengths, and mask represents the difference between the average and minimum values ​​of the range of slice lengths. max The maximum value (or upper limit) of the range of slice lengths.

[0114] For example, H is the information entropy E in formula (1) above. A is the minimum average length of the data block, and the value of A is, for example, 8KB. B is the influence coefficient of information entropy on the average length of the data block, and the value of B is, for example, 3KB. C is the minimum length of the data block, and the value of C is, for example, 4KB. D is, for example, the upper limit of information entropy mentioned above, and the value of D is 8 bits.

[0115] Regarding the second situation mentioned above, namely the case where the deduplication rate represented by the eigenvalue is negatively correlated with the average value of the slice length range, this embodiment of the application can determine the average value of the slice length range based on the eigenvalue according to the following formula (4), and determine the slice length range based on the average value of the slice length range according to the following formula (5). average=f(I)=F+G·I Formula (4) L min =J, mask = average - J, L max =L min +2·mask=average+mask formula (5)

[0116] In formulas (4) and (5), average represents the average value of the range of slice lengths, I represents the characteristic value representing the deduplication rate, and f(I) represents a function related to I. F, G, and J are all constants, and their values ​​can be determined based on experience or actual needs. min , mask and L max The content represented is the same as that in formula (3). Please refer to the explanation in formula (3). It will not be repeated here.

[0117] For example, determining the average value of the range of slice lengths based on feature values ​​includes: determining the average value of the range of slice lengths according to a first determination method based on the feature values. Determining the range of slice lengths based on the average value includes: determining the range of slice lengths according to a second determination method based on the average value.

[0118] Optionally, the first determining method refers to performing a first processing based on at least one first processing parameter. That is, determining the average value of the range of slice lengths according to the feature values ​​and the first determining method includes: performing a first processing on the feature values ​​according to at least one first processing parameter to obtain the average value. The first processing includes, but is not limited to, at least one of the following processing methods: summation, subtraction, product, division, rounding up, or rounding down.

[0119] In one example, at least one first processing parameter includes a first parameter, a second parameter, and a third parameter. Performing a first processing on the feature value according to at least one first processing parameter to obtain an average value includes: performing a subtraction operation on the first parameter and the feature value to obtain a first difference; performing a floor function operation on the first difference to obtain a floor function result; performing a product operation on the floor function result and the second parameter to obtain a first product; and performing a sum operation on the first product and the third parameter to obtain the average value.

[0120] For example, if the first determination method is the above formula (2), then the first parameter corresponds to D in formula (2), the second parameter corresponds to B in formula (2), and the third parameter corresponds to A in formula (2).

[0121] In another example, at least one first processing parameter includes a fourth parameter and a fifth parameter. Performing a first processing on the eigenvalues ​​according to at least one first processing parameter to obtain an average value includes: performing a product processing on the fourth parameter and the eigenvalues ​​to obtain a second product, and performing a summation processing on the second product and the fifth parameter to obtain the average value.

[0122] For example, if the first determination method is the above formula (4), then the fourth parameter corresponds to G in formula (4), and the fifth parameter corresponds to F in formula (4).

[0123] Optionally, the second determination method refers to performing a second process based on at least one second processing parameter. That is, determining the range of slice lengths according to the average value and the second determination method includes: performing a second process on the average value based on at least one second processing parameter to obtain the range of slice lengths. The second process may be the same as or different from the first process described above. The second processing parameters may be the same as or different from the first processing parameters described above.

[0124] In one example, at least one second processing parameter is a sixth parameter. The second processing is performed on the average value based on at least one second processing parameter to obtain a range of slice lengths, including: using the sixth parameter as the lower limit of the slice length range; performing a subtraction operation on the sixth parameter and the average value to obtain a second difference; and performing a summation operation on the second difference and the average value to obtain an upper limit of the slice length range, thereby obtaining a range of slice lengths between the lower and upper limits.

[0125] For example, if the second determination method is formula (3) above, then the sixth parameter corresponds to C in formula (3), and the second difference corresponds to mask in formula (3). For another example, if the second determination method is formula (5) above, then the sixth parameter corresponds to J in formula (5), and the second difference corresponds to mask in formula (5).

[0126] In another example, at least one second processing parameter includes a seventh parameter and an eighth parameter, which may be the same or different. A second processing step is performed on the average value based on at least one second processing parameter to obtain a range of slice lengths, including: subtracting the average value from the seventh parameter to obtain a lower limit of the slice length range; and summing the average value from the eighth parameter to obtain an upper limit of the slice length range, thus obtaining a range of slice lengths between the lower and upper limits.

[0127] The above descriptions of the first and second determination methods are merely examples and are not intended to limit the scope of the methods. At least one of the first or second determination methods is obtained based on the reduction rate of the third data preceding the first data. That is, the embodiments of this application can obtain the third data and its reference feature values. Based on the reference feature values, a reference average value is determined according to the first reference method, and based on the reference average value, a range of reference slice lengths is determined according to the second reference method. After slicing the third data according to the range of reference slice lengths, at least one of deduplication or compression is performed on the sliced ​​third data to obtain reduced data. Next, a first reduction rate of the reduced data relative to the third data is determined, the first reference method is updated according to the first reduction rate to obtain the first determination method, and the second reference method is updated according to the reduction rate of the third data to obtain the second determination method. Since at least one of the first or second determination methods is obtained based on the reduction rate of the third data, this at least one determination method is relatively accurate, and a relatively accurate range of slice lengths can be determined according to this at least one determination method.

[0128] This application does not limit the method of updating based on the reduction rate of the third data. Taking the first determination method as the above formula (2) as an example, updating based on the reduction rate of the third data can mean updating the values ​​of A, B and D in formula (2) based on the reduction rate of the third data, or it can mean updating the values ​​of A, B and D in formula (2) based on the reduction rate of the third data. This is the approach to handling it.

[0129] Optionally, embodiments of this application may also update at least one of the first or second determination methods. For example, after determining the range of slice length according to the first and second determination methods, steps 303 and 304 below are executed to obtain processed data. A second reduction rate of the processed data relative to the first data is determined, and at least one of the first or second determination methods is updated according to the second reduction rate to obtain an updated determination method. The updated determination method is used to determine the range of slice length for the fourth data after the first data. In other words, the method of determining the range of slice length based on feature values ​​in embodiments of this application can be continuously updated, ensuring that the determined range of slice length is relatively accurate.

[0130] The above explains the process of determining the range of slice length based on a single feature value. When the first data has multiple feature values ​​(i.e., the first data's feature values ​​include multiple feature values ​​of the second data), multiple ranges of slice length can be determined, allowing for dynamic adjustment of the slice length range. For example, taking information entropy as the feature value, when the first data includes second data A, second data B, and second data C, as shown in Figure 6, the average value X (smaller) of the slice length range is determined based on the information entropy 1 (larger) of second data A, resulting in a smaller data block length M. The average value Y (larger) of the slice length range is determined based on the information entropy 2 (smaller) of second data B, resulting in a larger data block length N. The average value Z (medium) of the slice length range is determined based on the information entropy 3 (medium) of second data C, resulting in a medium data block length Q. Here, information entropy 1 > information entropy 3 > information entropy 2, the average value X of the slice length range < the average value Z of the slice length range < the average value Y of the slice length range, and the data block length M < the data block length Q < the data block length N.

[0131] Step 303: Slice the first data according to the range of slice length to obtain the sliced ​​data.

[0132] After determining the range of slice lengths, the first data can be sliced ​​according to the range of slice lengths to obtain sliced ​​data, which includes multiple data blocks. In an exemplary embodiment, the slicing methods include, but are not limited to, the following two.

[0133] The first slicing method corresponds to the case in step 302 where the number of feature values ​​of the first data is one, and the range of slice length is determined based on one feature value.

[0134] In one example, at least one slice length is determined from the range of slice lengths. The first data is then sliced ​​according to this at least one slice length to obtain the sliced ​​data. Since the range of slice lengths is determined based on the characteristic values ​​of the first data, which reflect the inherent characteristics of the first data, regardless of whether there is one or more slice lengths, the resulting data blocks will have appropriate lengths, which helps ensure a good reduction rate. For example, if the first data is 20KB long and the slice length is 5KB, then slicing the first data according to 5KB lengths will result in four data blocks of 5KB length. As another example, if the first data is 20KB long and the slice lengths are 4KB and 8KB, then slicing the first data according to 4KB lengths will result in one data block of 4KB length, and then slicing the remaining portion of the first data according to 8KB lengths will result in two data blocks of 8KB length.

[0135] In another example, the first data is sliced ​​according to a sliding window within the range of slice lengths to obtain sliced ​​data. The length of the sliding window is, for example, the lower limit of the range of slice lengths. The sliding step size of the sliding window can be determined based on experience or actual needs. For example, the sliding step size is 1B. This application embodiment does not limit the sliding step size.

[0136] During the first slicing process, the lower limit of the sliding window is located at the starting position of the first data. If the data within the sliding window meets the conditions, then according to the range of slice lengths, slicing is performed on the first data starting from the starting position of the first data to obtain the first data block. The length of the first data block is equal to the lower limit of the slice length range. If the data within the sliding window does not meet the conditions, a sliding step is performed once to change the data within the sliding window. If the changed data meets the conditions, the sliding ends, and the position from the starting position of the first data to the upper limit of the sliding window at the end of the sliding is taken as the first data block. If the data within the sliding window still does not meet the conditions, one or more sliding steps are performed until, after a certain sliding, the data within the sliding window meets the conditions, and the position from the starting position of the first data to the upper limit of the sliding window at the end of the sliding is taken as the first data block. If, after sliding, the length between the upper limit of the sliding window and the starting position of the first data has reached the upper limit of the slice length range, then according to the range of slice lengths, slicing is performed on the first data starting from the starting position of the first data to obtain the first data block. The length of the first data block is equal to the upper limit of the slice length range. After obtaining the first data block, the data in the first data block excluding the first data block is the first remaining data.

[0137] During the second slicing process, the lower limit of the sliding window is located at the starting position of the first remaining data. The sliding window begins to slide from this point. The sliding method can be found in the explanation of the first slicing process, which will not be repeated here. After the sliding ends, the position from the starting position of the first remaining data to the upper limit of the sliding window at the end of the sliding is taken as the second data block. After obtaining the second data block, the data in the first remaining data excluding the second data block is the second remaining data.

[0138] This process continues until a certain slicing process is completed. When the remaining data (i.e., the data that has not yet been sliced, or the data that has not entered the sliding window) is less than or equal to the length of the sliding window, the remaining data is taken as the last data block, thus completing the slicing of the first data.

[0139] Optionally, the data portion within the sliding window may satisfy certain conditions, including but not limited to: the last 1B of data within the sliding window includes 8 bits with a value of 0; or, a first value (such as a hash value) calculated based on the data portion within the sliding window is divisible by a second value (such as 5), and so on. These conditions can be determined based on experience or actual needs, and the embodiments of this application do not limit these conditions.

[0140] For example, the first data block is 20KB long, the slice length ranges from 6KB (lower limit) to 10KB (upper limit), and the sliding window length is 6KB. During the first slice, the sliding window stops when the upper limit reaches 8KB, resulting in a first data block of 8KB, containing data from the first data block (items 1 to 8KB). During the second slice, the sliding window stops when the upper limit reaches 18KB, resulting in a second data block of 10KB, containing data from the first data block (items 9 to 18KB). The remaining data is used as the last data block, resulting in a last data block of 2KB, containing data from the first data block (items 19 and 20KB).

[0141] The second slicing method corresponds to the case in step 302 where the number of feature values ​​of the first data is multiple (i.e., the feature values ​​of the first data include the feature values ​​of multiple second data), and the range of multiple slice lengths is determined.

[0142] In one example, a slice length is determined from each range of slice lengths, resulting in multiple slice lengths. The first data is then sliced ​​according to these multiple slice lengths to obtain the sliced ​​data. For instance, if the first data is 20KB long, a slice length range 1 is determined based on feature value 1 of the second data A, and slice length 1 is determined to be 8KB from this range. Similarly, a slice length range 2 is determined based on feature value 2 of the second data B, and slice length 2 is determined to be 12KB from this range. Therefore, the first data can be sliced ​​at 8KB intervals to obtain a data block of 8KB. Alternatively, the remaining portion of the first data can be sliced ​​at 12KB intervals to obtain a data block of 12KB.

[0143] In another example, slicing the first data according to a range of slice lengths to obtain sliced ​​data includes: slicing the first data according to the range of slice lengths and the position of the second data within the first data, resulting in sliced ​​data. For example, for the second data, starting from its initial position within the first data, slicing the first data according to the range of slice lengths corresponding to the second data, resulting in sliced ​​data where the starting position of the sliced ​​data is the same as the starting position of the second data within the first data. The range of slice lengths corresponding to the second data refers to the range of slice lengths determined based on the feature values ​​of the second data.

[0144] Since the range of slice lengths is determined based on the feature values ​​of the second data, performing slicing according to the range of slice lengths and the position of the second data within the first data ensures the accuracy of the slicing process, resulting in more suitable data block lengths after slicing. For example, slicing the first data according to a sliding window is performed within each slice length range to obtain the sliced ​​data. The method of performing slicing the first data according to a sliding window within the slice length range can be found in the description of the first slicing method, and will not be repeated here.

[0145] For example, the process of obtaining second data, obtaining feature values ​​of the second data, and determining the range of slice length (referred to as process 1), and the process of slicing the first data according to the range of slice length and the position of the second data in the first data (referred to as process 2), can be performed alternately. That is, in the embodiments of this application, process 1 can be executed first to obtain a first piece of second data from the first data, obtain the feature values ​​of the first piece of second data, and determine the range of slice length based on the feature values ​​of the first piece of second data. Then, process 2 can be executed as described above to perform a first slice on the first data according to the range of slice length and the position of the first piece of second data in the first data, obtaining a first data block and a first remaining data, where the first remaining data is the data in the first data excluding the first data block. Afterward, process 1 and process 2 are repeated to obtain a second piece of second data from the first remaining data and perform a second slice to obtain a second data block and a second remaining data, where the second remaining data is the data in the first remaining data excluding the second data block. This process is repeated until the processing of the first data is completed.

[0146] For example, the length of the first data is 20KB. The first second data is the first to fourth KB of the first data (starting at the 1KB position). Based on the first second data, the slice length ranges from 4KB to 8KB, resulting in the first data block including the first to sixth KB of the first data, and the first remaining data including the seventh to twenty-first KB of the first data. The second second data is the seventh to tenth KB of the first remaining data (starting at the 7KB position). Based on the second second data, the slice length ranges from 12KB to 16KB, resulting in the second data block including the seventh to twenty-first KB of the first data.

[0147] The two slicing methods described above are merely examples and are not intended to limit the slicing methods used in the embodiments of this application. For example, the slicing method can be content-defined chunking (CDC).

[0148] Step 304: Perform at least one of the following processing on the sliced ​​data: deduplication or compression, to obtain the processed data.

[0149] As mentioned earlier, the sliced ​​data comprises multiple data blocks. Therefore, at least one of the following processes—deletion or compression—is performed on these multiple data blocks to obtain the processed data. That is, for multiple data blocks, only deleting, only compression, or both deleting and compression can be performed. When performing both deleting and compression, deleting can be performed first, followed by compression. Taking multiple data blocks including data block 1, data block 2, and data block 3 as an example, data block 2 is identical to data block 1, while data block 3 is different from data block 1; therefore, data block 2 and data block 1 are duplicate data blocks. During the deleting process, data block 1 is retained, data block 2 is deleted, a pointer is added to the location of data block 2 to reference data block 1, and data block 3 is retained. Therefore, the data after deleting includes data block 1, the pointer, and data block 3. During the compression process, a compression algorithm is used to compress the data blocks after deleting, resulting in the processed data.

[0150] For example, the methods for processing the sliced ​​data in this application embodiment may include the following six.

[0151] The first processing method uses feature values ​​to represent the compression ratio. It performs deduplication and compression on the sliced ​​data to obtain the processed data. This method is applicable to the first case in step 302 (including the first and second examples). The first case balances deduplication and compression ratios, ensuring that the processed data has a higher reduction rate compared to the first data.

[0152] The second processing method uses feature values ​​to represent the deduplication rate. Deduplication and compression are performed on the sliced ​​data to obtain the processed data. This method is applicable to the second case in step 302 (including the third and fourth examples). The second case balances deduplication and compression rates, ensuring that the processed data has a higher reduction rate compared to the first data.

[0153] The third processing method uses feature values ​​to represent the compression ratio. Only deduplication is performed on the sliced ​​data to obtain the processed data. This method is suitable for the first example in step 302, which ensures that the processed data has a high deduplication rate.

[0154] The fourth processing method uses feature values ​​to characterize the compression ratio. Only compression is performed on the sliced ​​data to obtain the processed data. This is applicable to the second example in step 302, which ensures that the processed data has a high compression ratio.

[0155] The fifth processing method uses feature values ​​to represent the deduplication rate. It performs only compression on the sliced ​​data to obtain the processed data. This method is applicable to the third example in step 302, which ensures a high compression rate for the processed data.

[0156] The sixth processing method uses feature values ​​to represent the deduplication rate. Only deduplication is performed on the sliced ​​data to obtain the processed data. This method is applicable to the fourth example in step 302, which ensures that the processed data has a high deduplication rate.

[0157] In exemplary embodiments, depending on the system architecture, the embodiments of this application may also perform other processing on the processed data, as detailed in the following description.

[0158] For the first system architecture mentioned above, namely the storage system, the method also includes storing the processed data. The processed data requires less storage space, which helps save storage space.

[0159] In one example, the first data is the uncompressed raw data. For example, the first data is the received raw data, or the first data is the generated raw data. Referring to Figure 7, (11) after the data processing unit receives or generates the raw data (i.e., the first data), (12) it performs processing according to the description in steps 301 to 304 to obtain the processed data, and (13) it sends the processed data to the storage unit so that the storage unit stores the processed data.

[0160] In another example, the first data is obtained by decompressing the compressed data. Referring to Figure 7, (21) the data processing unit obtains the compressed data from the storage unit, (22) decompresses the compressed data to obtain the decompressed data (i.e., the first data), (23) performs processing according to the description in steps 301 to 304 to obtain the processed data, and (24) sends the processed data (not shown in Figure 7) to the storage unit so that the storage unit stores the processed data.

[0161] Decompressing compressed data results in data bloat, which occurs when the number of bytes in the decompressed data exceeds the number of bytes in the compressed data. Different bloat factors result in different characteristic values ​​in the decompressed data. Since this embodiment can accurately slice the data based on these characteristic values, obtaining sliced ​​data (i.e., multiple data blocks), the length of these data blocks is related to the bloat factor. Consequently, the number of data blocks is also related to the bloat factor, ensuring a more balanced number of data blocks under different bloat factors. This avoids situations where there are too many data blocks with a large bloat factor or too few data blocks with a small bloat factor, thus preventing negative impacts on the overall performance of the storage system.

[0162] In one example, a larger data inflation factor indicates a higher compression ratio of the data's eigenvalue representation. As explained above, the compression ratio of eigenvalue representation can be positively correlated with the average value of the slice length range. Therefore, a larger average slice length range results in larger data block lengths and fewer data blocks, avoiding the situation where there are too many data blocks when the data inflation factor is large. Conversely, a smaller data inflation factor indicates a lower compression ratio of the data's eigenvalue representation. Consequently, a smaller average slice length range results in smaller data block lengths and more data blocks, avoiding the situation where there are too few data blocks when the data inflation factor is small.

[0163] In another example, a larger data inflation factor indicates a higher data compression ratio. As explained above, compression ratio and deduplication rate are relative; a higher compression ratio results in a lower deduplication rate, meaning a lower deduplication rate represented by the data's eigenvalues. Furthermore, as mentioned above, the deduplication rate represented by eigenvalues ​​is negatively correlated with the average slice length range. Therefore, a larger average slice length range results in larger data block lengths and fewer data blocks, avoiding an excessive number of data blocks when the data inflation factor is large. Conversely, a smaller data inflation factor indicates a higher deduplication rate represented by the data's eigenvalues, resulting in a smaller average slice length range, smaller data block lengths, and a larger number of data blocks, avoiding an excessively small number of data blocks when the data inflation factor is small.

[0164] For the second system architecture described above, namely the communication system, the method further includes: sending the processed data. The processed data requires only a small amount of bandwidth, and the time required to send the processed data is also short. Referring to the second system architecture shown in Figure 2, after the data processing unit receives or generates the first data, it performs processing according to the descriptions in steps 301 to 304 to obtain the processed data, and then sends the processed data to the transmission unit, causing the transmission unit to send the processed data.

[0165] For the third system architecture mentioned above, namely the combined storage and communication system, the method further includes: storing the processed data and sending the processed data. Explanations of the storage and sending processes can be found above and will not be repeated here.

[0166] In summary, the embodiments of this application obtain feature values ​​of the first data, which characterize the compression rate or deduplication rate of the first data and reflect the inherent characteristics of the first data. Then, the range of slice lengths is determined based on the feature values, and the first data is sliced ​​according to this range to obtain sliced ​​data. Because the feature values ​​reflect the inherent characteristics of the first data and have a strong correlation with the data itself, the range of slice lengths determined based on the feature values ​​is relatively accurate. Slicing according to this range results in sliced ​​data with a more suitable length. Therefore, performing at least one of deduplication or compression on the sliced ​​data yields processed data with a high data reduction rate.

[0167] The above describes the data processing method provided by the embodiments of this application. Corresponding to the above method, the embodiments of this application also provide a data processing apparatus. This apparatus is used to execute the data processing method shown in FIG3 through the various modules shown in FIG8. As shown in FIG8, the data processing apparatus provided by the embodiments of this application includes the following modules.

[0168] The acquisition module 801 is used to acquire the feature values ​​of the first data, where the feature values ​​represent the compression rate or deduplication rate of the first data.

[0169] Module 802 is used to determine the range of slice length based on feature values;

[0170] The slicing module 803 is used to slice the first data according to the range of slice length to obtain the sliced ​​data;

[0171] The processing module 804 is used to perform at least one of the following processing methods on the sliced ​​data: deduplication or compression, to obtain the processed data.

[0172] In an exemplary embodiment, the acquisition module 801 is used to acquire feature values ​​of the second data, the feature values ​​representing the compression ratio or deduplication ratio of the second data, the second data being a portion of the first data; the slicing module 803 is used to slice the first data according to the range of the slice length and the position of the second data in the first data, to obtain sliced ​​data.

[0173] In an exemplary embodiment, an average value is used to determine the range of slice lengths based on feature values; the range of slice lengths is determined based on the average value.

[0174] For example, the determining module 802 is configured to determine the average value of the range of slice lengths according to a first determining method based on feature values; and to determine the range of slice lengths according to a second determining method based on the average value. Wherein, at least one of the first or second determining methods is obtained based on the reduction rate of the third data preceding the first data.

[0175] For example, the compression ratio represented by the eigenvalues ​​is positively correlated with the average value of the range of slice lengths.

[0176] In an exemplary embodiment, the feature value is information entropy, which characterizes the compression ratio.

[0177] For example, the deduplication rate represented by the eigenvalues ​​is negatively correlated with the average value of the range of slice lengths.

[0178] In an exemplary embodiment, the apparatus further includes a storage module for storing processed data.

[0179] For example, the first data is obtained by decompressing the compressed data.

[0180] In an exemplary embodiment, the apparatus further includes a transmitting module for transmitting processed data.

[0181] It should be understood that the beneficial effects of the device shown in Figure 8 are the same as those of the method shown in Figure 3 when implementing its functions. The device shown in Figure 8 is only illustrated by the division of the functional modules described above. In practical applications, the functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. Furthermore, the device and method embodiments provided in the above embodiments belong to the same concept, and their specific implementation processes are detailed in the method embodiments, which will not be repeated here.

[0182] For example, embodiments of this application also provide a data processing device, which includes a memory and a processor; the memory stores at least one computer instruction, which is loaded and executed by the processor to enable the device to implement the data processing method shown in FIG3.

[0183] Referring to Figure 9, Figure 9 shows a schematic diagram of the structure of an exemplary data processing device 900 of this application. The data processing device 900 includes at least one processor 901, a memory 903, and at least one network interface 904.

[0184] Processor 901 is, for example, a general-purpose central processing unit (CPU), a digital signal processor (DSP), a network processor (NP), a graphics processing unit (GPU), a neural-network processing unit (NPU), a data processing unit (DPU), a microprocessor, or one or more integrated circuits or application-specific integrated circuits (ASICs), programmable logic devices (PLDs), other general-purpose processors or other programmable logic devices, discrete gates, transistor logic devices, discrete hardware components, or any combination thereof used to implement the scheme of this application. PLD is, for example, a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof. The general-purpose processor can be a microprocessor or any conventional processor. It is worth noting that the processor can be a processor supporting an advanced reduced instruction set machine (RISC) machine (ARM) architecture. It can implement or execute various logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0185] Optionally, the data processing device 900 also includes a bus 902. The bus 902 is used to transfer information between the components of the data processing device 900. The bus 902 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus 902 can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one line is used in Figure 9, but this does not indicate that there is only one bus or one type of bus.

[0186] The memory 903 may be, for example, volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory may be random access memory (RAM), which is used as an external cache.

[0187] By way of example, but not limitation, many forms of ROM and RAM are available. For example, ROM is a compact disc read-only memory (CD-ROM). RAM includes, but is not limited to, static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0188] The memory 903 can also be other types of storage devices capable of storing static information and instructions. Alternatively, it can be other types of dynamic storage devices capable of storing information and instructions. It can also be other optical disc storage, optical disk storage (including compressed optical discs, laser discs, optical discs, digital versatile optical discs, Blu-ray discs, etc.), magnetic disk storage media, or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer, but is not limited thereto. The memory 903 may exist independently, for example, and be connected to the processor 901 via bus 902. The memory 903 may also be integrated with the processor 901.

[0189] Network interface 904 uses any transceiver-like device for communicating with other devices or communication networks, such as Ethernet, radio access network (RAN), or wireless local area network (WLAN). Network interface 904 can include wired network interfaces and wireless network interfaces. Specifically, network interface 904 can be an Ethernet interface, such as Fast Ethernet (FE), Gigabit Ethernet (GE), Asynchronous Transfer Mode (ATM), WLAN, cellular network, or combinations thereof. The Ethernet interface can be an optical interface, an electrical interface, or a combination thereof. In some embodiments of this application, network interface 904 can be used by data processing device 900 to communicate with other devices.

[0190] In specific implementations, as some embodiments, processor 901 may include one or more CPUs, such as CPU0 and CPU1 shown in FIG9. Each of these processors may be a single-core processor or a multi-core processor. Here, processor may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0191] In specific implementations, as some embodiments, the data processing device 900 may include multiple processors, such as processor 901 and processor 905 shown in FIG9. Each of these processors may be a single-core processor or a multi-core processor. Here, "processor" may refer to one or more devices, circuits, and / or processing cores for processing data (such as computer program instructions).

[0192] In some embodiments, memory 903 is used to store program instructions 910 for executing the present application scheme, and processor 901 can execute the program instructions 910 stored in memory 903. That is, data processing device 900 can implement the method provided in the method embodiment, i.e., the method shown in FIG3, through processor 901 and program instructions 910 in memory 903. Program instructions 910 may include one or more software modules. Optionally, processor 901 itself may also store program instructions for executing the present application scheme.

[0193] In specific implementation, the data processing device 900 of this application can correspond to the data processing unit for performing the above method. The processor 901 in the data processing device 900 reads the instructions in the memory 903, so that the data processing device 900 shown in FIG9 can perform all or part of the steps in the method embodiment.

[0194] The data processing device 900 can also correspond to the device shown in FIG8 above, where each functional module is implemented in software by the data processing device 900. In other words, the functional modules included in the device shown in FIG8 are generated by the processor 901 of the data processing device 900 reading the program instructions 910 stored in the memory 903.

[0195] The steps of the method shown in Figure 3 are completed by the integrated logic circuitry of the hardware or by software instructions in the processor of the data processing device 900. The steps of the method embodiments disclosed in this application can be directly implemented by the hardware processor, or by a combination of hardware and software modules in the processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. Since the storage medium is located in memory, the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method embodiments; to avoid repetition, they will not be described in detail here.

[0196] For example, this application embodiment also provides a storage system, which includes a data processing unit and a storage unit. The data processing unit is used to execute the data processing method shown in FIG3 to obtain processed data. The storage unit is used to store the processed data.

[0197] By way of example, embodiments of this application also provide a computer-readable storage medium storing at least one computer instruction, which is loaded and executed by a processor to enable a computer including a processor to implement the data processing method shown in FIG3.

[0198] By way of example, embodiments of this application also provide a computer program product, which includes computer instructions that are executed by a processor to enable a computer including a processor to implement the data processing method shown in FIG3.

[0199] In some embodiments, this application also provides a chip, including a processor, for calling and executing instructions stored in a memory, causing a computer with the chip installed to perform the data processing method shown in FIG3.

[0200] In other embodiments, this application also provides another chip, including: an input interface, an output interface, a processor, and a memory. The input interface, output interface, processor, and memory are connected through an internal connection path. The processor is used to execute code in the memory. When the code is executed, the computer with the chip installed performs the data processing method shown in FIG3.

[0201] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions according to this application are generated, in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk).

[0202] In this application, the terms "first," "second," etc., are used to distinguish identical or similar items that have essentially the same function. It should be understood that there is no logical or temporal dependency between "first," "second," and "nth," nor does it limit the quantity or order of execution. It should also be understood that although the following description uses the terms "first," "second," etc., to describe various elements, these elements should not be limited by the terms. These terms are merely used to distinguish one element from another.

[0203] It should also be understood that, in the various embodiments of this application, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0204] In this application, the term "at least one" means one or more, and the term "multiple" means two or more; for example, multiple data blocks refer to two or more data blocks. The terms "system" and "network" are often used interchangeably in this document.

[0205] It should be understood that the terminology used in the description of the various examples herein is for the purpose of describing the particular examples only and is not intended to be limiting. As used in the description of the various examples and in the appended claims, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0206] It should also be understood that the term "and / or" as used herein refers to and covers any and all possible combinations of one or more of the associated listed items. The term "and / or" describes an association between related objects, indicating that three relationships can exist; for example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the character " / " in this application generally indicates that the preceding and following related objects are in an "or" relationship.

[0207] It should also be understood that the terms “if” and “if” can be interpreted as meaning “when” or “upon”, or “in response to determination” or “in response to detection”. Similarly, depending on the context, the phrases “if determination…” or “if detection [the stated condition or event]” can be interpreted as meaning “when determination…”, or “in response to determination…”, or “when detection [the stated condition or event]” or “in response to detection [the stated condition or event]”.

[0208] The above are merely embodiments of this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the principles of this application should be included within the protection scope of this application.

Claims

1. A method for processing data, characterized in that, The method includes: Obtain the feature value of the first data, wherein the feature value characterizes the compression ratio or deduplication ratio of the first data; The range of slice length is determined based on the aforementioned feature values; The first data is sliced ​​according to the range of the slice length to obtain the sliced ​​data; Perform at least one of the following processes, namely deduplication or compression, on the sliced ​​data to obtain the processed data.

2. The method according to claim 1, characterized in that, The acquisition of the feature values ​​of the first data includes: Obtain feature values ​​of the second data, wherein the feature values ​​characterize the compression ratio or deduplication ratio of the second data, and the second data is a portion of the first data; The step of slicing the first data according to the range of the slice length to obtain the sliced ​​data includes: Based on the range of the slice length and the position of the second data in the first data, the first data is sliced ​​to obtain the sliced ​​data.

3. The method according to claim 1 or 2, characterized in that, Determining the range of slice length based on the feature value includes: The average value of the range of slice lengths is determined based on the characteristic values; The range of slice lengths is determined based on the average value.

4. The method according to claim 3, characterized in that, The step of determining the average value of the range of slice lengths based on the feature values ​​includes: Based on the characteristic value, the average value of the range of slice lengths is determined according to the first determination method; Determining the range of slice lengths based on the average value includes: Based on the average value, the range of the slice length is determined according to the second determination method; Wherein, at least one of the first determination method or the second determination method is obtained based on the reduction rate of the third data preceding the first data.

5. The method according to any one of claims 1-4, characterized in that, The compression ratio, characterized by the eigenvalue, is positively correlated with the average value of the range of slice lengths.

6. The method according to any one of claims 1-5, characterized in that, The feature value is information entropy, which characterizes the compression ratio.

7. The method according to any one of claims 1-4, characterized in that, The deduplication rate, characterized by the eigenvalue, is negatively correlated with the average value of the range of slice lengths.

8. A data processing apparatus, characterized in that, The device includes: The acquisition module is used to acquire feature values ​​of the first data, wherein the feature values ​​characterize the compression ratio or deduplication ratio of the first data; The determination module is used to determine the range of slice length based on the feature values; The slicing module is used to slice the first data according to the range of the slice length to obtain the sliced ​​data; The processing module is used to perform at least one of the following processes on the sliced ​​data: deduplication or compression, to obtain processed data.

9. The apparatus according to claim 8, characterized in that, The acquisition module is used to acquire feature values ​​of the second data, the feature values ​​representing the compression rate or deduplication rate of the second data, the second data being a portion of the first data; The slicing module is used to slice the first data according to the range of the slice length and the position of the second data in the first data to obtain sliced ​​data.

10. The apparatus according to claim 8 or 9, characterized in that, The determining module is used to determine the average value of the range of slice lengths based on the feature values; and to determine the range of slice lengths based on the average value.

11. The apparatus according to claim 10, characterized in that, The determining module is configured to determine the average value of the range of slice lengths according to the feature value and a first determining method; and to determine the range of slice lengths according to the average value and a second determining method. Wherein, at least one of the first determination method or the second determination method is obtained based on the reduction rate of the third data preceding the first data.

12. The apparatus according to any one of claims 8-11, characterized in that, The compression ratio, characterized by the eigenvalue, is positively correlated with the average value of the range of slice lengths.

13. The apparatus according to any one of claims 8-12, characterized in that, The feature value is information entropy, which characterizes the compression ratio.

14. The apparatus according to any one of claims 8-11, characterized in that, The deduplication rate, characterized by the eigenvalue, is negatively correlated with the average value of the range of slice lengths.

15. A storage system, characterized in that, The storage system includes a data processing unit and a storage unit; The data processing unit is used to execute the method described in any one of claims 1-7 to obtain processed data; The storage unit is used to store the processed data.

16. A data processing device, characterized in that, The device includes a memory and a processor; the memory stores at least one computer instruction, which is loaded and executed by the processor to enable the device to implement the data processing method according to any one of claims 1-7.

17. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one computer instruction, which is loaded and executed by a processor to enable the computer to implement the method of processing data as described in any one of claims 1-7.

18. A computer program product, characterized in that, The computer program product includes computer instructions that are executed by a processor to cause the computer to perform the data processing method according to any one of claims 1-7.

Images