Data processing method, device and system and related equipment

Abstract

The invention discloses a data processing method, device and system and related equipment, and relates to the technical field of data processing. Original data to be processed are obtained, first data and second data in the original data are recognized, the first data are uncompressed data, and the second data are compressed data; decompressing the second data to obtain first intermediate data, and combining the first data and the first intermediate data to obtain second intermediate data; and finally, compressing the second intermediate data to obtain a compression result. By identifying and decompressing the second data in the original data, the overall compression effect for the original data can be improved in the process of merging the first data and the first intermediate data and compressing the second intermediate data, so that the storage space occupied by the compression result of the original data can be reduced as much as possible, and the compression efficiency of the original data is improved. Storage resources needing to be consumed when original data are stored are reduced, and the overall performance of data storage is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular, to a data processing method, device, system and related equipment. BACKGROUND

[0002] In the era of big data, data is usually compressed and the compressed data is persistently stored to reduce the storage resources required for data storage. In general, the same data compressed by different compression algorithms will have different compression results, and the compression effect obtained will also differ. For example, for the same data, the data amount of the compression result obtained based on compression algorithm A can be much smaller than the data amount of the compression result obtained based on compression algorithm B. Accordingly, when data is stored, the different compression results of the same data need to be saved respectively.

[0003] In actual application scenarios, the data to be stored can be a mixture of uncompressed data and compressed data, where uncompressed data refers to data that has not been compressed using a compression algorithm, and compressed data refers to data obtained by compressing using a compression algorithm. For example, in the scenario of periodic backup of data, the data to be stored can include not only incremental data generated on the same day of application, but also data generated before the same day of application and obtained by compression.

[0004] When storing such mixed data including uncompressed data and compressed data, high storage resource consumption is usually generated, which affects the overall performance of data storage. For example, when the mixed data includes data A (i.e., uncompressed data) and the compression result of data A (i.e., compressed data), both data A and the compression result of data A will be stored, which makes data A occupy at least two copies of storage resources. SUMMARY

[0005] The present application provides a data processing method to reduce the storage resources consumed when storing data and improve the overall performance of data storage. In addition, the present application also provides a corresponding data processing device, storage system, computing device, computer-readable storage medium and computer program product.

[0006] In a first aspect, the present application provides a data processing method, which can be executed by a corresponding data processing device. Specifically, the data processing device acquires original data to be processed, which can be a data stream or a plurality of files, etc. Then, the data processing device identifies first data and second data in the original data, wherein the first data is uncompressed data, and the second data is compressed data, such as the second data in the original data obtained by compressing part of the data by a client before providing the data to the data compression device. Next, the data compression device decompresses the second data to obtain first intermediate data, and combines the first data and the first intermediate data to obtain second intermediate data, so that the data compression device compresses the second intermediate data to obtain a compression result.

[0007] Since the data processing device identifies the compressed second data in the original data, and combines and compresses the first intermediate data obtained by decompressing the second data and the first data, the data amount of the compression result finally generated by the data processing device can be effectively smaller than the data amount of the compression result obtained by directly compressing the original data by the data processing device. For example, when the first intermediate data obtained by decompressing and the first data include the same data, the data processing device can avoid generating two different compression results for the two pieces of data by effectively compressing the two pieces of data, thereby improving the overall compression effect for the two pieces of data. Or, when the second data includes a plurality of different compression results for the same data, the data processing device can effectively improve the overall compression effect for the plurality of pieces of the same data by decompressing the plurality of compression results and uniformly compressing the plurality of pieces of the same data. In this way, in the case that the original data can be effectively compressed, the storage space required by the compression result can be reduced as much as possible, thereby reducing the storage resources consumed when storing the original data and improving the overall performance of data storage.

[0008] In a possible implementation, the original data is a data stream, and when identifying the first data and the second data in the original image, the data processing apparatus can specifically divide the original data to obtain a plurality of data segments, and calculate a data confusion degree corresponding to each data segment in the plurality of data segments, so as to determine the first data and the second data according to the data confusion degree corresponding to each data segment, and the data confusion degree corresponding to a data segment included in the second data is higher than the data confusion degree corresponding to a data segment included in the first data. Generally, when the data confusion degree corresponding to a data segment is low, the data segment is usually not compressed, and when the data confusion degree corresponding to a data segment is high, the data segment is usually data obtained by compression. Therefore, for the data in the form of a stream, the data processing apparatus can identify whether the data is obtained by compression by calculating the data confusion degree corresponding to each data segment, and can identify the second data obtained by compression in the original data, so as to improve the overall compression effect of the original data based on the identified second data.

[0009] In a possible implementation, the original data is a data stream, and when identifying the first data and the second data in the original image, the data processing apparatus can specifically divide the original data to obtain a plurality of data segments, and calculate a data confusion degree corresponding to each data segment in the plurality of data segments, so as to determine the first data and the second data according to the data confusion degree corresponding to each data segment, and the data confusion degree corresponding to a data segment included in the second data is higher than the data confusion degree corresponding to a data segment included in the first data. Generally, when the data confusion degree corresponding to a data segment is low, the data segment is usually not compressed, and when the data confusion degree corresponding to a data segment is high, the data segment is usually data obtained by compression. Therefore, for the data in the form of a stream, the data processing apparatus can identify whether the data is obtained by compression by calculating the data confusion degree corresponding to each data segment, and can identify the second data obtained by compression in the original data, so as to improve the overall compression effect of the original data based on the identified second data.

[0010] In a possible implementation, the first intermediate data includes the first decompressed data and the difference data, and when decompressing the second data, the data processing apparatus can specifically decompress the second data by using a decompression algorithm corresponding to the compression algorithm indicated by the compression-related information, obtain the first decompressed data, compress the first decompressed data by using the compression algorithm indicated by the compression-related information, obtain the re-compressed data, and generate the difference data according to the re-compressed data and the second data, where the difference data is used to indicate the difference between the re-compressed data and the second data. In this way, the final data obtained by decompressing the data segment by the data processing apparatus includes the first decompressed data and the difference data, so that when the data segment is recovered, the correct data segment can be recovered according to the first decompressed data and the difference data, thereby reducing the influence of the data precision caused by compression of the original data.

[0011] In a possible implementation, the compression-related information includes an identifier of a compression subject corresponding to the second data, and the data processing apparatus can find a mapping relationship according to the compression-related information to obtain the compression algorithm corresponding to the second data, the mapping relationship being used to indicate a compression algorithm that can be used by each of the at least one compression subject. Since each compression subject usually uses a fixed type of compression algorithm in an actual application scenario, the data processing apparatus can determine the compression algorithm corresponding to the second data compressed by the compression subject according to the identifier of the compression subject, so that the data processing apparatus can subsequently decompress the second data based on a decompression algorithm matched with the compression algorithm.

[0012] In a possible implementation, when the data processing apparatus merges the first data and the first intermediate data to obtain the second intermediate data, the data processing apparatus can specifically perform deduplication on the first data and the first intermediate data to obtain the second intermediate data. In this way, the data processing apparatus can implement efficient compression of the original data by deduplicating the same data in the first data and the first intermediate data.

[0013] In a possible implementation, when the data processing apparatus performs deduplication on the first data and the first intermediate data to obtain the second intermediate data, the data processing apparatus can specifically first determine a first compression algorithm for the first data and the first intermediate data, and then compress the first data by using the first compression algorithm to obtain a first preliminary compression result, and compress the first intermediate data by using the first compression algorithm to obtain a second preliminary compression result; and then the data processing apparatus can perform deduplication on the first preliminary compression result and the second preliminary compression result to obtain the second intermediate data. Since the data processing apparatus has compressed the first data and the first intermediate data before deduplication, the amount of data participating in deduplication can be effectively reduced, thereby reducing the consumption of storage resources and computing resources.

[0014] In a possible implementation, when the data processing apparatus compresses the second intermediate data, the data processing apparatus can specifically compress the second intermediate data by using a second compression algorithm, the complexity of the second compression algorithm being higher than that of the first compression algorithm. In this way, the data processing apparatus can compress the second intermediate data by using a compression algorithm with higher complexity, thereby improving the overall compression effect on the original data.

[0015] In a possible implementation, when the data processing apparatus compresses the second intermediate data to obtain the compression result, the data processing apparatus can first decompress the second intermediate data based on a decompression algorithm corresponding to the first compression algorithm to obtain second decompressed data, and then compress the second decompressed data by using the second compression algorithm to obtain the compression result. In this way, the data processing apparatus can fully exploit the correlation between different data by re-decompressing and then overall compressing the data, thereby helping to improve the overall compression effect of the original data.

[0016] In a second aspect, the present application provides a data processing apparatus, comprising an obtaining module configured to obtain original data to be processed; an identifying module configured to identify first data and second data in the original data, the first data being uncompressed data, and the second data being compressed data; a decompressing module configured to decompress the second data to obtain first intermediate data; a merging module configured to merge the first data and the first intermediate data to obtain second intermediate data; and a compressing module configured to compress the second intermediate data to obtain a compression result.

[0017] In a possible implementation, the original data is a data stream; and the identifying module is configured to: divide the original data to obtain a plurality of data segments; calculate a data disorder degree corresponding to each data segment in the plurality of data segments; and determine the first data and the second data according to the data disorder degree corresponding to each data segment, wherein the second data includes data segments with a data disorder degree higher than that of the first data.

[0018] In a possible implementation, the original data is a data stream; and the identifying module is configured to: divide the original data to obtain a plurality of data segments; and determine the first data and the second data from the plurality of data segments, wherein each data segment included in the second data is configured with compression-related information, and the compression-related information is used to indicate a compression algorithm used for data in the data segment.

[0019] In a possible implementation, the first intermediate data includes first decompressed data and difference data; and the decompressing module is configured to: decompress the second data by using a decompression algorithm corresponding to a compression algorithm indicated by the compression-related information to obtain the first decompressed data; compress the first decompressed data by using the compression algorithm indicated by the compression-related information to obtain re-compressed data; and generate the difference data according to the re-compressed data and the second data, wherein the difference data is used to indicate a difference between the re-compressed data and the second data.

[0020] In a possible implementation, the compression-related information includes an identifier of a compression subject corresponding to the second data, and the apparatus further includes a searching module configured to search a mapping relationship according to the compression-related information, to obtain a compression algorithm corresponding to the second data, the mapping relationship being used to indicate a compression algorithm that can be used by each of the at least one compression subject.

[0021] In a possible implementation, the merging module is configured to perform deduplication on the first data and the first intermediate data, to obtain the second intermediate data.

[0022] In a possible implementation, the merging module is configured to: determine a first compression algorithm for the first data and the first intermediate data; compress the first data by using the first compression algorithm to obtain a first preliminary compression result, and compress the first intermediate data by using the first compression algorithm to obtain a second preliminary compression result; and perform deduplication on the first preliminary compression result and the second preliminary compression result to obtain the second intermediate data.

[0023] In a possible implementation, the compression module is configured to compress the second intermediate data by using a second compression algorithm, the complexity of the second compression algorithm being higher than that of the first compression algorithm.

[0024] In a possible implementation, the compression module is configured to: decompress the second intermediate data based on a decompression algorithm corresponding to the first compression algorithm, to obtain second decompression data; and compress the second decompression data by using the second compression algorithm, to obtain the compression result.

[0025] The data processing apparatus provided in the second aspect corresponds to the data processing method provided in the first aspect, and therefore the technical effects of any one of the implementation manners of the data processing apparatus provided in the second aspect can be referred to the description of the corresponding implementation manners of the first aspect.

[0026] In a third aspect, the present application provides a computing device, which includes a processor and a memory. The processor and the memory are in communication with each other. The processor is configured to execute instructions stored in the memory, so that the computing device performs the data processing method in the first aspect or any one of the implementation manners of the first aspect. It should be noted that the memory can be integrated into the processor, or can be independent of the processor. The computing device can further include a bus. The processor is connected to the memory through the bus. The memory can include a readable memory and a random access memory.

[0027] In a fourth aspect, the present application provides a storage system, characterized in that the storage system comprises at least one storage medium and a processor, the processor is configured to execute the data processing method in the first aspect or any implementation manner of the first aspect, and the at least one storage medium is configured to persistently store the data compression result.

[0028] In a fifth aspect, the present application provides a computer readable storage medium, which stores instructions, and when the instructions are executed on a computing device, the computing device is caused to perform the operation steps of the data processing method in the first aspect or any implementation manner of the first aspect.

[0029] In a sixth aspect, the present application provides a computer program product comprising instructions, and when the instructions are executed on a computing device, the computing device is caused to perform the operation steps of the data processing method in the first aspect or any implementation manner of the first aspect.

[0030] On the basis of the implementation manners of the above aspects, the present application can be further combined to provide more implementation manners. BRIEF DESCRIPTION OF DRAWINGS

[0031] Figure 1 An exemplary structural schematic diagram of a storage system;

[0032] Figure 2 A flowchart of a data processing method provided by the present application;

[0033] Figure 3 An exemplary diagram of data segmentation including part of data in a compression frame provided by the present application;

[0034] Figure 4 An exemplary diagram of lightweight compression and re-deletion on multiple blocks provided by the present application;

[0035] Figure 5 A structural schematic diagram of a data processing apparatus provided by the present application;

[0036] Figure 6 A hardware structural schematic diagram of a computing device provided by the present application. DETAILED DESCRIPTION

[0037] In order to make the above purposes, features and advantages of the present application more obvious and easy to understand, various non-limiting implementation manners in the embodiments of the present application will be exemplarily described below in combination with the drawings. Obviously, the described embodiments are some embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained based on the above content all belong to the protection scope of the present application.

[0038] Reference is made to Figure 1Fig. 1 is a structural schematic diagram of an exemplary storage system. As shown in Fig. 1, the storage system 10 includes a processor 100 and at least one storage medium. Figure 1 Figure 1 For example, the storage system 10 includes N storage media, i.e., storage medium 1 to storage medium N.

[0039] The processor 100 can be a general-purpose processor, such as a central processing unit (CPU). Alternatively, the processor 100 can be an accelerator card, such as a graphics processing unit (GPU), a neural-network processing unit (NPU), or a micro processing unit (MPU). Alternatively, the processor 100 can be another hardware with data processing capability, such as an application-specific integrated circuit (ASIC), a programmable logic device (PLD), a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), a system on chip (SoC), a software-defined infrastructure (SDI) chip, an artificial intelligence (AI) chip, a data processing unit (DPU), or any combination thereof. The number of processors included in the storage system 10 can be one or more, and the types of processors included in the storage system 10 can be one or more. The number and types of processors can be determined according to the actual business requirements, and the present embodiment does not limit the number and types of processors.

[0040] Each storage medium can be a solid state drive (SSD), a hard disk drive (HDD), a disk, or the like, and can be used to store data, such as compressed business data. The storage system 10 can include N storage media of the same type or N storage media of different types. ​

[0041] As shown in Figure 1 The storage system 10 can establish a communication connection with one or more applications, and each application can send data to the storage system 10 for persistent storage in the storage system 10. For ease of description and differentiation, the data is referred to as original data below. Exemplarily, Figure 1 The application shown in

[0042] The processor 100 can receive the original data sent by the application through a communication interface or a network card, etc. The original data received by the application can be mixed with uncompressed data and compressed data. For example, in a data backup scenario, the application can backup the business data generated every N days to the storage system 10, where N is a positive integer. The application can use compression software to compress the business data generated from the first day to the (N-1)th day and save the compression result in a storage device other than the storage system 10. On the Nth day, the application can send the business data generated on the Nth day and the compressed data of the previous (N-1) days stored in the storage system 10 for data backup in the storage system 10.

[0043] In this process, if the storage system 10 directly saves the original data or saves the original data after compression, it is easy to cause the storage system 10 to occupy a large storage space to save the original data, that is, to consume a high storage resource. For example, when the original data includes uncompressed data A and compressed data B, the compressed data B is obtained by using compression software by the application to compress the data A, that is, the original data includes the data A and the compression result of the data A. At this time, the storage system 10 only needs to compress and save the data A to achieve the data backup effect, but the storage system 10 usually compresses and saves the data A and the data B as a whole, which makes the storage system 10 finally consume more storage resources to save the compression result of the data A and the data B than to save the compression result of the data A.

[0044] Based on this, in the storage system 10 shown in Figure 1 The storage system 10 can improve the storage resources required for data storage by decompressing and then compressing the original data.

[0045] Specifically, for the original data sent by the application, the processor 100 identifies the uncompressed data and the compressed data in the original data, decompresses the compressed data to obtain intermediate data, i.e., restores the compressed data to the state before compression. Then, the processor 100 merges the intermediate data and the uncompressed data, and the merged data is in an uncompressed state. Thus, the processor 100 compresses the merged data using a corresponding compression algorithm to obtain the final compression result.

[0046] Since the processor 100 identifies the compressed data in the original data, decompresses the compressed data, and merges the intermediate data obtained by decompressing the compressed data with the uncompressed data and compresses the merged data, the data amount of the compression result finally generated by the processor 100 is usually effectively smaller than the data amount of the compression result obtained by directly compressing the original data by the processor 100. Thus, in the case that the original data can be effectively compressed, the storage space required by the compression result can be reduced as much as possible, thereby reducing the storage resources consumed when storing the original data and improving the overall performance of data storage.

[0047] Still taking the example that the original data includes the data A and the data B (i.e., the compression result of the data A), the processor 100 can decompress the data B to the data A after identifying the data B in the original data, and merge the decompressed data A with the data A in the original data, such as re-deleting the two data A, so that only one data A is retained after merging. Finally, the processor 100 can compress the retained data A, and the compression result of the data A is taken as the compression result of the original data. Thus, the storage resources consumed for saving the compression result of the data A are obviously smaller than the storage resources consumed for saving the overall compression result of the data A and the data B. In addition, even if the two data A are not re-deleted, the data amount required to be occupied can be reduced by using a unified compression algorithm to compress the same two data A.

[0048] For example, the compressed data in the original data may include different compression results of the same data, such as compression result 1 and compression result 2 of data X. These compression results 1 and 2 can be obtained by applications using different compression software (based on different compression algorithms). After identifying compression results 1 and 2 in the original data, the processor 100 can decompress compression result 1 to recover data X, and decompress compression result 2 to recover data X. Thus, during the process of merging the uncompressed data of the original data and the two decompressed data X, the processor 100 can perform deduplication on these two data X. In this way, after the processor 100 compresses the merged data and obtains the compressed result, the data size of the compressed result will be significantly smaller than the data size of the compressed result obtained by directly compressing the original data.

[0049] In practical applications, the N storage media in storage system 10 can be deployed centrally or in a distributed manner.

[0050] In a centralized deployment, storage system 10 can be deployed on a single storage server, which is configured with N storage media, such as multiple SSDs (each SSD being N). Figure 1 (as shown in the diagram, a storage medium). Thus, after generating a compression result based on the original data, the processor 100 can write the compression result to one or more storage media.

[0051] When using a distributed deployment, the N storage media in storage system 10 can be distributed across multiple different storage servers. Each storage server can be configured with one or more storage media; for example, multiple SSDs can be inserted into each storage server (each SSD is...). Figure 1 (A storage medium is shown). Furthermore, the processor 100 in storage system 10 can be a processor in one of the storage servers, or it can be a separate device. Thus, after generating a compressed result based on the original data, the processor 100 can send the compressed result to the storage medium in one or more storage servers. In a distributed storage system, different storage servers (and processor 100) can communicate via remote direct memory access (RDMA) protocol, compute expresslink (CXL) protocol, or other protocols.

[0052] It is worth noting that the above Figure 1 The storage system 10 shown is merely an illustrative example and is not intended to be limiting. For example, Figure 1The storage system 10 shown is configured to perform the compression process on the raw data by the processor 100. In other possible storage systems, a controller and N storage media can be included, and the controller can be configured to perform the data compression process performed by the processor 100. Alternatively, in other possible storage systems, a CPU and an acceleration card can be included, and the acceleration card can be configured to perform the data compression process performed by the processor 100. Alternatively, in other possible storage systems, more other types of hardware can be included to support the storage system to have more other functions based on the storage system 10 shown. Figure 1 The storage system 10 shown can also include more other types of hardware to support the storage system to have more other functions.

[0053] For ease of understanding, the embodiments of the data processing method provided in the present application are described below with reference to the accompanying drawings.

[0054] Referring to Figure 2 , Figure 2 A flowchart of a data processing method provided in an embodiment of the present application is shown. The method can be applied to Figure 1 the storage system 10 or other applicable storage systems. For ease of illustration, the processor 100 in the storage system 10 shown is taken as an example for illustrative description in the embodiment. Figure 1

[0055] Among them, Figure 2 The data processing method shown can specifically include the following steps.

[0056] S201: An application sends raw data to be processed to the storage system 10, the raw data including uncompressed data and compressed data.

[0057] The raw data can be structured data or unstructured data. The structured data refers to data stored in a set format or rule, such as data stored in a table. The unstructured data refers to data with incomplete or irregular data structure, such as image data, audio / video data, etc.

[0058] In one possible implementation, the application can generate raw data during running and send the raw data to the storage system 10. For example, when the application is a client on the user side, the client can generate a data storage request according to a storage operation instruction of the user on the raw data, and send the data storage request to the storage system 10. Correspondingly, the storage system 10 can receive the data storage request through a communication interface, and pull the raw data to be processed from the application according to the data storage request. In actual application, the storage system 10 can also obtain the raw data in other manners, which is not limited in this regard.

[0059] ​S202: The processor 100 identifies first data and second data in the original data, wherein the first data is uncompressed data, and the second data is compressed data.

[0060] In this embodiment, the original data is a mixture of compressed data and uncompressed data. For ease of description, the uncompressed data in the original data is referred to as first data, and the data obtained by compression of the original data is referred to as second data.

[0061] In this embodiment, the following exemplary implementation modes of identifying the second data are provided.

[0062] In a first possible implementation, the original data received by the processor 100 can be multiple files, and each file can carry metadata that can be used to describe the compression algorithm used by the file in the data compression process. For example, the metadata can be a file format, so that the processor 100 can determine the compression algorithm used to compress the file data according to the file format. For example, when the file format is “.zip”, the processor 100 can determine that the compression algorithm corresponding to the file with “.zip” as the suffix is the DEFLATE compression algorithm (a lossless data compression algorithm).

[0063] In this way, the processor 100 can identify the uncompressed files (first data) and compressed files (second data) in the original data according to the metadata of each file.

[0064] In a second possible implementation, the application can continuously send data to the processor 100, and at this time, the original data received by the processor 100 can be a data stream. Then, the processor 100 can perform subsequent processing operations on the received data as the original data when the amount of data of the received data stream reaches a threshold.

[0065] Specifically, the processor 100 can divide the original data to obtain multiple data segments. The processor 100 can divide the original data according to a preconfigured fixed length to obtain multiple data segments of the same size. Alternatively, the processor 100 can divide the original data according to a dynamically variable length to obtain multiple data segments of different sizes.

[0066] Then, the processor 100 identifies whether each data segment is obtained by data compression.

[0067] In example one, the processor 100 can calculate a data disorder degree corresponding to each data segment, which is used to measure the disorder between the data in the data segment. In actual application, when the data disorder degree corresponding to the data segment is low, it indicates that the data in the data segment have strong similarity or regularity, and the data in this case are usually not compressed. On the contrary, when the data disorder degree corresponding to the data segment is high, it indicates that the data in the data segment have strong disorder, and the data in this case are usually compressed.

[0068] Then, the processor 100 can identify at least one data segment with high data disorder degree as the second data obtained by data compression, and identify at least one data segment with low data disorder degree as the first data not compressed.

[0069] For example, the data disorder degree can be information entropy. Then, the processor 100 can calculate the information entropy corresponding to each data segment based on the following formula (1), wherein the greater the information entropy, the higher the disorder degree between the data in the data segment; the smaller the information entropy, the lower the disorder degree between the data in the data segment.

[0070] H[x] = -∑p(x)log2p(x) (Formula 1)

[0071] Wherein, H[x] is the information entropy corresponding to the data segment; x is a single data in the data segment; p(x) is the frequency of data x in the data segment.

[0072] Then, for each data segment, the processor 100 can compare the information entropy corresponding to the data segment with a threshold. And the processor 100 can identify the data segment with information entropy greater than the threshold as the data segment in the second data, and identify the data segment with information entropy less than or equal to the threshold as the data segment in the first data. The threshold can be a fixed value set by the technician in advance, or a value dynamically configured by the technician for different applications or different business scenarios.

[0073] In example two, the processor 100 can traverse whether each data segment is configured with compression related information, which can be used to indicate that the data in the data segment is obtained by compression and can be used to indicate the compression algorithm adopted by the data. Then, for the data segment configured with the compression related information, the processor 100 can identify the data segment as the data segment in the second data; for the data segment not configured with the compression related information, the processor 100 can identify the data segment as the data segment in the first data.

[0074] The compression-related information may be, for example, an identifier of a compression algorithm. For example, the identifier of the compression algorithm may be added to the data in a process of compressing the data by using the compression algorithm to obtain the data in the data segment, so that the processor 100 can identify that the data in the data segment is obtained by compression and the compression algorithm used according to the identifier of the compression algorithm.

[0075] Alternatively, the compression-related information may be, for example, an identifier of a compression subject corresponding to the data segment, where the compression subject refers to an execution subject of a data compression process to obtain the data segment, such as a compression software. In actual application scenarios, each compression subject may use a specific type of compression algorithm to perform the data compression process. For example, when the compression subject is specifically Veeam software, the compression subject will usually use a compression algorithm such as run-length encoding (RLE), Lempel-Ziv 4 (LZ4), ZSTD (a lossless data compression algorithm), and the like to perform data compression. Therefore, the processor 100 can not only determine that the data segment is obtained by data compression according to the identifier of the compression subject, but also determine the compression algorithm used to obtain the data segment according to the identifier of the compression subject.

[0076] For example, the processor 100 may be preconfigured with a mapping relationship indicating the compression algorithm that can be used by each of the at least one compression subject (such as indicating the compression algorithm used by each of a plurality of compression software). In this way, the processor 100 can query the mapping relationship according to the identifier of the compression subject configured for the data segment to obtain the compression algorithm corresponding to the data segment. The mapping relationship may be, for example, a mapping between the identifier of the compression subject and the identifier of the compression algorithm, and the mapping relationship may be stored in an information library created for the processor 100.

[0077] In actual application, when a single compression subject can use multiple compression algorithms, the processor 100 can try to decompress the data in the data segment one by one by using the decompression algorithm corresponding to each of the multiple compression algorithms, and determine the compression algorithm corresponding to the data segment according to the decompression algorithm used for successful decompression.

[0078] It should be noted that the above implementation manner of identifying whether each data segment is obtained by data compression is only used for illustrative description, and in actual application, the processor 100 can also identify by other manners. For example, the processor 100 can first identify whether the data segment is obtained by data compression according to the data disorder degree corresponding to the data segment, and further identify the compression algorithm used by the data segment according to the compression-related information configured for the data segment.

[0079] Furthermore, when the data segments in the second data are configured with compression-related information, the processor 100 can also save the compression-related information and the data in the data segments separately so that the data in the data segments and the compression-related information can be processed separately in the future.

[0080] S203: The processor 100 decompresses the identified second data to obtain the first intermediate data.

[0081] In this embodiment, the processor 100 can decompress the second data obtained by compression in the original data to restore the second data to its state before compression.

[0082] In a first possible implementation, when the original data includes multiple files, the processor 100 can determine the corresponding decompression algorithm based on the compression algorithm indicated by the file's metadata. The decompression operation corresponding to the decompression algorithm is the inverse process of the compression operation corresponding to the compression algorithm. Thus, the processor 100 can recover the file data before compression by decompressing the second data.

[0083] In a second possible implementation, when the original data is a data stream, the processor 100 can determine the corresponding decompression algorithm based on the compression algorithm indicated by the compression-related information corresponding to each data segment, and use the decompression algorithm corresponding to each data segment to decompress the data segment to obtain the data before compression corresponding to the data segment.

[0084] In practical applications, since the data segments in the second data are obtained through partitioning, each data segment may include a portion of compressed data based on a single compressed frame. A compressed frame refers to a frame of data obtained after compressing the data using a compression algorithm. Therefore, the data obtained by decompressing the portion of compressed data included in the compressed frame within that data segment by the processor 100 may differ from the actual data.

[0085] For example, such as Figure 3 As shown, assuming the original data includes compressed frame x for data X and compressed frame y for data Y, then, when the processor 100 divides the original data, it may classify a portion of the compressed data in compressed frame x and compressed frame y into the same data segment (another portion of the compressed data in compressed frame x is classified into other data segments). This makes it difficult for the processor 100 to recover the correct decompressed data corresponding to the portion of compressed data in compressed frame x when decompressing the data segment. In other words, it leads to a difference between the decompressed data and the real data (usually, the correct data of data X can only be recovered from the complete compressed frame x).

[0086] To this end, for each data segment in the second data, the processor 100 can first determine a corresponding decompression algorithm according to the compression algorithm indicated by the compression-related information configured for the data segment, and decompress the data segment using the decompression algorithm to obtain first decompressed data. Then, the processor 100 can re-compress the first decompressed data using the compression algorithm indicated by the compression-related information to obtain re-compressed data, and generate difference data corresponding to the data segment according to the re-compressed data and the data segment, the difference data being used to indicate the difference between the re-compressed data and the data segment. In this way, the final data obtained by the processor 100 by decompressing the data segment includes the first decompressed data and the difference data. In this way, when the data segment is recovered later, the correct data segment can be recovered according to the first decompressed data and the difference data.

[0087] In actual application, when the number of decompression algorithms configured by the processor 100 is limited, the processor 100 can determine the decompression algorithm with the highest correlation between the compression algorithm corresponding to the data segment from the multiple configured decompression algorithms, and decompress the data segment using the decompression algorithm.

[0088] Similarly, the processor 100 can perform the above operation on each data segment in the second data to obtain first decompressed data and difference data corresponding to each data segment, that is, to obtain first intermediate data corresponding to the second data.

[0089] S204: The processor 100 merges the first data and the first intermediate data to obtain second intermediate data.

[0090] In the process of merging the first data and the first intermediate data, if there is data of the same part, the processor 100 can perform data deduplication and the like in the merging process, so as to effectively improve the deduplication rate of the original data, reduce the amount of data to be compressed later, that is, reduce the amount of data of the final generated compression result. In this embodiment, the following several implementation examples of merging the first data and the first intermediate data are provided.

[0091] In a first possible implementation, the processor 100 can first splice the first data and the first intermediate data, and divide the spliced data into multiple blocks according to a fixed length or an unfixed length, such as dividing the data into multiple 64-kilobyte data blocks. Then, the processor 100 can identify the blocks with the same data in the multiple blocks. For example, the processor 100 can traverse each block and calculate the fingerprint of each block, such as calculating the fingerprint of each block based on a secure hash algorithm 1 (SHA-1). Then, the processor 100 can compare the fingerprints of the multiple blocks, and when the fingerprints of two blocks are the same, the processor 100 can determine that the data in the two blocks is the same. In this way, in the process of merging the first data and the first intermediate data, the processor 100 can first perform deduplication on the identified same data, that is, for multiple repeated data, the processor 100 can only keep one copy of the data. Then, the processor 100 can splice the deduplicated first data and the first intermediate data to generate the second intermediate data.

[0092] In a second possible implementation, because the number of the first intermediate data obtained after the processor 100 decompresses the second data is significantly greater than the number of the second data, data inflation is likely to occur, and large resource consumption is likely to occur. Therefore, the processor 100 can first perform lightweight compression on the first data and the first intermediate data using a unified compression algorithm, such as an LZ compression algorithm, to reduce the number of the first data and the first intermediate data.

[0093] In a specific implementation, the processor 100 can first determine a first compression algorithm for the first data and the first intermediate data, which can be an LZ compression algorithm, for example. The specific implementation process of the processor 100 to determine the first compression algorithm can be referred to the description of determining the second compression algorithm below, and will not be described here. Then, the processor 100 can compress the first data using the first compression algorithm to obtain a first preliminary compression result, and compress the first intermediate data using the first compression algorithm to obtain a second preliminary compression result. Next, the processor 100 can perform deduplication on the first preliminary compression result and the second preliminary compression result to obtain the second intermediate data.

[0094] In actual application, the processor 100 can first splice the first data and the first intermediate data, and divide the spliced data into multiple blocks according to a fixed length or an unfixed length, such as Figure 4 as shown. Then, the processor 100 can determine the first compression algorithm for the multiple blocks, and compress each block using the first compression algorithm to obtain a compression result corresponding to each block, such asFigure 4 As shown in FIG. 6, the processor 100 can calculate the fingerprints of each chunk, and compare the fingerprints of the plurality of chunks. When there are two chunks with the same fingerprints, the processor 100 can determine that the data in the two chunks are the same. Thus, the processor 100 can perform deduplication on the chunks of the identified same parts, and perform concatenation on the remaining plurality of chunks, such as sequentially concatenating in the order of the positions of the chunks in the data stream, etc., to obtain the second intermediate data, such as Figure 4 As shown in FIG. 6.

[0095] Thus, since the processor 100 has compressed the plurality of chunks (i.e., the first data and the first intermediate data) before calculating the chunk fingerprints and performing data deduplication, this can effectively reduce the amount of data involved in calculating the chunk fingerprints and performing data deduplication, thereby reducing the consumption of storage resources and computing resources. Moreover, since the first data and the first intermediate data are compressed based on the unified compression algorithm, the compression results obtained by compressing the same data are generally the same, so that the deduplication effect on the plurality of compressed chunks is consistent with the deduplication effect on the plurality of uncompressed chunks.

[0096] Further, after the processor 100 compresses the plurality of chunks, there can be some continuously same data between the compression results of the plurality of chunks. For example, as shown in FIG. 6, the starting data part in the chunk 3 and the starting data part in the chunk 4 are the same. At this time, the processor 100 can perform deduplication on the continuously same part of data in the plurality of chunks in the process of concatenating the plurality of chunks after deduplication, so as to further reduce the amount of the second intermediate data, so as to further reduce the data amount of the compression result generated subsequently. Figure 4

[0097] In actual application, when the fingerprints of the plurality of chunks are all different, the processor 100 can only concatenate the plurality of chunks, or perform deduplication on the continuously same part of data in the plurality of chunks before concatenation, etc., in the process of merging the first data and the first intermediate data, and the embodiment does not limit the specific implementation process of merging the first data and the first intermediate data.

[0098] S205: The processor 100 compresses the second intermediate data to obtain a compression result.

[0099] In this embodiment, the processor 100 can reduce the data amount of the second intermediate data by compressing the second intermediate data.

[0100] ​In the first implementation example, the processor 100 can sample the second intermediate data, for example, by using a random sampling algorithm, to obtain a small number of sampled data. Then, the processor 100 can extract features from the sampled data and determine one or more data types based on these features. For example, when the sampled data includes both numerical values ​​and strings, the processor 100 can calculate the ratio 1 (number of numerical values ​​relative to the total number of sampled data) and use the numerical values ​​and the ratio 1 as one feature of the sampled data. Furthermore, the processor 100 can also calculate the ratio 2 (number of string data relative to the total number of sampled data) and use the string data and the ratio 2 as another feature of the sampled data. Typically, since the sampled data is obtained by sampling the second intermediate data, the features of the sampled data are similar to those of the second intermediate data. Therefore, the processor 100 can use the features extracted from the sampled data as features of the second intermediate data. In practical applications, when the number of second intermediate data is small, the processor 100 can directly extract features from the second intermediate data to obtain its features.

[0101] Next, the processor 100 can determine the data type of the second intermediate data based on its characteristics. For example, when the ratio 1 corresponding to the numerical value in the second intermediate data is large (e.g., greater than 0.4), the processor 100 can determine that the second intermediate data belongs to the numeric type; simultaneously, when the ratio 2 corresponding to the string in the second intermediate data is large (e.g., greater than 0.5), the processor 100 can determine that the second intermediate data also belongs to the string type. In this case, the second intermediate data belongs to both the numeric and string types. However, when the ratio 1 corresponding to the numerical value in the second intermediate data is small, and the ratio 2 corresponding to the string is large, the processor 100 can also determine that the data type of the second intermediate data is only the string type.

[0102] Finally, the processor 100 can determine a compression algorithm that matches the data type of the second intermediate data based on the pre-saved correspondence between data types and compression algorithms. This algorithm is referred to as the second compression algorithm. The processor 100 then uses this second compression algorithm to compress the second intermediate data, obtaining the final compressed result for the original data. For example, when the data type of the second intermediate data includes numeric types, the processor 100 can determine that the applicable second compression algorithm is specifically the LZ compression algorithm. The correspondence between data types and compression algorithms can be pre-configured in the processor 100 by technicians based on experience, so that the processor 100 can determine a suitable compression algorithm for the second intermediate data based on this correspondence.

[0103] In the second implementation example, the processor 100 can compress the second intermediate data based on a second compression algorithm specified in advance to obtain the compression result. For example, the technician can pre-configure the processor 100 to compress the second intermediate data based on a specified LZ compression algorithm and an entropy encoding algorithm (such as a Huffman compression algorithm, etc.).

[0104] In the third implementation example, when the processor 100 generates the second intermediate data by performing lightweight compression on the first data and the first intermediate data, the processor 100 can also decompress the second intermediate data using a decompression algorithm corresponding to the first compression algorithm to obtain second decompressed data. Then, the processor 100 can determine a second compression algorithm for the second decompressed data, which can be a compression algorithm with higher complexity relative to the first compression algorithm (in general, a second compression algorithm with higher complexity also has a relatively better compression effect). For example, the second compression algorithm can include more data compression processes than the first compression algorithm, such as the first compression algorithm being an LZ compression algorithm and the second compression algorithm being an LZ compression algorithm + an entropy encoding algorithm, etc. Thus, the processor 100 can compress the second decompressed data using the second compression algorithm to obtain the final compression result for the original data. In this way, the processor 100 performs deep compression on the second decompressed data that has not been compressed using the second compression algorithm, which can further improve the data compression effect relative to directly compressing the second intermediate data using the second compression algorithm. That is, the data amount of the compression result obtained for the second decompressed data will be less than the data amount of the compression result obtained for the second intermediate data.

[0105] In actual applications, in addition to the above various implementation examples, the processor 100 can also compress the second intermediate data based on other manners to obtain the compression result, which is not limited. Moreover, the second compression algorithm used by the processor 100 to compress the second intermediate data can be one compression algorithm or can include multiple compression algorithms.

[0106] In this way, the processor 100 identifies the compressed second data in the original data, and merges and compresses the intermediate data obtained by decompressing the second data with the uncompressed data, which enables the same part of data to be re-deleted in the merging process, so that the data amount of the compression result finally generated by the processor 100 can be effectively less than the data amount of the compression result obtained by directly compressing the original data by the processor 100. Moreover, even if no data re-deletion is performed in the merging process, the processor 100 can also perform data compression based on the data similarity and regularity between the first data and the first intermediate data, thereby obtaining a higher compression effect for the original data.

[0107] Further, after the compression of the original data is completed and the data compression result is obtained, the processor 100 can also send the data compression result to one or more storage media, so as to persistently store the data compression result by using the storage media, as shown in Figure 2 . Thus, in the storage media, the storage space required for storing the data compression result is usually smaller than the storage space required for storing the original data, so as to reduce the occupation of the storage resource by the data storage.

[0108] Further, after the compression of the data is completed and the data is output to the storage media for persistent storage, the processor 100 can also feed back a notification message to the application through the communication interface, to notify the application that the data storage is completed, or to notify the application that the data is stored after compression, and the like, as shown in Figure 2 .

[0109] It should be noted that, according to the above description, other reasonable step combinations that can be thought of by those skilled in the art also belong to the protection scope of the present application. Secondly, those skilled in the art should also be familiar with the fact that the embodiments described in the specification all belong to preferred embodiments, and the actions involved are not necessarily required by the present application.

[0110] The above introduces the data processing method provided by the embodiments of the present application, and next, the structure of the data processing device and the computing device provided by the embodiments of the present application will be introduced in combination with the drawings. Figures 1 to 4 .

[0111] Referring to Figure 5 , a structural schematic diagram of a data processing device is shown. As shown in Figure 5 , the data processing device 500 includes:

[0112] The acquisition module 501 is configured to acquire original data to be processed.

[0113] The identification module 502 is configured to identify first data and second data in the original data, the first data being uncompressed data, and the second data being compressed data.

[0114] The decompression module 503 is configured to decompress the second data to obtain first intermediate data.

[0115] The merging module 504 is configured to merge the first data and the first intermediate data to obtain second intermediate data.

[0116] The compression module 505 is configured to compress the second intermediate data to obtain a compression result.

[0117] In a possible implementation, the original data is a data stream; and the identification module 502 is configured to:

[0118] divide the original data to obtain a plurality of data segments;

[0119] calculate a data disorder degree corresponding to each data segment in the plurality of data segments;

[0120] determine first data and second data according to the data disorder degree corresponding to each data segment, the data disorder degree corresponding to each data segment included in the second data being higher than the data disorder degree corresponding to each data segment included in the first data.

[0121] In a possible implementation, the original data is a data stream; the identification module 502 is configured to:

[0122] divide the original data to obtain a plurality of data segments;

[0123] determine first data and second data from the plurality of data segments, each data segment included in the second data being configured with compression-related information, the compression-related information being used to indicate a compression algorithm adopted by data in the data segment.

[0124] In a possible implementation, the first intermediate data includes first decompressed data and difference data; the decompression module 503 is configured to:

[0125] decompress the second data by using a decompression algorithm corresponding to the compression algorithm indicated by the compression-related information to obtain the first decompressed data;

[0126] compress the first decompressed data by using the compression algorithm indicated by the compression-related information to obtain re-compressed data;

[0127] generate the difference data according to the re-compressed data and the second data, the difference data being used to indicate a difference between the re-compressed data and the second data.

[0128] In a possible implementation, the compression-related information includes an identifier of a compression subject corresponding to the second data; the data processing apparatus 500 further includes:

[0129] The lookup module 506 is configured to find a mapping relationship according to the compression-related information to obtain the compression algorithm corresponding to the second data, the mapping relationship being used to indicate a compression algorithm that can be adopted by each compression subject in at least one compression subject.

[0130] In a possible implementation, the merging module 504 is configured to:

[0131] perform re-deletion on the first data and the first intermediate data to obtain second intermediate data.

[0132] In a possible implementation, the merging module 504 is configured to:

[0133] determine a first compression algorithm for the first data and the first intermediate data;

[0134] compress the first data by using the first compression algorithm to obtain a first preliminary compression result, and compress the first intermediate data by using the first compression algorithm to obtain a second preliminary compression result;

[0135] perform re-deletion on the first preliminary compression result and the second preliminary compression result to obtain the second intermediate data.

[0136] In a possible implementation, the compression module 505 is configured to compress the second intermediate data by using a second compression algorithm, and a complexity of the second compression algorithm is higher than a complexity of the first compression algorithm.

[0137] In a possible implementation, the compression module 505 is configured to:

[0138] perform decompression on the second intermediate data based on a decompression algorithm corresponding to the first compression algorithm to obtain second decompression data;

[0139] compress the second decompression data by using the second compression algorithm to obtain a compression result.

[0140] Because Figure 5 the data processing apparatus 500 shown in FIG. 5 corresponds to the processor 100 in the above Figure 2 embodiment, the data processing apparatus 500 has the same technical effects as the processor 100 in the above Figure 5 embodiment. Figure 2 embodiment, the data processing apparatus 500 has the same technical effects as the processor 100 in the above embodiment.

[0141] Figure 6 A hardware structure schematic diagram of a computing device 600 is provided in the present application, and the computing device 600 can implement the processor 100 in the above Figure 2 embodiment, or can implement the data processing apparatus 500 shown in FIG. 5, and the like. Figure 5

[0142] As Figure 6As shown, the computing device 600 includes a processor 601, a memory 602, and a communication interface 603. The processor 601, the memory 602, and the communication interface 603 communicate with each other through a bus 604, and can also communicate through wireless transmission and other means. The memory 602 is configured to store instructions, and the processor 601 is configured to execute the instructions stored in the memory 602. Further, the computing device 600 can also include a memory unit 605, which can also be connected to the processor 601, the memory medium 602, and the communication interface 603 through the bus 604. The memory 602 stores program codes, and the processor 601 can invoke the program codes stored in the memory 602 to perform the following operations:

[0143] Obtaining raw data to be processed;

[0144] Identifying first data and second data in the raw data, the first data being uncompressed data, and the second data being compressed data;

[0145] Decompressing the second data to obtain first intermediate data;

[0146] Merging the first data and the first intermediate data to obtain second intermediate data;

[0147] Compressing the second intermediate data to obtain a compression result.

[0148] It should be understood that, in the present embodiment, the processor 601 can be a CPU, and can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete device components, or the like. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0149] The memory 602 can include read-only memory and random access memory, and provide instructions and data to the processor 601. The memory 602 can also include non-volatile random access memory.

[0150] The memory 602 can be volatile memory or non-volatile memory, or it can include both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0151] The communication interface 603 is used to communicate with other devices connected to the computing device 600. The bus 604 may include a data bus, a power bus, a control bus, and a status signal bus, etc. However, for clarity, all buses are labeled as bus 604 in the figure.

[0152] It should be understood that the computing device 600 according to the embodiments of this application may correspond to the above. Figure 2 The processor 100 in the illustrated embodiment, or the processor corresponding to the above... Figure 5 The data processing apparatus 500 shown can be used to perform operations according to the embodiments of this application. Figure 2 The methods shown, the above and other operations and / or functions implemented by the computing device 500 are respectively for the purpose of implementing Figure 2 The process of the corresponding methods in [the document] will not be elaborated here for the sake of brevity.

[0153] The embodiments of the present application further provide a computer readable storage medium. The computer readable storage medium can be any available medium or data storage device that can be used to store the instructions that are capable of being executed by a computing device or a data center containing one or more available media. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk), etc. The computer readable storage medium includes instructions that instruct the computing device to execute the data processing method described above.

[0154] The embodiments of the present application further provide a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computing device, the flow or function described in the embodiments of the present application is generated in whole or in part.

[0155] The computer instructions can be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions can be transmitted from one website, computer or data center to another website, computer or data center through a wired (for example, coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (for example, infrared, wireless, microwave, etc.) manner.

[0156] The computer program product can be a software installation package, which can be downloaded and executed on a computing device in any case where the data processing method described above is needed.

[0157] The above-described embodiments can be implemented in part or in whole through software, hardware, firmware or any combination thereof. When implemented in software, the above-described embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed by a computer, the computer program instructions produce the above-described processes or functions described in the embodiments of the present application. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable apparatus. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, from a website site, a computer, a server, or a data center to another website site, a computer, a server, or a data center through a wired (for example, coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (for example, infrared, wireless, microwave, etc.) manner. The computer-readable storage medium can be any available medium accessible by a computer or a data storage device such as a server, data center, etc. containing a set of one or more available media. The available media can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid state disk.

[0158] The terms used in the above-described embodiments are only for the purpose of describing particular embodiments and are not intended to be limiting of the present application. As used in the specification and the appended claims of the application, the singular forms "a," "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term "one or more" as used in the application embodiments refers to one, two, or more than two. The character " / " generally indicates that the associated objects before and after the " / " are in an "or" relationship. In the embodiments of the present application, "simultaneously" means in the same time period, including the case of being at the same time. The terms "first", "second", and the like in the specification and claims of the present application and the accompanying drawings are used to distinguish similar objects, and are not necessarily used to describe a particular order or sequence. It should be understood that the terms used in this way can be interchanged as appropriate, and this is merely a way of distinguishing the objects of the same attribute used in describing the embodiments of the present application.

[0159] Reference throughout this specification to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of the phrase "in one embodiment" or "in an embodiment" in various places throughout this specification are not necessarily all referring to the same embodiment, but can refer to one or more but not all embodiments. Furthermore, the particular features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. The terms "including", "containing", and "having" are intended to be

[0160] The above description is only specific embodiments of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of various equivalent modifications or replacements within the technical range disclosed by the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A data processing method, characterized in that, The method includes: Obtain the raw data to be processed; Identify the first data and the second data in the original data, wherein the first data is uncompressed data and the second data is compressed data; The second data is decompressed to obtain the first intermediate data; The first data and the first intermediate data are combined to obtain the second intermediate data; The second intermediate data is compressed to obtain the compressed result.

2. The method according to claim 1, characterized in that, The original data is a data stream, and the identification of the first data and the second data in the original image includes: The original data is divided into multiple data segments; Calculate the degree of data disorder for each data segment among the multiple data segments; Based on the degree of data disorder corresponding to each data segment, the first data and the second data are determined, wherein the degree of data disorder corresponding to the data segments included in the second data is higher than the degree of data disorder corresponding to the data segments included in the first data.

3. The method according to claim 1, characterized in that, The original data is a data stream, and the identification of the first data and the second data in the original image includes: The original data is divided into multiple data segments; The first data and the second data are determined from the plurality of data segments, wherein each data segment of the second data is configured with compression-related information, which is used to indicate the compression algorithm used for the data in the data segment.

4. The method according to claim 3, characterized in that, The first intermediate data includes first decompressed data and difference data, and the decompression of the second data includes: The second data is decompressed using the decompression algorithm corresponding to the compression algorithm indicated by the compression-related information to obtain the first decompressed data; Using the compression algorithm indicated by the compression-related information, the first decompressed data is compressed to obtain recompressed data; Based on the recompressed data and the second data, difference data is generated, which is used to indicate the difference between the recompressed data and the second data.

5. The method according to claim 4, characterized in that, The compression-related information includes the identifier of the compression subject corresponding to the second data, and the method further includes: The compression algorithm corresponding to the second data is obtained by finding the mapping relationship based on the compression-related information. The mapping relationship is used to indicate the compression algorithm that can be used by each compression subject in at least one compression subject.

6. The method according to any one of claims 1 to 5, characterized in that, The process of merging the first data and the first intermediate data to obtain the second intermediate data includes: The first data and the first intermediate data are deleting data to obtain the second intermediate data.

7. The method according to claim 6, characterized in that, The step of deleting the first data and the first intermediate data to obtain the second intermediate data includes: A first compression algorithm is determined based on the first data and the first intermediate data; The first data is compressed using the first compression algorithm to obtain a first preliminary compression result, and the first intermediate data is compressed using the first compression algorithm to obtain a second preliminary compression result. The first preliminary compression result and the second preliminary compression result are then deleting to obtain the second intermediate data.

8. The method according to claim 7, characterized in that, The second intermediate data is compressed, including: The second intermediate data is compressed using a second compression algorithm, which has a higher complexity than the first compression algorithm.

9. The method according to claim 6 or 7, characterized in that, The compression of the second intermediate data to obtain the compression result includes: Based on the decompression algorithm corresponding to the first compression algorithm, the second intermediate data is decompressed to obtain the second decompressed data; The second decompressed data is compressed using a second compression algorithm to obtain the compression result.

10. A data processing apparatus, characterized in that, The device includes: The acquisition module is used to acquire the raw data to be processed. The identification module is used to identify the first data and the second data in the original data, wherein the first data is uncompressed data and the second data is compressed data; The decompression module is used to decompress the second data to obtain the first intermediate data; The merging module is used to merge the first data and the first intermediate data to obtain the second intermediate data. The compression module is used to compress the second intermediate data to obtain the compression result.

11. The apparatus according to claim 10, characterized in that, The original data is a data stream; The identification module is used for: The original data is divided into multiple data segments; Calculate the degree of data disorder for each data segment among the multiple data segments; Based on the degree of data disorder corresponding to each data segment, the first data and the second data are determined, wherein the degree of data disorder corresponding to the data segments included in the second data is higher than the degree of data disorder corresponding to the data segments included in the first data.

12. The apparatus according to claim 10, characterized in that, The original data is a data stream; The identification module is used for: The original data is divided into multiple data segments; The first data and the second data are determined from the plurality of data segments, wherein each data segment of the second data is configured with compression-related information, which is used to indicate the compression algorithm used for the data in the data segment.

13. The apparatus according to claim 12, characterized in that, The first intermediate data includes first decompressed data and difference data. The decompression module is used for: The second data is decompressed using the decompression algorithm corresponding to the compression algorithm indicated by the compression-related information to obtain the first decompressed data; Using the compression algorithm indicated by the compression-related information, the first decompressed data is compressed to obtain recompressed data; Based on the recompressed data and the second data, difference data is generated, which is used to indicate the difference between the recompressed data and the second data.

14. The apparatus according to claim 13, characterized in that, The compression-related information includes the identifier of the compression subject corresponding to the second data, and the device further includes: The lookup module is used to look up the mapping relationship based on the compression-related information to obtain the compression algorithm corresponding to the second data. The mapping relationship is used to indicate the compression algorithm that each compression subject in at least one compression subject can use.

15. The apparatus according to any one of claims 10 to 14, characterized in that, The merging module is used for: The first data and the first intermediate data are deleting data to obtain the second intermediate data.

16. The apparatus according to claim 15, characterized in that, The merging module is used for: A first compression algorithm is determined based on the first data and the first intermediate data; The first data is compressed using the first compression algorithm to obtain a first preliminary compression result, and the first intermediate data is compressed using the first compression algorithm to obtain a second preliminary compression result. The first preliminary compression result and the second preliminary compression result are then deleting to obtain the second intermediate data.

17. The apparatus according to claim 16, characterized in that, A compression module is used to compress the second intermediate data using a second compression algorithm, the second compression algorithm having a higher complexity than the first compression algorithm.

18. The apparatus according to claim 15 or 16, characterized in that, The compression module is used for: Based on the decompression algorithm corresponding to the first compression algorithm, the second intermediate data is decompressed to obtain the second decompressed data; The second decompressed data is compressed using a second compression algorithm to obtain the compression result.

19. A computing device, characterized in that, The computing device includes a processor and a memory; The processor is configured to execute instructions stored in the memory to cause the computing device to perform the steps of the method as described in any one of claims 1 to 9.

20. A storage system, characterized in that, The storage system includes at least one storage medium and a processor, the processor being configured to perform the method as described in any one of claims 1 to 9, wherein the at least one storage medium is configured to persistently store the data compression result.

21. A computer-readable storage medium, characterized in that, Includes instructions that, when executed on a computing device, cause the computing device to perform the steps of the method as described in any one of claims 1 to 9.

22. A computer program product containing instructions, characterized in that, When it is run on at least one computing device, it causes the at least one computing device to perform the steps of the method as described in any one of claims 1 to 9.

Images