Data compression method, apparatus, computing device, and storage system

Abstract

The application discloses a data compression method and device, a computing device and a storage system, and relates to the field of data compression. The method comprises the following steps: when the computing device compresses to-be-compressed data, determining a compression characteristic value of the to-be-compressed data according to a parameter that influences a compression result of the to-be-compressed data, determining a compression strategy for compressing the to-be-compressed data according to the compression characteristic value, compressing the to-be-compressed data according to the compression strategy to obtain compressed data, and storing the compressed data. Thus, compared with the fact that the computing device uses one compression algorithm to compress data with different characteristics, the compression strategy used by the data compression method provided in the application is selected according to characteristics that influence the compression result of the to-be-compressed data, and the to-be-compressed data is compressed by using the compression strategy that matches the characteristics of the to-be-compressed data, so that the compression rate of the data can be effectively improved.

Description

[0001] This application is a divisional application of the Chinese application with the application number 202111109332.X, the filing date of which is September 22, 2021, and both this application and the Chinese application with the application number 202111109332.X claim the priority of the Chinese patent application with the application number 202110773759.3, the filing date of which is July 8, 2021, and the title of which is "A data processing method, device and system", the whole content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to the field of data compression, in particular to a data compression method, device, computing device and storage system. BACKGROUND

[0003] With the prosperity of cloud computing and big data industry, the larger the data center is, the more data the data center stores. Larger data volume and longer storage period mean higher storage cost. At present, computing devices can use compression algorithms (such as LZ4, LZO or Snappy, etc.) to compress the data to be stored before storing, thereby reducing the data volume of the data to be stored and saving the corresponding hard disk, node, rack, site and operation and maintenance overhead, etc. In the computing device, when the user sets the compression algorithm, all data is compressed based on the set compression algorithm, resulting in a relatively low compression rate of the data. SUMMARY

[0004] The present application provides a data compression method, device, computing device and storage system, thereby improving the compression rate of data.

[0005] In a first aspect, a data compression method is provided, which can be executed by a computing device, and specifically includes the following steps: when the computing device compresses data to be compressed, determining a compression characteristic value of the data to be compressed according to a parameter affecting the compression result of the data to be compressed, determining a compression strategy for compressing the data to be compressed according to the compression characteristic value, compressing the data to be compressed according to the compression strategy to obtain compressed data, and storing the compressed data.

[0006] In this way, compared with the computing device using one compression algorithm to compress data with different characteristics, the compression strategy used by the computing device in the data compression method provided by the present application is selected according to the characteristics affecting the compression result of the data to be compressed, and the data to be compressed is compressed by using the compression strategy matching the characteristics of the data to be compressed, which can effectively improve the compression rate of data.

[0007] In a possible implementation, the parameters include parameters of hardware resources used when compressing the data to be compressed and / or parameters describing data characteristics of the data to be compressed. The parameters of the data characteristics include at least one of a data type, a data block size, and a distribution of characters included in the data to be compressed. The parameters of the hardware resources include at least one of an occupancy rate of a processor of the computing device, a network bandwidth between the computing device and a storage device when storing the compressed data to the storage device, and an available storage capacity of the storage device.

[0008] In another possible implementation, the computing device stores a plurality of corresponding relationships between compression characteristic values and compression strategies. The compression strategy for compressing the data to be compressed is determined according to the compression characteristic value, including: determining, by the computing device, a compression characteristic value in the corresponding relationship corresponding to the compression characteristic value of the data to be compressed; and determining, by the computing device, the compression strategy corresponding to the compression characteristic value in the corresponding relationship as the compression strategy for compressing the data to be compressed. In this way, by preconfiguring the plurality of corresponding relationships between the compression characteristic values and the compression strategies, the computing device can quickly and accurately select a compression strategy matching the characteristics of the data to be compressed when compressing the data to be compressed in real time.

[0009] In this way, the plurality of corresponding relationships between the compression characteristic values and the compression strategies can be obtained according to a neural network.

[0010] In another possible implementation, the compression strategy includes a plurality of compression windows, and the data to be compressed is compressed according to the compression strategy, including: compressing, by the computing device, the data to be compressed according to the plurality of compression windows to obtain a plurality of compressed data; and comparing, by the computing device, compression rates of the plurality of compressed data, and selecting, by the computing device, the compressed data with the highest compression rate as the compressed data. It can be understood that the compression window refers to a range for searching for the same character string in the data to be compressed. By searching for the same character string in the data to be compressed through a plurality of different compression windows, the lengths of the searched same character strings can be different, and therefore the compressed data with different compression rates can be obtained. In this way, the data to be compressed is compressed by using the multi-level elastic compression window, and the compressed data with the highest compression rate is obtained.

[0011] In another possible implementation, after obtaining the compressed data, the method further includes: determining, by the computing device, a compression rate of the compression strategy used when compressing the data; and adjusting, by the computing device, parameters of the compression characteristic values and the compression strategies in the corresponding relationship according to the compression rate. In this way, by updating the compression strategies in the corresponding relationship, the accuracy of the corresponding relationship is improved, so that the computing device can obtain a compression strategy that better matches the characteristics of the data to be compressed, and the compression rate of the data is improved.

[0012] In a second aspect, a data compression apparatus is provided, including various modules for performing the data compression method in the first aspect or any possible design in the first aspect.

[0013] In a third aspect, a computing device is provided, comprising at least one processor and a memory, the memory being configured to store a set of computer instructions; when the processor executes the set of computer instructions as the execution device in the first aspect or any possible implementation of the first aspect, the operation steps of the data compression method in the first aspect or any possible implementation of the first aspect are performed.

[0014] In a fourth aspect, a storage system is provided, comprising at least one computing node and at least one storage node, the computing node being configured to perform the operation steps of the data compression method in the first aspect or any possible implementation of the first aspect to obtain compressed data, and the storage node being configured to store the compressed data; or the computing node transmits to-be-compressed data to the storage node, the storage node performs the operation steps of the data compression method in the first aspect or any possible implementation of the first aspect to obtain compressed data, and stores the compressed data.

[0015] In a fifth aspect, a computer-readable storage medium is provided, comprising: computer software instructions; when the computer software instructions are run in a computing device, the computing device performs the operation steps of the method in the first aspect or any possible implementation of the first aspect.

[0016] In a sixth aspect, a computer program product is provided, when the computer program product is run on a computer, the computing device performs the operation steps of the method in the first aspect or any possible implementation of the first aspect.

[0017] On the basis of the implementation manners of the above aspects of the present application, further combinations can be made to provide more implementation manners. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 Architectural diagram of a distributed storage system according to an embodiment of the present application;

[0019] Figure 2 Architectural diagram of a disk control separation structure according to an embodiment of the present application;

[0020] Figure 3 Architectural diagram of a centralized storage system according to an embodiment of the present application;

[0021] Figure 4 Flowchart of a data compression method according to an embodiment of the present application;

[0022] Figure 5 Schematic diagram of determining a compression characteristic value of to-be-compressed data according to an embodiment of the present application;

[0023] Figure 6 A process diagram of dictionary compression according to an embodiment of the present application;

[0024] Figure 7 A process diagram of dictionary compression according to an embodiment of the present application;

[0025] Figure 8 A structure diagram of a data packet according to an embodiment of the present application;

[0026] Figure 9 A structure diagram of a data compression device according to an embodiment of the present application;

[0027] Figure 10 A structure diagram of a computing device according to an embodiment of the present application. DETAILED DESCRIPTION

[0028] Data compression refers to a method of reorganizing the data to be compressed according to a compression algorithm, so as to reduce the redundant data in the data to be compressed, thereby reducing the data amount of the data to be compressed, reducing the storage space required for storing the data to be compressed, and improving the compression rate, transmission speed and compression speed of the data to be compressed. The specific form of the computing device for data compression is not limited in the present application. The computing device may, for example, be a mobile terminal, a tablet computer, a notebook computer, a virtual reality (VR) device, an augmented reality (AR) device, a mixed reality (MR) device, an extended reality (ER) device, or a vehicle terminal. The computing device may also be an edge device (for example, a box with a processing-capable chip), a server, a computing-capable device in a distributed storage system, or a computing-capable device in a centralized storage system.

[0029] Figure 1 A structure diagram of a distributed storage system according to an embodiment of the present application. As shown in Figure 1 The distributed storage system provided by the present embodiment includes a computing node cluster and a storage node cluster.

[0030] The computing node cluster includes one or more computing nodes 110, Figure 1 Three computing nodes 110 are shown in the figure, but the number of computing nodes 110 is not limited to three. The computing nodes 110 can communicate with each other through a network. The computing node 110 is a kind of computing device, which is a node and hub of network data, and is responsible for providing information services for multiple clients or users in the network at the same time, such as servers, computers, etc.

[0031] In hardware, as Figure 1As shown, the computing node 110 includes at least a processor 112, a memory 113, and a network card 114. The processor 112 is a central processing unit (CPU) configured to process data access requests from outside the computing node 110 or internally generated requests.

[0032] For example, when the processor 112 receives a write data request sent by a user, the processor 112 temporarily stores the data in the write data request in the memory 113. When the total amount of data in the memory 113 reaches a certain threshold, the processor 112 sends the data stored in the memory 113 to the storage node 100 for persistent storage. In addition, the processor 112 is configured to perform calculations or processing on the data, such as metadata management, deduplication, data compression, virtual storage space, and address translation.

[0033] Figure 1 In the embodiment, only one CPU 112 is shown. In actual applications, the number of CPUs 112 in one computing node 110 can be multiple, and one CPU 112 can have one or more CPU cores. The number of CPUs and the number of CPU cores are not limited in the embodiment.

[0034] The memory 113 refers to an internal memory that exchanges data directly with the processor. It can read and write data at any time, and the speed is very fast, serving as a temporary data storage for an operating system or other programs running at the moment. The memory includes at least two types of memories, such as random access memory or read only memory (ROM). For example, the random access memory is dynamic random access memory (DRAM) or storage class memory (SCM).

[0035] DRAM is a kind of semiconductor memory, like most random access memory (RAM), it belongs to a kind of volatile memory device. SCM is a kind of composite storage technology combining traditional storage device and memory characteristics. Storage class memory can provide faster read and write speed than hard disk, but the access speed is slower than DRAM, and the cost is cheaper than DRAM. However, DRAM and SCM are only exemplary in the embodiment, the memory can also include other random access memory, such as static random access memory (SRAM) and the like. For read-only memory, for example, it can be programmable read-only memory (PROM), erasable programmable read-only memory (EPROM) and the like.

[0036] In addition, the memory 113 can also be a dual in-line memory module (DIMM), that is, a module composed of dynamic random access memory (DRAM), and can also be a solid state disk (SSD).

[0037] In practical application, multiple memories 113 can be configured in the computing node 110, and memories 113 of different types can be configured. The number and type of the memory 113 are not limited in the embodiment. In addition, the memory 113 can be configured to have a power retention function. The power retention function refers to that when the system is powered off and then powered on again, the data stored in the memory 113 will not be lost. The memory with the power retention function is called non-volatile memory.

[0038] The network card 114 is used for communication with the storage node 100. For example, when the total amount of data in the memory 113 reaches a certain threshold, the computing node 110 can send a request to the storage node 100 through the network card 114 to store the data persistently. In addition, the computing node 110 can also include a bus for communication between the components in the computing node 110. In terms of function, since Figure 1The main function of the computing node 110 is to compute the business, and the remote memory can be used to realize the persistent storage when storing the data, so it has less local memory than the conventional server, thereby realizing the cost and space saving. However, this does not mean that the computing node 110 cannot have the local memory, and in the actual implementation, the computing node 110 can also be built-in with a small amount of hard disk or externally connected with a small amount of hard disk. Any one of the computing nodes 110 can access any one of the storage nodes 100 in the storage node cluster through the network.

[0039] The storage node cluster can include a plurality of storage nodes 100 for persistently storing the data transmitted by the computing node, Figure 1 Three storage nodes 100 are shown in the figure, but are not limited to three storage nodes 100. For example, there can be more than three storage nodes, or less than three storage nodes.

[0040] One storage node 100 includes one or more control units 101, a network card 104 and a plurality of hard disks 105. The network card 104 is used for communication with the computing node 110. The hard disk 105 is used for storing data, which can be a magnetic disk or other types of storage media, such as a solid state disk or a shingled magnetic recording hard disk, etc. The control unit 101 is used to write data into the hard disk 105 or read data from the hard disk 105 according to the read / write data request sent by the computing node 110. In the process of reading and writing data, the control unit 101 needs to convert the address carried in the read / write data request into an address that can be recognized by the hard disk. As can be seen, the control unit 101 also has some simple computing functions.

[0041] In actual application, the control unit 101 can have various forms. In one case, the control unit 101 includes a CPU and a memory. The CPU is used to perform address conversion and read / write data operations. The memory is used to temporarily store data to be written into the hard disk 105 or data read from the hard disk 105 to be sent to the computing node 110. In another case, the number of control units 101 can be one, two or more. When the storage node 100 contains at least two control units 101, the hard disk 105 and the control unit 101 can have a belonging relationship. When the hard disk 105 and the control unit 101 have a belonging relationship, each controller can only access the hard disk belonging to it, so this often involves forwarding the read / write data request between the control units 101, resulting in a longer data access path. In addition, if the storage space is insufficient, when a new hard disk 105 is added to the storage node 100, the belonging relationship between the hard disk 105 and the control unit 101 needs to be re-bound, which is complicated and results in poor scalability of the storage space.

[0042] Therefore in another embodiment, the function of the control unit 101 can be offloaded to the network card 104. In other words, in Figure 1 In the embodiment shown, the storage node 100 does not have a control unit 101 inside, but the data read and write, address translation and other computing functions are completed by the network card 104. At this time, the network card 104 is an intelligent network card. It can contain a CPU and a memory. The CPU is used to perform address translation and read and write data operations. The memory is used to temporarily store data to be written to the hard disk 105 or data read from the hard disk 105 to be sent to the computing node 110. There is no ownership relationship between the network card 104 and the hard disk 105 in the storage node 100, and the network card 104 can access any hard disk 105 in the storage node 100, so it is more convenient to expand the hard disk when the storage space is insufficient.

[0043] Figure 1 The structure of the storage node shown belongs to the architecture of integrated control and disk, that is, the control unit and the hard disk 105 are included in the same physical device, but in some embodiments, the structure of the storage node can also be a structure of separated control and disk, such as Figure 2 As shown, the control and storage are two independent devices, that is, the control device 200 and the hard disk frame 300 are connected to each other. The control device 200 includes multiple control units, for example, control unit 0 and control unit 1, and the multiple control units are connected to the computing node 110 through the network card 104. Each control unit has the same structure, and the control unit 0 is taken as an example, which includes a front-end interface 201, a CPU 202, a memory 203, and a back-end interface 204. The front-end interface 201 is connected to the network card 104 to transmit the data transmitted by the computing node through the network card to the control unit 0, the back-end interface 204 is connected to the hard disk frame to transmit the data processed by the control unit 0 to the hard disk frame 300 for storage, the CPU 202 and the memory 203 are used to process the data transmitted by the computing node 110 or the data of the locally running program, and the specific description is referred to the description of the CPU 112 and the memory 113 in the computing node 110, which will not be repeated here. The hard disk frame 300 as an independent device includes a control unit 301 in addition to the hard disk 105, which is used to store the data transmitted by the computing node after processing and manage the hard disk 105.

[0044] Figure 1 The architecture diagram of the distributed storage system is shown. The scheme of the embodiment of the application can also be applied to a centralized storage system. In the centralized storage system, the data of the computing node is stored in a storage node, which can be a storage array for example. The architecture of the centralized storage system is shown. Figure 3 The structure of the storage node 100 and the computing node 110 is the same as that in Figure 1 , which will not be repeated here. Figure 3The shown storage node 100 is a disk control integrated structure. In the integrated storage system, the structure of the storage node can also be Figure 2 The shown disk control separated structure.

[0045] The data compression method provided by the embodiment of the present application, when the computing device compresses the to-be-compressed data, the computing device determines the compression characteristic value of the to-be-compressed data according to the parameter affecting the compression result of the to-be-compressed data, adaptively determines the compression strategy according to the compression characteristic value, and compresses the to-be-compressed data according to the compression strategy to obtain the compressed data. In this way, compared with the computing device using one compression algorithm to compress data of different characteristics, the compression strategy used by the computing device in the data compression method provided by the present application is selected according to the characteristics affecting the compression result of the to-be-compressed data, and the compression strategy matching the characteristics of the to-be-compressed data is used to compress the to-be-compressed data, which can effectively improve the compression rate of the data, so that the storage device stores more data and improves the storage space utilization rate of the storage device.

[0046] Next, the data compression method provided by the embodiment of the present application will be described in detail. Figures 4 to 8 The data compression method provided by the embodiment of the present application will be described in detail. Figure 4 A flowchart of a data compression method provided by the embodiment of the present application is shown in FIG. 1. As shown in the figure, the method includes the following steps. Figure 4 The shown method includes the following steps.

[0047] Step 410, the computing device acquires to-be-compressed data.

[0048] The computing device can receive the to-be-compressed data sent by an external device (such as a keyboard, a mouse, a camera, a voice receiver, etc.); or the computing device generates the to-be-compressed data by running an application. If the computing device is a storage device, the computing device can also receive the to-be-compressed data sent by another computing device (such as a server) through a network. The to-be-compressed data can be text, numbers, letters, images, and voice, etc.

[0049] Step 420, the computing device collects a parameter affecting the compression result of the to-be-compressed data.

[0050] The parameter affecting the compression result of the to-be-compressed data includes a parameter of a hardware resource used when the to-be-compressed data is compressed and / or a parameter describing the data characteristics of the to-be-compressed data.

[0051] The computing device collects the parameter of the hardware resource used when the to-be-compressed data is compressed in real time within a preset time period (such as several seconds or tens of milliseconds) before the to-be-compressed data is compressed. The parameter of the hardware resource includes at least one of the processor occupancy rate of the computing device, the network bandwidth between the computing device and the storage device when the compressed data is stored to the storage device, the bandwidth of the storage device, and the available storage capacity of the storage device, etc.

[0052] The processor occupancy represents the busy state of the processor. If the processor occupancy is greater than or equal to the threshold, it indicates that the processor is in the busy state, and the available computing resources are less. Conversely, if the processor occupancy is less than the threshold, it indicates that the processor is in the idle state, and the available computing resources are more.

[0053] It can be understood that, in order to search as many same strings as possible from the to-be-compressed data and improve the compression rate of the to-be-compressed data, the compression algorithm used by the computing device can be relatively complex. If the computing device compresses the to-be-compressed data by using the compression algorithm with high complexity, the computing device needs to occupy more computing resources. When the processor is in the busy state, if the computing device occupies more computing resources for data compression, the compression time can be relatively long due to the less available computing resources. When the processor is in the idle state, since the available computing resources are more, even if the computing device occupies more computing resources for data compression, the compression time is relatively short compared with when the processor is in the busy state. Therefore, when the processor is in the busy state, the computing device can compress the to-be-compressed data by using a compression algorithm with low complexity, to shorten the compression time as much as possible under the premise of ensuring the compression rate of the data. When the processor is in the idle state, the computing device can compress the to-be-compressed data by using a compression algorithm with high complexity, to improve the compression rate of the data as much as possible under the premise of ensuring the compression time.

[0054] The computing device and the storage device can be two independent physical devices. For example, the computing node 110 or the storage node 100 in the distributed storage system shown in the foregoing embodiment. Figure 1 The computing device transmits the compressed data to the storage device through a network. The network is, for example, a storage area network (SAN). The network bandwidth between the computing device and the storage device represents the amount of data transmitted by the computing device and the storage device in a unit of time (for example, 1 second). The greater the network bandwidth between the computing device and the storage device, the more the amount of data transmitted by the computing device and the storage device in a unit of time; the smaller the network bandwidth between the computing device and the storage device, the less the amount of data transmitted by the computing device and the storage device in a unit of time.

[0055] It can be understood that when the network bandwidth between the computing device and the storage device is small, if the computing device compresses the data by using a compression algorithm with a high compression rate, the data amount of the compressed data is small, and the transmission time of the compressed data is short. When the network bandwidth between the computing device and the storage device is large, the data amount transmitted by the computing device and the storage device in a unit of time is large, even if the data amount of the compressed data is large, the transmission time is relatively short when the network bandwidth is small. Therefore, when the network bandwidth between the computing device and the storage device is small, the computing device can compress the data to be compressed by using a compression algorithm with a high compression rate, and improve the compression rate of the data as much as possible under the premise of ensuring the transmission time.

[0056] The bandwidth of the storage device represents the data amount accessed by the storage device in a unit of time (for example, 1 second). It can be understood that when the bandwidth of the storage device is small, if the computing device compresses the data by using a compression algorithm with a high compression rate, the data amount of the compressed data is small, and the time of accessing the compressed data by the storage device is short. Therefore, when the bandwidth of the storage device is small, the computing device can compress the data to be compressed by using a compression algorithm with a high compression rate, and improve the compression rate of the data as much as possible under the premise of ensuring the access time.

[0057] The available storage capacity of the storage device refers to the remaining available storage space of the storage device for storing compressed data. It can be understood that the larger the remaining available storage capacity of the storage device is, the larger the data amount that can be stored is. The smaller the remaining available storage capacity of the storage device is, the smaller the data amount that can be stored is. When the remaining available storage capacity of the storage device is small, the computing device can compress the data to be compressed by using a compression algorithm with a high compression rate, and reduce the data amount of the compressed data as much as possible, and improve the utilization rate of the storage space of the storage device.

[0058] The computing device can analyze the data to be compressed to obtain a parameter of a data feature. The parameter of the data feature includes at least one of a data type, a data block size, and a distribution of characters contained in the data to be compressed.

[0059] The data type includes text, numbers, letters, images, and the like. It can be understood that the repetition degree of characters in different data types of data can be different. For example, the repeated characters contained in numbers, letters, and images can be more. The computing device can compress the data to be compressed by using a compression algorithm with a high complexity, search for as many same strings as possible, and improve the compression rate of the data.

[0060] The data block size refers to a unit data amount for which the computing device performs data compression. For example, the data block size ranges from 8 kilobytes (KB) to 256 KB. The data block size can be pre-set by a system administrator. Understandably, the smaller the data block size, the fewer the same characters in the data block; the larger the data block size, the more the same characters in the data block. When the data block size is large, the computing device can use a compression algorithm with high complexity to compress the data to be compressed, search for as many same strings as possible, and improve the compression rate of the data.

[0061] The distribution of characters contained in the data to be compressed represents the distribution rule of the characters contained in the data to be compressed. For example, the data to be compressed contains two characters a and b, and the distribution rule of a and b can be aaabbb, ababab, or abbabb, etc. Understandably, for the data to be compressed containing more repeated characters, the computing device can use a compression algorithm with high complexity to compress the data to be compressed, search for as many same strings as possible, and improve the compression rate of the data.

[0062] In actual applications, the parameters needed can be determined according to specific application scenarios, which can be at least one of the hardware resource parameters, at least one of the parameters representing data characteristics, or a combination of the hardware resource parameters and the data characteristic parameters. The specific parameter values can be system defaults or set by the user as needed.

[0063] Step 430, the computing device determines the compression characteristic value of the data to be compressed according to the parameters affecting the compression result of the data to be compressed.

[0064] Since the dimensions of the parameters collected by the computing device are often inconsistent, in order to facilitate the comprehensive consideration of the influence of each parameter on the data compression result, the computing device can normalize each parameter to obtain a parameter with unified dimensions. In some embodiments, the computing device can structure the parameters to obtain the compression characteristic value of the data to be compressed. The compression characteristic value can be a feature vector. One element in the feature vector represents one parameter.

[0065] For example, I A represents the feature vector of the data to be compressed, I A =[a1, a2, a3, a4, a5, a6, a7], wherein the element a1 represents the occupation rate vector of the processor of the computing device, the element a2 represents the network bandwidth vector between the computing device and the storage device, the element a3 represents the bandwidth vector of the storage device, the element a4 represents the available storage capacity vector of the storage device, the element a5 represents the data type vector, the element a6 represents the data block size vector, and the element a7 represents the distribution vector of the characters contained in the data to be compressed.

[0066] Optionally, to improve the accuracy of matching the compression characteristic value of the to-be-compressed data to the compression strategy according to the parameter affecting the compression result of the to-be-compressed data, and eliminate redundant parameters, the computing device can also perform merging processing on parameters having common characteristics (such as the same dimension unit). For example, the computing device can merge the network bandwidth between the computing device and the storage device and the bandwidth of the storage device.

[0067] The computing device stores a plurality of corresponding relationships between compression characteristic values and compression strategies. The computing device can calculate the similarity between the compression characteristic value of the to-be-compressed data and each compression characteristic value in the corresponding relationship, and determine the compression characteristic value with the highest similarity in the corresponding relationship as the compression characteristic value matched to the compression characteristic value of the to-be-compressed data.

[0068] In some embodiments, the computing device calculates the Hamming distance between the compression characteristic value of the to-be-compressed data and each compression characteristic value in the corresponding relationship, to obtain a plurality of Hamming distances. The shorter the Hamming distance, the more similar the compression characteristic value of the to-be-compressed data is to the compression characteristic value of the corresponding relationship. The computing device can compare the plurality of Hamming distances, and select the compression characteristic value with the shortest Hamming distance as the compression characteristic value determined by the corresponding relationship. The Hamming distance between the compression characteristic value of the to-be-compressed data and the compression characteristic value of the corresponding relationship satisfies formula (1).

[0069]

[0070] wherein, I A represents the compression characteristic value of the to-be-compressed data, I B represents the compression characteristic value of the corresponding relationship, a i represents an element in the compression characteristic value of the to-be-compressed data, b i represents an element in the compression characteristic value of the corresponding relationship, n represents the number of elements, and D ham represents the Hamming distance between the compression characteristic value of the to-be-compressed data and the compression characteristic value of the corresponding relationship.

[0071] In other embodiments, the computing device can also calculate the Euclidean distance between the compression characteristic value of the to-be-compressed data and each compression characteristic value in the corresponding relationship, to obtain a plurality of Euclidean distances. The computing device can compare the plurality of Euclidean distances, and select the compression characteristic value with the shortest Euclidean distance as the compression characteristic value determined by the corresponding relationship.

[0072] In some embodiments, the computing device can select the compressed feature value determined by the corresponding relationship from the plurality of Hamming distances or the plurality of Euclidean distances according to a preset distance range. If there are multiple compressed feature values determined by the corresponding relationship of the Hamming distances or the Euclidean distances within the preset range, the computing device can select the compressed feature value determined by the corresponding relationship according to the nearest neighbor distance ratio, thereby improving the accuracy of the compressed feature value in the matching corresponding relationship. The nearest neighbor distance ratio satisfies formula (2).

[0073]

[0074] wherein β represents the nearest neighbor distance ratio threshold, and the value range of β is 0-1. For example, as shown in (a) of FIG. 1, points B1 and B2 represent two different compressed feature values in the corresponding relationship. Point O represents the compressed feature value of the data to be compressed. Within the preset distance range, the Hamming distance or the Euclidean distance between point O and points B1 and B2 is similar. As shown in (b) of FIG. 1, the computing device determines that the compressed feature value represented by point B1 is the compressed feature value determined by the corresponding relationship according to formula (2). Figure 5 Figure 5

[0075] In some embodiments, the computing device can also calculate the normalized cross-correlation value of the compressed feature value of the data to be compressed and each compressed feature value in the corresponding relationship, to obtain a plurality of normalized cross-correlation values. The closer the normalized cross-correlation value is to 1, the more similar the compressed feature value of the data to be compressed is to the compressed feature value of the corresponding relationship. The closer the normalized cross-correlation value is to 0, the less similar the compressed feature value of the data to be compressed is to the compressed feature value of the corresponding relationship. The computing device determines the compressed feature value with the normalized cross-correlation value closest to 1 as the compressed feature value determined by the corresponding relationship. The normalized cross-correlation value of the compressed feature value of the data to be compressed and the compressed feature value of the corresponding relationship satisfies formula (3).

[0076]

[0077] wherein NCC(I A ,I B ) represents the normalized cross-correlation value of the compressed feature value of the data to be compressed and the compressed feature value of the corresponding relationship, S a represents the variance of the compressed feature value of the data to be compressed. S b represents the variance of the compressed feature value of the corresponding relationship. u a represents the average value of the compressed feature value of the data to be compressed. u b represents the average value of the compressed feature value of the corresponding relationship.

[0078] Step 440, the computing device determines the compression strategy for compressing the data to be compressed according to the compressed feature value.

[0079] ​​The computing device determines the compression strategy corresponding to the compression characteristic value determined according to the correspondence as the compression strategy of compressing the data to be compressed.

[0080] It can be understood that the correspondence between the plurality of compression characteristic values and the compression strategies is a pre-set template library, that is, the template library contains a plurality of entries, and each entry contains a compression characteristic value and a compression strategy corresponding to the compression characteristic value. In an example, the compression strategy indicates a compression algorithm corresponding to the compression characteristic value. The compression algorithms corresponding to different compression characteristic values in the template library are different. The compression algorithms include but are not limited to: Shannon coding, Huffman coding, dictionary compression algorithms (such as: LZ77, LZSS, LZ78 and LZW). In another example, the compression strategy indicates the parameters of the compression algorithm corresponding to the compression characteristic value, for example, the size of the dictionary area in the dictionary compression algorithm. The different parameters of the same compression algorithm corresponding to different compression characteristic values in the template library.

[0081] The compression strategy corresponding to the compression characteristic value in the correspondence is set according to the influence of the parameters of the hardware resources and the parameters of the data characteristics on the compression rate in step 420, that is, the compression algorithm or the parameters of the compression algorithm indicated by the compression strategy corresponding to the compression characteristic value in the correspondence is set according to the principle of as high compression rate as possible.

[0082] For example, the compression characteristic value is the processor occupancy rate. If the processor occupancy rate is less than a threshold value, the compression strategy corresponding to the processor occupancy rate indicates a dictionary compression algorithm with a longer dictionary area. If the processor occupancy rate is greater than or equal to the threshold value, the compression strategy corresponding to the processor occupancy rate indicates a dictionary compression algorithm with a shorter dictionary area.

[0083] For another example, the compression characteristic value is the remaining available storage capacity of the storage device. If the remaining available storage capacity of the storage device is small, the compression strategy corresponding to the remaining available storage capacity of the storage device indicates a dictionary compression algorithm with a longer dictionary area. If the remaining available storage capacity of the storage device is large, the compression strategy corresponding to the remaining available storage capacity of the storage device indicates a dictionary compression algorithm with a shorter dictionary area.

[0084] For another example, the compression characteristic value is the network bandwidth between the computing device and the storage device or the bandwidth of the storage device. If the network bandwidth or the bandwidth of the storage device is small, the compression strategy corresponding to the remaining available storage capacity of the storage device indicates a dictionary compression algorithm with a longer dictionary area.

[0085] For another example, the compression characteristic value is the data block size. If the data block size is large, the compression strategy corresponding to the remaining available storage capacity of the storage device indicates a dictionary compression algorithm with a longer dictionary area.

[0086] For another example, the compression characteristic value is a distribution of characters contained in the data to be compressed. If the data to be compressed contains more repeated characters, the compression strategy corresponding to the remaining available storage capacity of the storage device indicates the dictionary compression algorithm of a longer dictionary area.

[0087] In some embodiments, the compression strategy corresponding to the compression characteristic value can be set by comprehensively considering the hardware resource parameter and the data characteristic parameter, thereby improving the compression rate of the data to be compressed.

[0088] For example, the compression characteristic value includes the occupancy rate of the processor and the network bandwidth between the computing device and the storage device. Assuming that the occupancy rate of the processor is greater than or equal to a threshold value, the available computing resource of the processor is less, and the network bandwidth between the computing device and the storage device is large, which means that the computing device and the storage device transmit more data in a unit of time. Then, the compression strategy corresponding to the compression characteristic value indicates the dictionary compression algorithm of a shorter dictionary area. The computing device can use the dictionary compression algorithm of a shorter dictionary area to compress the data to be compressed, so that the computing device can occupy less computing resource and shorten the compression time as much as possible under the premise of ensuring the compression rate of the data. Since the network bandwidth between the computing device and the storage device is large, the computing device can also transmit the compressed data faster.

[0089] For another example, the compression characteristic value includes the occupancy rate of the processor and the available storage capacity of the storage device. Assuming that the occupancy rate of the processor is less than a threshold value and the available storage capacity of the storage device is less, the compression strategy corresponding to the compression characteristic value indicates the dictionary compression algorithm of a longer dictionary area. Since the available computing resource of the processor is more, the computing device can use the dictionary compression algorithm of a longer dictionary area to compress the data to be compressed, so as to reduce the data amount of the compressed data as much as possible, save the storage space of the storage device, and improve the utilization rate of the storage space of the storage device.

[0090] For another example, the compression characteristic value includes the occupancy rate of the processor and the distribution of characters contained in the data to be compressed. Assuming that the occupancy rate of the processor is less than a threshold value and the data to be compressed contains more repeated characters, the compression strategy corresponding to the compression characteristic value indicates the dictionary compression algorithm of a longer dictionary area. Since the available computing resource of the processor is more, the computing device can use the dictionary compression algorithm of a longer dictionary area to compress the data to be compressed, search for as many same strings as possible, reduce the data amount of the compressed data as much as possible, and improve the compression rate of the data.

[0091] For example, the compression characteristic value includes the processor occupancy rate, the available storage capacity of the storage device, and the distribution of characters contained in the data to be compressed. Assuming that the processor occupancy rate is less than a threshold value, the available storage capacity of the storage device is small, and the data to be compressed contains more repeated characters, the compression strategy corresponding to the compression characteristic value indicates the dictionary compression algorithm of the long dictionary area. Since the available computing resources of the processor are more and the data to be compressed contains more repeated characters, the computing device can use the dictionary compression algorithm of the long dictionary area to compress the data to be compressed, thereby reducing the data amount of the compressed data as much as possible, saving the storage space of the storage device, and improving the utilization rate of the storage space of the storage device.

[0092] In actual applications, when setting the compression strategy corresponding to the compression characteristic value according to the compression characteristic value, at least one of the hardware resource parameters can be considered, at least one of the data characteristic parameters can be considered, or the hardware resource parameters and the data characteristic parameters can be considered comprehensively.

[0093] Therefore, since the compression strategy is set according to the influence of the parameters of the hardware resources and the parameters of the data characteristics on the compression result of the data to be compressed, that is, the compression algorithm or the parameters of the compression algorithm indicated by the compression strategy corresponding to the compression characteristic value in the corresponding relationship is set according to the principle of the highest compression rate, the computing device selects the compression strategy corresponding to the compression characteristic value similar to the compression characteristic value of the data to be compressed from the corresponding relationship, and uses the compression strategy to compress the data to be compressed, which can effectively improve the compression rate of the data.

[0094] It should be noted that the corresponding relationship can be pre-set by a system administrator according to experience. Alternatively, the corresponding relationship can be obtained by training a neural network. For example, the computing device labels the compression strategy corresponding to the compression characteristic value, inputs the compression characteristic value and the labeled compression strategy into the neural network, trains the neural network until the loss function in the neural network converges and the loss function value is less than a specific threshold value, and then the neural network training is completed, so that the neural network has the function of predicting the compression strategy according to the compression characteristic value and reaches the prediction accuracy. The computing device inputs the compression characteristic value of the data to be compressed into the trained neural network to obtain the compression strategy of the compression characteristic value of the data to be compressed. It should be noted that the computing device for training the neural network and the computing device for determining the compression strategy of the data to be compressed can be the same device or different devices, which are not limited.

[0095] Step 450, the computing device compresses the data to be compressed according to the compression strategy to obtain compressed data.

[0096] The compression strategy can refer to a dictionary compression algorithm. The basic idea of the dictionary compression algorithm is to replace original data in the data to be compressed with position information to achieve compression, and to restore the data according to the position information to achieve decompression. Specifically, the dictionary compression algorithm searches for a string in the dictionary area, obtains the longest string matched with the to-be-encoded area, returns the distance between the start character of the longest string and the boundary between the dictionary area and the to-be-encoded area, and the matching length (literal length) of the matched string. Then, the compression window is shifted right by the matching length + 1 characters, and the search is continued until the to-be-encoded data is encoded, so as to achieve the purpose of data compression. The output of the dictionary compression algorithm is a triple (offset, matching length, character). The offset represents the offset distance of the longest string. The offset can refer to the distance of the longest string from the left of the dictionary area or the distance of the longest string from the right of the dictionary area (the boundary between the dictionary area and the to-be-encoded area). The matching length represents the length of the matched string. The character represents the next character after the matching.

[0097] The specific process of dictionary compression is illustrated below. For example, it is assumed that the to-be-encoded data is aacaacabcabaaac. The size of the dictionary area is 6 characters, and the size of the to-be-encoded area is 4 characters. Since the dictionary is dynamically generated, the dictionary is empty at the beginning of encoding, and the dictionary contains different characters during the encoding process.

[0098] As shown in (a) of FIG. 1, Figure 6 At the beginning of encoding, the left end of the to-be-encoded area is aligned with the left end of the to-be-encoded data (the boundary between the dictionary area and the to-be-encoded area), and the to-be-encoded area includes the characters aaca, the dictionary is empty, and there is no matching character in the dictionary. Therefore, the offset is 0, and the matching length is 0. Since no identical character is matched, the character in the triple is the first character a of the to-be-encoded data. Therefore, the output triple is <0, 0, a>.

[0099] As shown in (b) of FIG. 1, Figure 6 The compression window is shifted right by len (matching length 0) + 1 characters, and the dictionary area contains the character a. The to-be-encoded area includes the characters acaa. The longest string matched between the dictionary area and the to-be-encoded area is a, the matching string is a, the offset is 1, the matching length is 1, and the next character after the matching is c. Therefore, the output triple is <1, 1, c>.

[0100] As shown in (c) of FIG. 1, Figure 6As shown in (c) in the example, if the compression window is shifted right by len (match length 1) + 1 characters, that is, by 2 characters, the dictionary area contains the characters aac. The encoding area contains the characters aaca. The longest matching character string in the dictionary and encoding areas is aac, and the matching character is aaca. Therefore, the offset is 3, the match length is 4, and the next character after the match is b. Therefore, the output triple is <3,4,b>.

[0101] like Figure 6 As shown in (d) in the example, if the compression window is shifted right by len (match length 4) + 1 characters, that is, by 5 characters, the dictionary area contains the characters caacab. The encoding area contains the characters caba. The longest matching character string between the dictionary area and the encoding area is caba. If the matching character is cab, the offset is 3, the match length is 3, and the next character after the match is a. Therefore, the output triple is <3,3,a>.

[0102] like Figure 6 As shown in (e) in the example, if the compression window is shifted right by len (match length 3) + 1 characters, that is, by 4 characters, the dictionary area contains the characters abcaba. The encoding area contains the characters aac. The longest matching string between the dictionary area and the encoding area is a. If the matching string is aa, the offset is 1, the match length is 2, and the next character after the match is c. Therefore, the output triple is <1,2,c>.

[0103] like Figure 6 As shown in (f) in the figure, the compression window is moved right by len (match length 2) + 1 characters, that is, the compression window is moved right by 3 characters. The dictionary area contains the characters abaaac. The area to be encoded contains no characters, and the encoding is complete.

[0104] Therefore, after the computing device compresses the encoded data aacaacabcabaaac using a dictionary compression algorithm, the resulting triple sequence includes <0,0,a>, <1,1,c>, <3,4,b>, <3,3,a>, and <1,2,c>.

[0105] As the length of the dictionary area varies, the dictionary area may also contain different characters, so during the dictionary compression process, the matching strings found will also be different. For example, Figure 7 FIG. 1 is a schematic diagram of another dictionary compression process provided by an embodiment of the present application, wherein the length of the dictionary area is modified from 6 characters to 2 characters. Figure 7 References to (a) and (b) Figure 6 Description of (a) and (b) in .

[0106] likeFigure 7 As shown in (c) of FIG. 10, the compression window is moved right by len (match length 1) + 1 characters, i.e., the compression window is moved right by 2 characters, then the dictionary area contains the characters ac. The area to be encoded includes the characters aaca. There is no matching character in the dictionary, the longest matching string in the dictionary area and the area to be encoded is 0, then the offset is 0, the match length is 0, and the next character after the match is a. Therefore, the output triple is <0, 0, a>.

[0107] As shown in (d) of FIG. 10, the compression window is moved right by len (match length 0) + 1 characters, i.e., the compression window is moved right by 1 character, then the dictionary area contains the characters ca. The area to be encoded includes the characters acba. The longest matching string in the dictionary area and the area to be encoded is a, the matching string is a, then the offset is 1, the match length is 1, and the next character after the match is c. Therefore, the output triple is <1, 1, c>. Figure 7 As shown in (e) of FIG. 10, the compression window is moved right by len (match length 1) + 1 characters, i.e., the compression window is moved right by 2 characters, then the dictionary area contains the characters ac. The area to be encoded includes the characters abca. There is no matching character in the dictionary, the longest matching string in the dictionary area and the area to be encoded is 0, then the offset is 0, the match length is 0, and the next character after the match is a. Therefore, the output triple is <0, 0, a>.

[0108] Figure 7 The characters bcaba after aacacac have no matching characters in the dictionary, the longest matching string in the dictionary area and the area to be encoded is 0, the offset is 0, the match length is 0, and the output triple contains <0, 0, b>, <0, 0, c>, <0, 0, a>, <0, 0, b>, <0, 0, a>.

[0109] As shown in (f) of FIG. 10, the compression window is moved right by len (match length 0) + 1 characters, i.e., the compression window is moved right by 1 character, then the dictionary area contains the characters ba. The area to be encoded includes the characters aac. The longest matching string in the dictionary area and the area to be encoded is a, the matching string is aa, then the offset is 1, the match length is 2, and the next character after the match is c. Therefore, the output triple is <1, 2, c>.

[0110] As shown in (g) of FIG. 10, the compression window is moved right by len (match length 2) + 1 characters, i.e., the compression window is moved right by 3 characters, then the dictionary area contains the characters ac. The area to be encoded is empty, and the encoding is completed. Figure 7 As shown in (h) of FIG. 10, the compression window is moved right by len (match length 2) + 1 characters, i.e., the compression window is moved right by 3 characters, then the dictionary area contains the characters ac. The area to be encoded includes the characters aac. There is no matching character in the dictionary, the longest matching string in the dictionary area and the area to be encoded is 0, then the offset is 0, the match length is 0, and the next character after the match is a. Therefore, the output triple is <0, 0, a>.

[0111] Figure 7 As shown in (i) of FIG. 10, the compression window is moved right by len (match length 0) + 1 characters, i.e., the compression window is moved right by 1 character, then the dictionary area contains the characters ba. The area to be encoded includes the characters aac. The longest matching string in the dictionary area and the area to be encoded is a, the matching string is aa, then the offset is 1, the match length is 2, and the next character after the match is c. Therefore, the output triple is <1, 2, c>.

[0112] ​​Therefore, after the computing device compresses the encoded data aacaacabcabaaac using the dictionary compression algorithm, the resulting triple sequence includes <0,0,a>, <1,1,c>, <0,0,a>, <1,1,c>, <0,0,a>, <0,0,b>, <0,0,c>, <0,0,a>, <0,0,b>, <0,0,a> and <1,2,c>.

[0113] Depend on Figure 6 and Figure 7 It can be seen that the compressed data obtained by encoding the length of the dictionary area as 6 characters contains 5 triples, and the compressed data obtained by encoding the length of the dictionary area as 2 characters contains 11 triples. The smaller the dictionary area, the lower the compression rate. Figure 6 and Figure 7 The compression rate is 0.0, the compression window is selected as 6 characters, and the compressed data are <0,0,a>, <1,1,c>, <3,4,b>, <3,3,a> and <1,2,c>.

[0114] Understandably, if the dictionary area is smaller and contains fewer characters, the computing device will find fewer matching characters in the area to be encoded, resulting in a lower compression rate. If the dictionary area is larger and contains more characters, the computing device will likely find more matching characters in the area to be encoded, resulting in a higher compression rate. However, this will require the computing device to spend more time searching for duplicate characters, resulting in slower compression speeds and, in turn, impacting system performance.

[0115] In some embodiments, the compression policy may indicate at least two compression algorithms, causing the computing device to select the compression algorithm with the highest compression ratio from the at least two compression algorithms to compress the data to be compressed. Alternatively, the compression policy may indicate the values ​​of multiple parameters in the compression algorithm, causing the computing device to select the parameter values ​​of the compression algorithm with the highest compression ratio from the multiple parameter values ​​to compress the data to be compressed.

[0116] For example, a compression strategy may include multiple compression windows. A compression window refers to the dictionary area in a dictionary compression algorithm. A compression strategy includes multiple dictionary areas of different sizes. The computing device compresses the data to be compressed according to the multiple compression windows to obtain multiple compressed data. The computing device compares the compression ratios of the multiple compressed data and selects the compressed data with the highest compression ratio as the compressed data. Therefore, after compressing the data to be compressed using multiple compression windows of different sizes, the computing device selects the compressed data with the highest compression ratio as the compressed data, thereby further improving the data compression ratio.

[0117] In some other embodiments, the computing device can further divide the data to be compressed into a plurality of data blocks to be compressed according to the data block size, and compress each data block to be compressed according to the data compression method provided in the embodiments of the present application, so as to further improve the compression rate of the data.

[0118] In some other embodiments, the computing device can divide the data to be compressed into data blocks to be compressed according to the data block size, and store the hash value in the dictionary area after performing hash operation on the data block to be compressed. The computing device matches the hash value of the data block to be compressed with the hash value in the dictionary area, and performs compression if the matching is successful, or stores the new hash value in the dictionary area if the matching is unsuccessful. Since different hash algorithms can be used to perform hash operation on different data blocks to be compressed, the probability of hash collision is effectively reduced compared with using one hash algorithm to perform hash operation on the data to be compressed.

[0119] Optionally, the compression strategy can further indicate whether the compressed data is compressed again. For example, the compression strategy includes a re-compression identifier. The value of the re-compression identifier is 1, indicating that the compressed data is compressed again. The value of the re-compression identifier is 0, indicating that the compressed data is not compressed again. The computing device can perform entropy encoding on the compressed data to further improve the compression rate of the data.

[0120] Optionally, after obtaining the compressed data, the computing device can further adjust the parameters of the compression characteristic value and the compression strategy in the correspondence, so as to improve the accuracy of the correspondence. For example, the method further includes step 460 and step 470.

[0121] In step 460, the computing device determines the compression rate of the compression strategy used when compressing the data.

[0122] The compression rate is a name describing the effect of compressed files. The compression rate can be the ratio of the size of the compressed data to the size of the data before compression. The compression rate of the compression strategy used when compressing the data determined by the computing device can be the ratio of the data amount of the compressed data to the data amount of the data to be compressed. Generally, the smaller the compression rate is, the smaller the data amount of the compressed data is. For example, the size of the data before compression is 100 megabytes (MB), and the size of the compressed data is 90 MB, so the compression rate is 90 / 100*100% = 90%.

[0123] In step 470, the computing device adjusts the parameters of the compression characteristic value and the compression strategy according to the compression rate.

[0124] If the compression ratio of the compression strategy used to compress the data to be compressed is still low, the computing device can also adjust the parameters of the compression strategy used to compress the data to be compressed according to the compression ratio, so as to more accurately set the compression strategy according to the real-time characteristics of the scene, and make the compression strategy used to compress the data to be compressed optimal. For example, if the processor occupancy is less than the threshold value, the compression ratio of the compression strategy is low, and the computing device can increase the size of the dictionary area. If the processor occupancy is greater than or equal to the threshold value, the network bandwidth between the computing device and the storage device is large, and the compression ratio is high, the computing device can reduce the size of the dictionary area.

[0125] The computing device can adjust the entire compressed feature value in the corresponding relationship, that is, adjust each element in the compressed feature vector. The computing device can also adjust part of the elements in the compressed feature vector. The computing device can also retain the history vector of the same element to form a chain structure, thereby improving the reliability of the update of the compression strategy. For example, the computing device takes the average of the historical value and the current value of the element (such as the processor occupancy, the network bandwidth between the computing device and the storage device, the bandwidth of the storage device, the available storage capacity of the storage device, the data type, the data block size, or the distribution of characters contained in the data to be compressed) in the compressed feature vector, and determines the average value as the updated value.

[0126] The computing device compresses the data to be compressed to obtain compressed data, and stores the compressed data into the storage device. For example, the method further includes step 480.

[0127] Step 480, the computing device sends the compressed data and the compression window to the storage device.

[0128] The computing device can send a data packet to the storage device (such as the hard disk 105), and the data packet contains the compressed data and the compression window used to compress the data to be compressed. As shown in Figure 8 , the data packet can include a packet header 810 and a payload 820. The packet header 810 includes a basic header, that is, the address required to transmit the data packet. The payload 820 includes an extended header and data to be transmitted. The computing device can incorporate the offset into the compression header. The compression header contains the triplets of the compressed data and the compression window. The compression header can be located in the payload 820 of the data packet. For example, the computing device selects the compression window as 6 characters according to the compression ratio of Figure 6 and Figure 7 , and the compressed data is <0, 0, a>, <1, 1, c>, <3, 4, b>, <3, 3, a>, and <1, 2, c>.

[0129] In some embodiments, the computing device can dynamically adjust the total length of the compression header occupying bits according to the length of the compressed data, i.e., the offset, the matching length and the respective bit number of the characters support dynamic adjustment, and the respective bit length of the triplets in the compression header is dynamically adjusted under different data types (e.g., special characters, Chinese characters and English characters, etc.). Compared with the compression header of fixed length, each byte in the compression header is more fully utilized, and the compression ratio of the data is increased.

[0130] After the computing device reads the compression header data from the storage device, the computing device can determine the size of the compression window used by the computing device during compression according to the compression window contained in the compression header, decode the triplets of the compressed data according to the compression window, and obtain the original data to be compressed.

[0131] It can be understood that, in order to implement the functions in the above embodiments, the computing device includes respective hardware structures and / or software modules for performing the functions. Those skilled in the art should easily realize that, in combination with the units and method steps of the examples described in the embodiments disclosed in the present application, the present application can be realized in the form of hardware or a combination of hardware and computer software. Whether a certain function is implemented in hardware or computer software driven hardware depends on the specific application scenario and design constraints of the technical solution.

[0132] The data compression method provided by the present embodiment is described in detail above. Figures 1 to 8 The data compression apparatus provided by the present embodiment is described below. Figure 9

[0133] Figure 9 The structure of the data compression apparatus provided by the present embodiment is shown in the following figure. The data compression apparatus can be used to implement the functions of the computing device in the above method embodiments, and thus can also achieve the beneficial effects of the above method embodiments. In the present embodiment, the data compression apparatus can be a computing node 110 or a storage node 100 as shown in Figures 1 to 3 , or a module (such as a chip) applied to a server.

[0134] As shown in Figure 9 , the data compression apparatus 900 includes a communication module 910, a preprocessing module 920, a policy matching module 930, a compression module 940 and a storage module 950. The data compression apparatus 900 is used to implement the functions of the computing device in the method embodiments shown in the above Figure 4 .

[0135] The communication module 910 is used to obtain the data to be compressed, and transmit the compressed data and the compression window to the hard disk 105. For example, the communication module 910 is used to perform steps 410 and 480 in the above Figure 4 . ​

[0136] The preprocessing module 920 is configured to acquire a parameter affecting a compression result of the to-be-compressed data, and determine a compression characteristic value of the to-be-compressed data according to the parameter affecting the compression result of the to-be-compressed data. For example, the preprocessing module 920 is configured to perform the following step. Figure 4 The step 420 and the step 430.

[0137] The policy matching module 930 is configured to determine a compression policy for compressing the to-be-compressed data according to the compression characteristic value. For example, the policy matching module 930 is configured to perform the following step. Figure 4 The step 440.

[0138] The parameter includes a parameter of a hardware resource used when the to-be-compressed data is compressed and / or a parameter describing a data characteristic of the to-be-compressed data. The parameter of the data characteristic includes at least one of a data type, a data block size, and a distribution of characters contained in the to-be-compressed data. The parameter of the hardware resource includes at least one of an occupancy rate of a processor of the computing device, a network bandwidth between the computing device and a storage device when the compressed data is stored to the storage device, and an available storage capacity of the storage device.

[0139] The compression module 940 is configured to compress the to-be-compressed data according to the compression policy to obtain compressed data. For example, the compression module 940 is configured to perform the following step. Figure 4 The step 450.

[0140] The policy matching module 930 is specifically configured to determine a compression characteristic value corresponding to the compression characteristic value of the to-be-compressed data in the corresponding relationship; and determine, according to the compression characteristic value determined by the corresponding relationship, that the compression policy corresponding to the compression characteristic value is the compression policy for compressing the to-be-compressed data.

[0141] The compression module 940 is specifically configured to compress the to-be-compressed data according to the plurality of compression windows respectively to obtain a plurality of compressed data; compare compression rates of the plurality of compressed data, and select the compressed data with the highest compression rate as the compressed data. The compression policy includes the plurality of compression windows.

[0142] The storage module 950 is configured to store a corresponding relationship between a plurality of compression characteristic values and compression policies.

[0143] The data compression apparatus 900 further includes an updating module 960. The updating module 960 is configured to determine a compression rate of the compression policy used when the data is compressed; adjust parameters of the compression characteristic value and the compression policy in the corresponding relationship according to the compression rate to obtain adjusted parameters of the compression characteristic value and the compression policy, and store the adjusted parameters of the compression characteristic value and the compression policy into the storage module 950. For example, the updating module 960 is configured to perform the following step. Figure 4The step 460 and the step 470.

[0144] It should be understood that the data compression apparatus 900 of the embodiments of the present application can be implemented by an ASIC, or a programmable logic device (PLD), which can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof. It can also be implemented by software Figure 4 When the data compression method is implemented, the data compression apparatus 900 and each module thereof can also be a software module.

[0145] The data compression apparatus 900 according to the embodiments of the present application can correspond to the method described in the embodiments of the present application, and the above and other operations and / or functions of each unit in the data compression apparatus 900 are respectively implemented to realize Figure 4 The corresponding flow of each method in the above description is not repeated here for brevity.

[0146] Figure 10 A structural schematic diagram of a computing device 1000 is provided in the embodiments. As shown in the figure, the computing device 1000 includes a processor 1010, a bus 1020, a memory 1030, a memory unit 1040, and a communication interface 1050.

[0147] It should be understood that in the embodiments, the processor 1010 can be a CPU, and the processor 1010 can also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), FPGAs or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0148] The processor can also be a graphics processing unit (GPU), a neural network processing unit (NPU), a microprocessor, an ASIC, or one or more integrated circuits for controlling the program execution of the solutions of the present application.

[0149] The communication interface 1050 is configured to enable the communication between the computing device 1000 and external devices or apparatuses. In the present embodiment, the communication interface 1050 is configured to receive the data to be compressed and transmit the compressed data.

[0150] The bus 1020 can include a path for transmitting information between the above-mentioned components (e.g., the processor 1010, the memory unit 1040, and the storage 1030). The bus 1020 can include, in addition to the data bus, a power supply bus, a control bus, and a status signal bus, etc. However, for the purpose of clarity, all the buses are marked as the bus 1020 in the figure. The bus 1020 can be a Peripheral Component Interconnect Express (PCIe) bus, or an extended industry standard architecture (EISA) bus, a unified bus (Ubus or UB), a compute express link (CXL), a cache coherent interconnect for accelerators (CCIX), etc.

[0151] As an example, the computing device 1000 can include multiple processors. The processor can be a multi-CPU processor. The processor herein can refer to one or more devices, circuits, and / or computing units for processing data (e.g., computer program instructions). The processor 1010 can invoke the correspondence between the plurality of compressed feature values and the compression policy stored in the storage 1030, determine the compression policy for compressing the data to be compressed according to the compressed feature values, and compress the data to be compressed according to the compression policy to obtain the compressed data.

[0152] It is worth noting that, Figure 10 In the present embodiment, only the computing device 1000 includes one processor 1010 and one storage 1030 as an example. Here, the processor 1010 and the storage 1030 are respectively configured to indicate a type of device or apparatus. In specific embodiments, the number of each type of device or apparatus can be determined according to business needs.

[0153] The memory unit 1040 can correspond to the storage medium used for storing information such as the correspondence between the plurality of compressed feature values and the compression strategy in the above method embodiments. The memory unit 1040 can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically EPROM (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM) used as an external cache. By way of example but not limitation, many forms of RAM can be used, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous dynamic RAM (SDRAM), double data rate synchronous dynamic RAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchlink DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0154] The memory 1030 is used for storing data and can be a disk, such as a mechanical hard disk or a solid-state disk.

[0155] The above computing device 1000 can be a general-purpose device or a special-purpose device. For example, the computing device 1000 can be a mobile phone terminal, a tablet computer, a notebook computer, a VR device, an AR device, an MR device, or an ER device, a vehicle-mounted terminal, etc., and can also be an edge device (for example, a box with a processing-capable chip) and the like. Alternatively, the computing device 1000 can also be a server or other computing-capable device.

[0156] It should be understood that the computing device 1000 according to the present embodiment can correspond to the data compression apparatus 900 in the present embodiment, and can correspond to the corresponding subject performing the respective processes in any of the methods according to the present embodiment, and the above and other operations and / or functions of each module in the data compression apparatus 900 are respectively for realizing the respective processes of each method in the present embodiment, and for brevity, will not be repeated here. Figure 4 Figure 4 It should be understood that the computing device 1000 according to the present embodiment can correspond to the data compression apparatus 900 in the present embodiment, and can correspond to the corresponding subject performing the respective processes in any of the methods according to the present embodiment, and the above and other operations and / or functions of each module in the data compression apparatus 900 are respectively for realizing the respective processes of each method in the present embodiment, and for brevity, will not be repeated here.

[0157] ​The method steps in the embodiments can be implemented by hardware, or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, which can be stored in a random access memory (RAM), a flash memory, a read-only memory (ROM), a programmable ROM (PROM), an erasable PROM (EPROM), an electrically EPROM (EEPROM), a register, a hard disk, a mobile hard disk, a CD-ROM, or any other form of storage medium well known in the art. An exemplary storage medium is coupled to a processor, so that the processor can read information from, and write information to, the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC. In addition, the ASIC can be located in a network device or a terminal device. Of course, the processor and the storage medium can also exist as discrete components in a network device or a terminal device.

[0158] In the above embodiments, the implementation can be wholly or partially achieved by software, hardware, firmware, or any combination thereof. When implemented by software, the implementation can be wholly or partially in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When loaded and executed by a computer, the computer programs or instructions perform the processes or functions described in the embodiments of the present application. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user equipment, or other programmable apparatus. The computer programs or instructions can be stored in a computer-readable storage medium, or transferred from one computer-readable storage medium to another, for example, the computer programs or instructions can be transferred from one website, computer, server, or data center to another via wired or wireless manner. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that integrates one or more available media. The available media can be a magnetic medium, such as a floppy disk, a hard disk, a magnetic tape; an optical medium, such as a digital video disc (DVD); or a semiconductor medium, such as a solid state disk (SSD).

[0159] The above merely provides the specific implementation of the present application, but the protection scope of the present application is not limited to this. Any person skilled in the art can easily think of various equivalent modifications or replacements within the technical range disclosed by the present application, and these modifications or replacements should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A data compression method, characterized in that: The method comprises: Determining a compression characteristic value of the data to be compressed according to a preset parameter, wherein the preset parameter is a parameter that affects a compression result of the data to be compressed; Determining a compression strategy corresponding to the compression feature value from a plurality of compression strategies; the compression strategy includes a plurality of compression windows; the compression window refers to a range for searching for an identical character string in the data to be compressed; Compressing the data to be compressed according to the multiple compression windows respectively to obtain multiple compressed data; The compression rates of the plurality of compressed data are compared, compressed data with the highest compression rate is selected as compressed data, and the compressed compressed data is stored.

2. The method according to claim 1, characterized in that The preset parameters include parameters of hardware resources used when compressing the data to be compressed and / or parameters describing data characteristics of the data to be compressed.

3. The method according to claim 2, characterized in that The parameters of the data features include at least one of a data type, a data block size, and a distribution of characters included in the data to be compressed.

4. The method according to claim 2, characterized in that The parameters of the hardware resources include at least one of a processor occupancy rate of the computing device, a network bandwidth between the computing device and the storage device when storing the compressed data in the storage device, and an available storage capacity of the storage device.

5. The method according to any one of claims 1 to 4, characterized in that The computing device stores a correspondence between a plurality of compression feature values ​​and compression strategies; Determining a compression strategy corresponding to the compression characteristic value from a plurality of compression strategies includes: Determine the compression characteristic value in the corresponding relationship corresponding to the compression characteristic value of the data to be compressed; According to the compression characteristic value determined by the corresponding relationship, a compression strategy corresponding to the compression characteristic value is determined as the compression strategy for compressing the data to be compressed.

6. The method according to claim 5, characterized in that The correspondence between the plurality of compression feature values ​​and the compression strategies is obtained based on neural network training.

7. The method according to any one of claims 1 to 4 or 6, characterized in that After obtaining the compressed data, the method further includes: Determining a compression ratio of the compression strategy used when compressing data; The compression characteristic value and the parameters of the compression strategy in the corresponding relationship are adjusted according to the compression rate.

8. A data compression device, characterized in that: The device comprises: A preprocessing module, configured to determine compression characteristic values ​​of the data to be compressed according to preset parameters, wherein the preset parameters are parameters that may affect the compression result of the data to be compressed; A strategy matching module is used to determine a compression strategy corresponding to the compression feature value from a plurality of compression strategies; the compression strategy includes a plurality of compression windows; the compression window refers to a range for searching for the same character string in the data to be compressed; A compression module is used to compress the data to be compressed according to the multiple compression windows to obtain multiple compressed data; and compare the compression rates of the multiple compressed data, select the compressed data with the highest compression rate as the compressed data, and store the compressed data.

9. The device according to claim 8, characterized in that The preset parameters include parameters of hardware resources used when compressing the data to be compressed and / or parameters describing data characteristics of the data to be compressed.

10. The device according to claim 9, characterized in that The parameters of the data features include at least one of a data type, a data block size, and a distribution of characters included in the data to be compressed.

11. The device according to claim 9, characterized in that The parameters of the hardware resources include at least one of a processor occupancy rate of the computing device, a network bandwidth between the computing device and the storage device when storing the compressed data in the storage device, and an available storage capacity of the storage device.

12. The device according to any one of claims 8 to 11, characterized in that The computing device stores a correspondence between a plurality of compression feature values ​​and compression strategies; When the strategy matching module determines the compression strategy for compressing the data to be compressed according to the compression feature value, it is specifically used to: Determine the compression characteristic value in the corresponding relationship corresponding to the compression characteristic value of the data to be compressed; According to the compression characteristic value determined by the corresponding relationship, a compression strategy corresponding to the compression characteristic value is determined as the compression strategy for compressing the data to be compressed.

13. The device according to claim 12, characterized in that The correspondence between the plurality of compression feature values ​​and the compression strategies is obtained based on neural network training.

14. The device according to any one of claims 8 to 11 or 13, characterized in that After obtaining the compressed data, the device further includes an updating module; The updating module is used to determine the compression ratio of the compression strategy used when compressing data; and to adjust the compression characteristic value and the parameters of the compression strategy in the corresponding relationship according to the compression ratio.

15. A computing device, characterized in that The computing device includes a memory and at least one processor, wherein the memory is used to store a set of computer instructions; when the processor executes the set of computer instructions, the operation steps of any one of the methods described in claims 1-7 are performed.

16. A storage system, characterized in that: The storage system includes at least one computing node and at least one storage node, the computing node is used to perform the operation steps of the method according to any one of claims 1 to 7 to obtain compressed data, and the storage node is used to store the compressed data; Alternatively, the computing node transmits the data to be compressed to the storage node, and the storage node executes the operation steps of the method according to any one of claims 1 to 7 to obtain compressed data and stores the compressed data.

Images