Data compression method and device, computing equipment and storage system

Abstract

The invention discloses a data compression method and device, computing equipment and a storage system, and relates to the field of data compression. The method comprises the following steps: when a computing device compresses to-be-compressed data, determining a compression feature value of the to-be-compressed data according to a parameter influencing a compression result of the to-be-compressed data, determining a compression strategy for compressing the to-be-compressed data according to the compression feature value, and compressing the to-be-compressed data according to the compression strategy to obtain compressed data, and storing the compressed data. Therefore, compared with the calculation equipment which adopts a compression algorithm to compress the data with different characteristics, the compression strategy used by the calculation equipment in the data compression method provided by the invention is selected according to the characteristics influencing the compression result of the to-be-compressed data; the to-be-compressed data is compressed by using the compression strategy matched with the characteristics of the to-be-compressed data, so that the compression rate of the data can be effectively improved.

Description

[0001] This application is a divisional application of the Chinese application with application date of September 22, 2021 and application number 202111109332.X. This application and the Chinese application with application number 202111109332.X both claim the priority of the Chinese patent application with application number 202110773759.3 and application name “A method, device and system for data processing” filed on July 8, 2021, all contents of which are incorporated into this application by reference. Technical Field

[0002] The present application relates to the field of data compression, and in particular to a data compression method, apparatus, computing device and storage system. Background Art

[0003] With the prosperity of cloud computing and big data industries, the larger the data center, the more data it stores. Larger data volumes and longer storage cycles mean higher storage costs. Currently, computing devices can use compression algorithms (such as LZ4, LZO, or Snappy, etc.) to compress data to be stored before storing it, thereby reducing the amount of data to be stored and saving corresponding hard disks, nodes, racks, sites, and operation and maintenance expenses. In computing devices, when the user sets the compression algorithm, all data is compressed based on the set compression algorithm, resulting in a relatively low data compression rate. Summary of the invention

[0004] The present application provides a data compression method, apparatus, computing device and storage system, thereby improving the compression rate of data.

[0005] In a first aspect, a data compression method is provided, which can be executed by a computing device and specifically includes the following steps: when the computing device compresses data to be compressed, the computing device determines a compression characteristic value of the data to be compressed according to a parameter that affects the compression result of the data to be compressed, determines a compression strategy for compressing the data to be compressed according to the compression characteristic value, compresses the data to be compressed according to the compression strategy to obtain compressed data, and stores the compressed data.

[0006] In this way, compared with a computing device using a compression algorithm to compress data with different characteristics, the compression strategy used by the computing device in the data compression method provided in the present application is selected based on the characteristics that affect the compression results of the data to be compressed. The data to be compressed is compressed using a compression strategy that matches the characteristics of the data to be compressed, which can effectively improve the data compression rate.

[0007] In a possible implementation, the parameters include parameters of hardware resources used when compressing the data to be compressed and / or parameters describing data characteristics of the data to be compressed. The parameters of the data characteristics include at least one of the data type, the data block size, and the distribution of characters included in the data to be compressed. The parameters of the hardware resources include at least one of the occupancy rate of the processor of the computing device, the network bandwidth between the computing device and the storage device when storing the compressed data in the storage device, and the available storage capacity of the storage device.

[0008] In another possible implementation, the computing device stores a plurality of correspondences between compression feature values ​​and compression strategies. Determining the compression strategy for compressing the data to be compressed according to the compression feature values ​​includes: the computing device determines the compression feature values ​​in the correspondence corresponding to the compression feature values ​​of the data to be compressed; and determining the compression strategy corresponding to the compression feature values ​​as the compression strategy for compressing the data to be compressed according to the compression feature values ​​determined by the correspondence. Thus, by presetting the correspondences between the plurality of compression feature values ​​and the compression strategies, the computing device can quickly and accurately select the compression strategy that matches the features of the data to be compressed when compressing the data to be compressed in real time.

[0009] The corresponding relationship between the plurality of compression feature values ​​and the compression strategy may be obtained based on neural network training.

[0010] In another possible implementation, the compression strategy includes multiple compression windows; compressing the data to be compressed according to the compression strategy includes: the computing device compresses the data to be compressed according to the multiple compression windows respectively to obtain multiple compressed data; compares the compression rates of the multiple compressed data, and selects the compressed data with the highest compression rate as the compressed data. It can be understood that the compression window refers to the range for searching the same character string in the data to be compressed. The same character string in the data to be compressed is searched through multiple different compression windows. The lengths of the searched same character strings may be different, so compressed data with different compression rates may be obtained. Therefore, the data to be compressed is compressed using a multi-level elastic compression window to obtain compressed data with the highest possible compression rate.

[0011] In another possible implementation, after obtaining the compressed data, the method further includes: the computing device determines the compression rate of the compression strategy used when compressing the data; and adjusts the compression feature value and the parameters of the compression strategy in the corresponding relationship according to the compression rate. Thus, by updating the compression strategy in the corresponding relationship, the accuracy of the corresponding relationship is improved, so that the computing device obtains a compression strategy that better matches the characteristics of the data to be compressed, thereby improving the compression rate of the data.

[0012] In a second aspect, a data compression device is provided, comprising modules for executing the data compression method in the first aspect or any possible design of the first aspect.

[0013] In a third aspect, a computing device is provided, which includes at least one processor and a memory, wherein the memory is used to store a set of computer instructions; when the processor executes the set of computer instructions as an execution device in the first aspect or any possible implementation of the first aspect, the operating steps of the data compression method in the first aspect or any possible implementation of the first aspect are executed.

[0014] In a fourth aspect, a storage system is provided, comprising at least one computing node and at least one storage node, wherein the computing node is used to execute the operating steps of the data compression method in the first aspect or any possible implementation of the first aspect to obtain compressed data, and the storage node is used to store the compressed data; or, the computing node transmits the data to be compressed to the storage node, and the storage node executes the operating steps of the data compression method in the first aspect or any possible implementation of the first aspect to obtain compressed data and stores the compressed data.

[0015] In a fifth aspect, a computer-readable storage medium is provided, comprising: computer software instructions; when the computer software instructions are executed in a computing device, the computing device executes the operating steps of the method described in the first aspect or any possible implementation of the first aspect.

[0016] In a sixth aspect, a computer program product is provided. When the computer program product is run on a computer, the computing device executes the operating steps of the method described in the first aspect or any possible implementation of the first aspect.

[0017] Based on the implementations provided in the above aspects, this application can also be further combined to provide more implementations. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is a schematic diagram of the architecture of a distributed storage system according to an embodiment of the present application;

[0019] Figure 2 This is a schematic diagram of the structure of the disk-controller separation according to an embodiment of the present application;

[0020] Figure 3 This is a schematic diagram of the architecture of a centralized storage system according to an embodiment of the present application;

[0021] Figure 4 A flowchart of a data compression method according to an embodiment of the present application;

[0022] Figure 5 A schematic diagram of determining a compression characteristic value of data to be compressed according to an embodiment of the present application;

[0023] Figure 6 A schematic diagram of a dictionary compression process according to an embodiment of the present application;

[0024] Figure 7 A schematic diagram of a dictionary compression process according to an embodiment of the present application;

[0025] Figure 8 A schematic diagram of the structure of a data packet according to an embodiment of the present application;

[0026] Fig. 9 This is a schematic diagram of the structure of a data compression device according to an embodiment of the present application;

[0027] Fig.10 A schematic diagram of the structure of a computing device according to an embodiment of the present application. DETAILED DESCRIPTION

[0028] Data compression refers to the method of reorganizing compressed data according to a compression algorithm to reduce redundant data in the data to be compressed, thereby reducing the amount of data to be compressed, reducing the storage space required to store the data to be compressed, and improving the compression rate, transmission speed and compression speed of the data to be compressed. The present application does not limit the specific form of the computing device for data compression. The computing device is, for example, a mobile phone terminal, a tablet computer, a laptop computer, a virtual reality (VR) device, an augmented reality (AR) device, a mixed reality (MR) device, an extended reality (ER) device or a vehicle-mounted terminal. The computing device can also be an edge device (for example, a box carrying a chip with processing power), a server, a device with computing power in a distributed storage system, or a device with computing power in a centralized storage system.

[0029] Figure 1 This is a schematic diagram of the architecture of a distributed storage system according to an embodiment of the present application. Figure 1 As shown, the distributed storage system provided in this embodiment includes a computing node cluster and a storage node cluster.

[0030] The computing node cluster includes one or more computing nodes 110, Figure 1 Three computing nodes 110 are shown in the figure, but are not limited to three computing nodes 110. The computing nodes 110 can communicate with each other through the network. The computing node 110 is a computing device, a node and hub of network data, responsible for providing information services to multiple clients or users in the network at the same time, such as a server, a computer, etc.

[0031] In hardware, such as Figure 1As shown, the computing node 110 includes at least a processor 112, a memory 113 and a network card 114. The processor 112 is a central processing unit (CPU) for processing data access requests from outside the computing node 110 or requests generated inside the computing node 110.

[0032] Exemplarily, when the processor 112 receives a write data request sent by a user, the data in the write data request is temporarily stored in the memory 113. When the total amount of data in the memory 113 reaches a certain threshold, the processor 112 sends the data stored in the memory 113 to the storage node 100 for persistent storage. In addition, the processor 112 is also used to calculate or process the data, such as metadata management, deduplication, data compression, virtualized storage space, and address conversion.

[0033] Figure 1 Only one CPU 112 is shown in the figure. In actual applications, there may be multiple CPUs 112 in a computing node 110, wherein a CPU 112 may have one or more CPU cores. This embodiment does not limit the number of CPUs and the number of CPU cores.

[0034] The memory 113 refers to an internal memory that directly exchanges data with the processor. It can read and write data at any time and at a high speed, and serves as a temporary data storage for the operating system or other running programs. The memory includes at least two types of memories. For example, the memory can be either a random access memory or a read-only memory (ROM). For example, the random access memory is a dynamic random access memory (DRAM) or a storage class memory (SCM).

[0035] DRAM is a semiconductor memory, and like most random access memories (RAM), it is a volatile memory device. SCM is a composite storage technology that combines the characteristics of traditional storage devices and memory. Storage-class memory can provide faster read and write speeds than hard disks, but the access speed is slower than DRAM, and the cost is also cheaper than DRAM. However, DRAM and SCM are only exemplary in this embodiment, and the memory can also include other random access memories, such as static random access memory (SRAM), etc. As for read-only memory, for example, it can be programmable read only memory (PROM), erasable programmable read only memory (EPROM), etc.

[0036] In addition, the memory 113 may also be a dual in-line memory module or a dual-line memory module (DIMM), that is, a module composed of a dynamic random access memory (DRAM), or a solid state disk (SSD).

[0037] In actual applications, multiple memories 113 and different types of memories 113 may be configured in the computing node 110. This embodiment does not limit the number and type of the memory 113. In addition, the memory 113 may be configured to have a power-saving function. The power-saving function means that when the system loses power and then powers on again, the data stored in the memory 113 will not be lost. The memory with the power-saving function is called a non-volatile memory.

[0038] The network card 114 is used to communicate with the storage node 100. For example, when the total amount of data in the memory 113 reaches a certain threshold, the computing node 110 can send a request to the storage node 100 through the network card 114 to store the data persistently. In addition, the computing node 110 can also include a bus for communication between the components inside the computing node 110. Figure 1The main function of the computing node 110 in the storage node is computing services. When storing data, remote storage can be used to achieve persistent storage. Therefore, it has less local storage than a conventional server, thereby saving cost and space. However, this does not mean that the computing node 110 cannot have local storage. In actual implementation, the computing node 110 can also have a small number of hard disks built in or externally connected. Any computing node 110 can access any storage node 100 in the storage node cluster through the network.

[0039] The storage node cluster may include multiple storage nodes 100, which are used to persistently store data transmitted by computing nodes. Figure 1 Three storage nodes 100 are shown in FIG. 1 , but the number is not limited to three storage nodes 100. For example, there may be more than three storage nodes or less than three storage nodes.

[0040] A storage node 100 includes one or more control units 101, a network card 104 and multiple hard disks 105. The network card 104 is used to communicate with the computing node 110. The hard disk 105 is used to store data and can be a disk or other types of storage media, such as a solid state drive or a shingled magnetic recording hard disk. The control unit 101 is used to write data to the hard disk 105 or read data from the hard disk 105 according to the read / write data request sent by the computing node 110. In the process of reading and writing data, the control unit 101 needs to convert the address carried in the read / write data request into an address that can be recognized by the hard disk. It can be seen that the control unit 101 also has some simple computing functions.

[0041] In practical applications, the control unit 101 may have various forms. In one case, the control unit 101 includes a CPU and a memory. The CPU is used to perform operations such as address conversion and reading and writing data. The memory is used to temporarily store data to be written to the hard disk 105, or data read from the hard disk 105 to be sent to the computing node 110. In another case, the number of control units 101 may be one, or two or more. When the storage node 100 includes at least two control units 101, there may be an ownership relationship between the hard disk 105 and the control unit 101. When there is an ownership relationship between the hard disk 105 and the control unit 101, each controller can only access the hard disk belonging to it, so this often involves forwarding read / write data requests between the control units 101, resulting in a longer data access path. In addition, if the storage space is insufficient, when adding a new hard disk 105 in the storage node 100, it is necessary to rebind the ownership relationship between the hard disk 105 and the control unit 101, which is complicated and leads to poor scalability of the storage space.

[0042] Therefore, in another embodiment, the functions of the control unit 101 can be offloaded to the network card 104. In other words, Figure 1 In the illustrated implementation, the storage node 100 does not have a control unit 101 inside, but the network card 104 performs data reading and writing, address conversion, and other computing functions. In this case, the network card 104 is an intelligent network card. It may include a CPU and a memory. The CPU is used to perform operations such as address conversion and reading and writing data. The memory is used to temporarily store data to be written to the hard disk 105, or data read from the hard disk 105 to be sent to the computing node 110. There is no ownership relationship between the network card 104 and the hard disk 105 in the storage node 100, and the network card 104 can access any hard disk 105 in the storage node 100, so it is more convenient to expand the hard disk when the storage space is insufficient.

[0043] Figure 1 The structure of the storage node shown in the figure belongs to the disk-controller integrated architecture, that is, the control unit and the hard disk 105 are included in the same physical device, but in some embodiments, the structure of the storage node can also be a disk-controller separated structure, such as Figure 2 As shown, control and storage are two independent devices, that is, the control device 200 and the hard disk frame 300 are connected to each other. The control device 200 includes multiple control units, such as control unit 0 and control unit 1, and multiple control units are connected to the computing node 110 through the network card 104. The structure of each control unit is the same. Taking the control unit 0 as an example, it includes a front-end interface 201, a CPU 202, a memory 203, and a back-end interface 204, wherein the front-end interface 201 is connected to the network card 104 to transmit the data sent by the computing node through the network card to the control unit 0, the back-end interface 204 is connected to the hard disk frame, and the data processed by the control unit 0 is transmitted to the hard disk frame 300 for storage, and the CPU 202 and the memory 203 are used to process the data transmitted by the computing node 110 or the data of the locally running program. For details, please refer to the description of the CPU 112 and the memory 113 in the computing node 110, which will not be repeated here. As an independent device, the hard disk frame 300 includes a control unit 301 in addition to the hard disk 105, which is used to process the data transmitted by the computing node and store it in the hard disk 105, and manage the hard disk 105.

[0044] Figure 1 The diagram shows the architecture of a distributed storage system. The solution of the embodiment of the present application can also be applied to a centralized storage system. In a centralized storage system, the data of a computing node is stored in a storage node, which can be a storage array, for example. The architecture of a centralized storage system is shown in FIG. Figure 3 As shown in FIG. 1 , the storage node 100 and the computing node 110 have the same structure as Figure 1 The same as in , I will not repeat them here. Figure 3The storage node 100 shown in the figure is a disk-controller integrated structure. In an integrated storage system, the structure of the storage node can also be Figure 2 The disk-control separation structure shown.

[0045] In the data compression method provided by the embodiment of the present application, when the computing device compresses the data to be compressed, the computing device determines the compression characteristic value of the data to be compressed according to the parameters that affect the compression result of the data to be compressed, adaptively determines the compression strategy according to the compression characteristic value, and compresses the data to be compressed according to the compression strategy to obtain the compressed data. In this way, compared with the computing device using a compression algorithm to compress data with different characteristics, the compression strategy used by the computing device in the data compression method provided by the present application is selected based on the characteristics that affect the compression result of the data to be compressed, and the data to be compressed is compressed using the compression strategy that matches the characteristics of the data to be compressed, which can effectively improve the compression rate of the data, so that the storage device can store more data and improve the storage space utilization rate of the storage device.

[0046] Next, combine Figures 4 to 8 The data compression method provided in the embodiment of the present application is described in detail. Figure 4 A flow chart of a data compression method provided in an embodiment of the present application. Figure 4 As shown, the method includes the following steps.

[0047] Step 410: The computing device obtains data to be compressed.

[0048] The computing device can receive data to be compressed sent by an external device (such as a keyboard, mouse, camera, voice receiver, etc.); or the computing device runs an application to generate data to be compressed. If the computing device is a storage device, the computing device can also receive data to be compressed sent by other computing devices (such as a server) through the network. The data to be compressed can be text, numbers, letters, images, and voice, etc.

[0049] Step 420: The computing device collects parameters that affect the compression result of the data to be compressed.

[0050] The parameters that affect the compression result of the data to be compressed include parameters of the hardware resources used when compressing the data to be compressed and / or parameters that describe the data characteristics of the data to be compressed.

[0051] The computing device collects the parameters of the hardware resources used for compression in real time within a preset period (e.g., a few seconds or tens of milliseconds) before compressing the data to be compressed. The parameters of the hardware resources include at least one of the hardware resources that affect the data compression result, such as the occupancy rate of the processor of the computing device, the network bandwidth between the computing device and the storage device when storing the compressed data in the storage device, the bandwidth of the storage device, and the available storage capacity of the storage device.

[0052] The processor occupancy rate indicates the busy or idle state of the processor. If the processor occupancy rate is greater than or equal to the threshold, it means that the processor is in a busy state and the available computing resources are small. On the contrary, if the processor occupancy rate is less than the threshold, it means that the processor is in an idle state and the available computing resources are large.

[0053] Understandably, in order to search for as many identical strings as possible from the data to be compressed and improve the compression rate of the data to be compressed, the compression algorithm used by the computing device may be more complex. If the computing device compresses the data to be compressed using a compression algorithm with higher complexity, the computing device needs to occupy more computing resources. When the processor is in a busy state, if the computing device occupies more computing resources for data compression, the compression time may be longer due to the small number of available computing resources. When the processor is in an idle state, due to the large number of available computing resources, even if the computing device occupies more computing resources for data compression, the compression time is shorter relative to when the processor is in a busy state. Therefore, when the processor is in a busy state, the computing device can compress the data to be compressed using a compression algorithm with lower complexity, and shorten the compression time as much as possible while ensuring the compression rate of the data. When the processor is in an idle state, the computing device can compress the data to be compressed using a compression algorithm with higher complexity, and improve the compression rate of the data as much as possible while ensuring the compression time.

[0054] The computing device and the storage device can be two independent physical devices. Figure 1 The computing node 110 or storage node 100 in the distributed storage system shown. The computing device transmits compressed data to the storage device via a network. The network is, for example, a storage area network (SAN). The network bandwidth between the computing device and the storage device represents the amount of data transmitted between the computing device and the storage device per unit time (e.g., 1 second). The larger the network bandwidth between the computing device and the storage device, the more data is transmitted between the computing device and the storage device per unit time; the smaller the network bandwidth between the computing device and the storage device, the less data is transmitted between the computing device and the storage device per unit time.

[0055] It is understandable that when the network bandwidth between the computing device and the storage device is small, if the computing device uses a compression algorithm with a higher compression rate to compress the data, the transmission time of the compressed data is shorter because the amount of data after compression is smaller. When the network bandwidth between the computing device and the storage device is large, since the amount of data transmitted by the computing device and the storage device per unit time is large, even if the amount of data after compression is large, the transmission time may be shorter when the network bandwidth is relatively small. Therefore, when the network bandwidth between the computing device and the storage device is small, the computing device can use a compression algorithm with a higher compression rate to compress the data to be compressed, and maximize the data compression rate while ensuring the transmission time.

[0056] The bandwidth of a storage device indicates the amount of data accessed by the storage device in a unit of time (e.g., 1 second). Understandably, when the bandwidth of a storage device is small, if the computing device uses a compression algorithm with a high compression ratio to compress data, the amount of compressed data is small, and the time it takes for the storage device to access the compressed data is short. Therefore, when the bandwidth of a storage device is small, the computing device can use a compression algorithm with a high compression ratio to compress the data to be compressed, and maximize the data compression ratio while ensuring the access time.

[0057] The available storage capacity of a storage device refers to the available storage space remaining in the storage device for storing compressed data. It is understandable that the larger the remaining available storage capacity of a storage device is, the larger the amount of data that can be stored is. The smaller the remaining available storage capacity of a storage device is, the smaller the amount of data that can be stored is. When the remaining available storage capacity of a storage device is small, the computing device can compress the data to be compressed using a compression algorithm with a high compression rate, thereby reducing the amount of compressed data as much as possible and improving the utilization rate of the storage space of the storage device.

[0058] The computing device may analyze the data to be compressed to obtain parameters of data features. The parameters of data features include at least one of data type, data block size, and distribution of characters included in the data to be compressed.

[0059] Data types include text, numbers, letters, and images. It is understandable that the degree of repetition of characters in data of different data types may be different. For example, numbers, letters, and images may contain more repeated characters. The computing device may compress the data to be compressed using a more complex compression algorithm, search for as many identical character strings as possible, and improve the compression rate of the data.

[0060] The data block size refers to the unit data volume of the computing device for data compression. For example, the data block size ranges from 8 kilobytes (KB) to 256KB. The data block size can be preset by the system administrator. Understandably, the smaller the data block size, the fewer identical characters may be in the data block; the larger the data block size, the more identical characters may be in the data block. When the data block size is large, the computing device can use a more complex compression algorithm to compress the data to be compressed, search for as many identical character strings as possible, and improve the data compression rate.

[0061] The distribution of characters contained in the data to be compressed indicates the distribution pattern of the characters contained in the data to be compressed. For example, the data to be compressed contains two characters a and b, and the distribution pattern of a and b can be aaabbb or ababab or abbabb, etc. It is understandable that when the data to be compressed contains a large number of repeated characters, the computing device can use a more complex compression algorithm to compress the data to be compressed, search for as many identical character strings as possible, and improve the compression rate of the data.

[0062] In actual applications, the parameters to be used can be determined according to specific application scenarios. They can be only at least one of the hardware resource parameters, or only at least one of the parameters representing data characteristics, or a combination of hardware resource parameters and data characteristic parameters. The specific parameter values ​​can be system defaults or can be set by the user as needed.

[0063] Step 430: The computing device determines a compression characteristic value of the data to be compressed according to a parameter that affects a compression result of the data to be compressed.

[0064] Since the dimensional units of the various parameters collected by the computing device are often inconsistent, in order to comprehensively consider the impact of various parameters on the data compression result, the computing device can normalize the various parameters to obtain parameters with unified dimensions. In some embodiments, the computing device can perform structured processing on the parameters to obtain the compressed eigenvalue of the data to be compressed. The compressed eigenvalue can be a eigenvector. An element in the eigenvector represents a parameter.

[0065] For example, I A Represents the feature vector of the data to be compressed, I A =[a1,a2,a3,a4,a5,a6,a7], where element a1 represents the processor occupancy vector of the computing device, element a2 represents the network bandwidth vector between the computing device and the storage device, element a3 represents the bandwidth vector of the storage device, element a4 represents the available storage capacity vector of the storage device, element a5 represents the data type vector, element a6 represents the data block size vector, and element a7 represents the distribution vector of the characters contained in the data to be compressed.

[0066] Optionally, in order to improve the accuracy of matching the compression strategy with the compression feature value of the data to be compressed determined based on the parameters that affect the compression result of the data to be compressed, and to eliminate redundant parameters, the computing device may also merge parameters with common characteristics (such as the same dimensional unit). For example, the computing device may merge the network bandwidth between the computing device and the storage device and the bandwidth of the storage device.

[0067] The computing device stores a plurality of correspondences between compression feature values ​​and compression strategies. The computing device can calculate the similarity between the compression feature value of the data to be compressed and each compression feature value in the correspondence, and determine the compression feature value with the highest similarity in the correspondence as the compression feature value matching the compression feature value of the data to be compressed.

[0068] In some embodiments, the computing device calculates the Hamming distance between the compressed eigenvalue of the data to be compressed and each compressed eigenvalue in the corresponding relationship to obtain multiple Hamming distances. The shorter the Hamming distance, the more similar the compressed eigenvalue of the data to be compressed is to the compressed eigenvalue of the corresponding relationship. The computing device can compare multiple Hamming distances and select the compressed eigenvalue with the shortest Hamming distance as the compressed eigenvalue determined by the corresponding relationship. The Hamming distance between the compressed eigenvalue of the data to be compressed and the compressed eigenvalue of the corresponding relationship satisfies formula (1).

[0069]

[0070] Among them, I A Indicates the compression characteristic value of the data to be compressed, I B The compressed eigenvalue representing the corresponding relationship, a i Represents the element in the compressed eigenvalue of the data to be compressed, b i represents the element in the compressed eigenvalue of the corresponding relationship, n represents the number of elements, D ham Indicates the Hamming distance between the compressed eigenvalue of the data to be compressed and the compressed eigenvalue of the corresponding relationship.

[0071] In other embodiments, the computing device may also calculate the Euclidean distance between the compressed feature value of the data to be compressed and each compressed feature value in the corresponding relationship to obtain multiple Euclidean distances. The computing device may compare the multiple Euclidean distances and select the compressed feature value with the shortest Euclidean distance as the compressed feature value determined by the corresponding relationship.

[0072] In other embodiments, the computing device may select a compressed feature value determined by the corresponding relationship from a plurality of Hamming distances or a plurality of Euclidean distances according to a preset distance range. If there are compressed feature values ​​of the corresponding relationship of multiple Hamming distances or Euclidean distances within the preset range, the computing device may select the compressed feature value determined by the corresponding relationship according to the nearest neighbor distance ratio, thereby improving the accuracy of the compressed feature value in the matching corresponding relationship. The nearest neighbor distance ratio satisfies formula (2).

[0073]

[0074] Among them, β represents the nearest neighbor distance ratio threshold, and the value range of β is 0-1. Figure 5 As shown in (a) in FIG. 1 , point B1 and point B2 represent two different compression feature values ​​in the corresponding relationship. Point O represents the compression feature value of the data to be compressed. Within the preset distance range, the Hamming distance or Euclidean distance of point O is similar to that of point B1 and point B2. Figure 5 As shown in (b), it means that the computing device determines the compression characteristic value represented by point B1 according to formula (2) as the compression characteristic value determined by the corresponding relationship.

[0075] In other embodiments, the computing device may also calculate the normalized mutual correlation value between the compression eigenvalue of the data to be compressed and each compression eigenvalue in the corresponding relationship to obtain a plurality of normalized mutual correlation values. The closer the normalized mutual correlation value is to 1, the more similar the compression eigenvalue of the data to be compressed is to the compression eigenvalue of the corresponding relationship. The closer the normalized mutual correlation value is to 0, the less similar the compression eigenvalue of the data to be compressed is to the compression eigenvalue of the corresponding relationship. The computing device determines the compression eigenvalue whose normalized mutual correlation value is closest to 1 as the compression eigenvalue determined by the corresponding relationship. The normalized mutual correlation value between the compression eigenvalue of the data to be compressed and the compression eigenvalue of the corresponding relationship satisfies formula (3).

[0076]

[0077] Among them, NCC (I A ,I B ) represents the normalized cross-correlation value between the compressed eigenvalue of the data to be compressed and the compressed eigenvalue of the corresponding relationship, S a Represents the variance of the compressed eigenvalues ​​of the data to be compressed. S b The variance of the compressed eigenvalue representing the corresponding relationship. a Represents the average value of the compression characteristic value of the data to be compressed. b The average value of the compressed eigenvalues ​​representing the corresponding relationship.

[0078] Step 440: The computing device determines a compression strategy for compressing the data to be compressed according to the compression characteristic value.

[0079] The computing device determines the compression strategy corresponding to the compression characteristic value determined according to the corresponding relationship as the compression strategy for compressing the data to be compressed.

[0080] It is understandable that the correspondence between multiple compression feature values ​​and compression strategies is a pre-set template library, that is, the template library contains multiple entries, each entry contains a compression feature value and a compression strategy corresponding to the compression feature value. In one example, the compression strategy indicates the compression algorithm corresponding to the compression feature value. Different compression feature values ​​in the template library correspond to different compression algorithms. Compression algorithms include but are not limited to: Shannon coding, Huffman coding, dictionary compression algorithms (such as: LZ77, LZSS, LZ78 and LZW). In another example, the compression strategy indicates the parameters of the compression algorithm corresponding to the compression feature value, such as the size of the dictionary area in the dictionary compression algorithm. Different compression feature values ​​in the template library correspond to different parameters of the same compression algorithm.

[0081] The compression strategy corresponding to the compression feature value in the corresponding relationship is set based on the influence of the hardware resource parameters and data feature parameters on the compression rate described in step 420, that is, the compression algorithm or compression algorithm parameters indicated by the compression strategy corresponding to the compression feature value in the corresponding relationship is set based on the principle of the highest possible compression rate.

[0082] For example, the compression characteristic value is the processor occupancy rate. If the processor occupancy rate is less than a threshold, the compression strategy corresponding to the processor occupancy rate indicates a dictionary compression algorithm for a longer dictionary area. If the processor occupancy rate is greater than or equal to the threshold, the compression strategy corresponding to the processor occupancy rate indicates a dictionary compression algorithm for a shorter dictionary area.

[0083] For another example, the compression characteristic value is the remaining available storage capacity of the storage device. If the remaining available storage capacity of the storage device is small, then the compression strategy corresponding to the remaining available storage capacity of the storage device indicates a dictionary compression algorithm for a longer dictionary area; if the remaining available storage capacity of the storage device is large, then the compression strategy corresponding to the remaining available storage capacity of the storage device indicates a dictionary compression algorithm for a shorter dictionary area.

[0084] For example, the compression characteristic value is the network bandwidth between the computing device and the storage device or the bandwidth of the storage device. If the network bandwidth or the bandwidth of the storage device is small, the compression strategy corresponding to the remaining available storage capacity of the storage device indicates a dictionary compression algorithm for a longer dictionary area.

[0085] For another example, the compression characteristic value is the data block size. If the data block size is relatively large, the compression strategy corresponding to the remaining available storage capacity of the storage device indicates a dictionary compression algorithm for a relatively long dictionary area.

[0086] For another example, the compression characteristic value is the distribution of characters contained in the data to be compressed. If the data to be compressed contains more repeated characters, the compression strategy corresponding to the remaining available storage capacity of the storage device indicates a dictionary compression algorithm for a longer dictionary area.

[0087] In some embodiments, the compression strategy corresponding to the compression characteristic value may be set by comprehensively considering the hardware resource parameters and the data characteristic parameters, thereby improving the compression rate of the data to be compressed.

[0088] For example, the compression characteristic value includes the processor occupancy rate and the network bandwidth between the computing device and the storage device. Assuming that the processor occupancy rate is greater than or equal to the threshold, it means that the available computing resources of the processor are relatively small, and the network bandwidth between the computing device and the storage device is relatively large, which means that the amount of data transmitted by the computing device and the storage device per unit time is relatively large. Then, the compression strategy corresponding to the compression characteristic value indicates the dictionary compression algorithm of the shorter dictionary area. The computing device compresses the data to be compressed using the dictionary compression algorithm of the shorter dictionary area. The computing device can occupy fewer computing resources and shorten the compression time as much as possible while ensuring the compression rate of the data. Since the network bandwidth between the computing device and the storage device is relatively large, the computing device can also transmit the compressed data faster.

[0089] For another example, the compression characteristic value includes the processor occupancy rate and the available storage capacity of the storage device. Assuming that the processor occupancy rate is less than the threshold value and the available storage capacity of the storage device is less, the compression strategy corresponding to the compression characteristic value indicates the dictionary compression algorithm of the longer dictionary area. Since the processor has more available computing resources, the computing device can use the dictionary compression algorithm of the longer dictionary area to compress the data to be compressed, thereby reducing the amount of compressed data as much as possible, saving the storage space of the storage device, and improving the utilization rate of the storage space of the storage device.

[0090] For another example, the compression feature value includes the processor occupancy rate and the distribution of characters contained in the data to be compressed. Assuming that the processor occupancy rate is less than a threshold value and the data to be compressed contains a large number of repeated characters, the compression strategy corresponding to the compression feature value indicates a dictionary compression algorithm with a longer dictionary area. Since the processor has more available computing resources, the computing device can use the dictionary compression algorithm with a longer dictionary area to compress the data to be compressed, search for as many identical character strings as possible, reduce the amount of compressed data as much as possible, and improve the data compression rate.

[0091] For another example, the compression feature value includes the processor occupancy rate, the available storage capacity of the storage device, and the distribution of characters contained in the data to be compressed. Assuming that the processor occupancy rate is less than a threshold value, the available storage capacity of the storage device is small, and the data to be compressed contains a large number of repeated characters, the compression strategy corresponding to the compression feature value indicates a dictionary compression algorithm with a longer dictionary area. Since the processor has more available computing resources and the data to be compressed contains a large number of repeated characters, the computing device can use a dictionary compression algorithm with a longer dictionary area to compress the data to be compressed, thereby reducing the amount of compressed data as much as possible, saving the storage space of the storage device, and improving the utilization rate of the storage space of the storage device.

[0092] In practical applications, when setting a compression strategy corresponding to a compression characteristic value according to the compression characteristic value, you can focus on at least one of the hardware resource parameters, or you can focus on at least one of the data characteristic parameters, or you can comprehensively consider the hardware resource parameters and the data characteristic parameters.

[0093] Therefore, since the compression strategy is set based on the influence of the parameters of hardware resources and the parameters of data characteristics on the compression result of the data to be compressed, that is, the compression algorithm or the parameters of the compression algorithm indicated by the compression strategy corresponding to the compression characteristic value in the corresponding relationship is set according to the principle of the highest possible compression rate, the computing device selects the compression strategy corresponding to the compression characteristic value similar to the compression characteristic value of the data to be compressed from the corresponding relationship, and uses the compression strategy to compress the data to be compressed, which can effectively improve the compression rate of the data.

[0094] It should be noted that the corresponding relationship can be pre-set by the system administrator based on experience. Alternatively, the corresponding relationship can be obtained based on neural network training. For example, the computing device marks the compression strategy corresponding to the compression eigenvalue, inputs the compression eigenvalue and the marked compression strategy into the neural network, and trains the neural network until the loss function in the neural network converges, and the loss function value is less than a specific threshold value, then the neural network training is completed, so that the neural network has the function of predicting the compression strategy based on the compression eigenvalue to achieve prediction accuracy. The computing device inputs the compression eigenvalue of the data to be compressed into the trained neural network, and can obtain the compression strategy of the compression eigenvalue of the data to be compressed. It should be noted that the computing device for training the neural network and the computing device for determining the compression strategy for compressing the data to be compressed can be the same device or different devices, without limitation.

[0095] Step 450: The computing device compresses the data to be compressed according to the compression strategy to obtain compressed data.

[0096] The compression strategy may refer to a dictionary compression algorithm. The basic idea of ​​the dictionary compression algorithm is to replace the original data in the data to be compressed with position information to achieve compression, and to restore the data according to the position information during decompression. Specifically, the dictionary compression algorithm searches the character string in the area to be encoded in the dictionary area, obtains the longest character string that matches the area to be encoded, returns the distance between the starting character of the longest character string and the boundary between the dictionary area and the area to be encoded, and the matching length (literal length) of the matching character string. Then, the compression window is shifted to the right by the matching length + 1 character, and the search continues until the encoding of the data to be encoded is completed, so as to achieve the purpose of data compression. The output of the dictionary compression algorithm is a triple (offset, matching length, character). The offset represents the offset distance of the longest character string. The offset may refer to the distance of the longest character string from the left side of the dictionary area or the distance of the longest character string from the right side of the dictionary area (the boundary between the dictionary area and the area to be encoded). The matching length represents the length of the matching character string. The character represents the next character after the match.

[0097] The following example illustrates the specific process of dictionary compression. For example, assume that the data to be encoded is aacaacabcabaaac. The size of the dictionary area is 6 characters long, and the size of the area to be encoded is 4 characters long. Since the dictionary is dynamically generated, the dictionary is empty at the beginning of encoding, and the dictionary contains different characters during the encoding process.

[0098] like Figure 6 As shown in (a), the encoding starts, aligning the left end of the area to be encoded with the left end of the data to be encoded (the boundary between the dictionary area and the area to be encoded). The area to be encoded includes the character aaca, the dictionary is empty, and there is no matching character in the dictionary, so the offset is 0 and the matching length is 0. Since the same character is not matched, the character in the triple is the first character a of the data to be encoded. Therefore, the output triple is <0,0,a>.

[0099] like Figure 6 As shown in (b) in the figure, if the compression window is moved right by len (match length 0) + 1 characters, the dictionary area contains character a. The area to be encoded includes character acaa. The longest matching string in the dictionary area and the area to be encoded is a. If the matching string is a, the offset is 1, the matching length is 1, and the next character after the match is c. Therefore, the output triple is <1,1,c>.

[0100] like Figure 6As shown in (c), the compression window is moved right by len (matching length 1) + 1 characters, that is, the compression window is moved right by 2 characters, then the dictionary area contains the character aac. The area to be encoded includes the character aaca. The longest matching string in the dictionary area and the area to be encoded is aac, and the matching string is aaca, then the offset is 3, the matching length is 4, and the next character after the match is b. Therefore, the output triple is <3,4,b>.

[0101] like Figure 6 As shown in (d) in the figure, the compression window is moved right by len (matching length 4) + 1 characters, that is, the compression window is moved right by 5 characters, and the dictionary area contains the characters caacab. The area to be encoded includes the characters caba. The longest matching string in the dictionary area and the area to be encoded is cab. If the matching string is cab, the offset is 3, the matching length is 3, and the next character after the match is a. Therefore, the output triple is <3,3,a>.

[0102] like Figure 6 As shown in (e), the compression window is moved right by len (matching length 3) + 1 characters, that is, the compression window is moved right by 4 characters, and the dictionary area contains the characters abcaba. The area to be encoded includes the characters aac. The longest matching string in the dictionary area and the area to be encoded is a, and the matching string is aa, then the offset is 1, the matching length is 2, and the next character after the match is c. Therefore, the output triple is <1,2,c>.

[0103] like Figure 6 As shown in (f) in the figure, the compression window is moved right by len (matching length 2) + 1 characters, that is, the compression window is moved right by 3 characters, and the dictionary area contains the characters abaaac. The area to be encoded includes characters that are empty, and the encoding is completed.

[0104] Therefore, after the computing device compresses the encoded data aacaacabcabaaac using a dictionary compression algorithm, the resulting triple sequence includes <0,0,a>, <1,1,c>, <3,4,b>, <3,3,a>, and <1,2,c>.

[0105] As the length of the dictionary area varies, the dictionary area may also contain different characters, so during the dictionary compression process, the matching string searched will also be different. Figure 7 FIG. 1 is a schematic diagram of another dictionary compression process provided by an embodiment of the present application. The length of the dictionary area is changed from 6 characters to 2 characters. Figure 7 References to (a) and (b) Figure 6 Description of (a) and (b) in .

[0106] like Figure 7 As shown in (c), the compression window is moved right by len (match length 1) + 1 characters, that is, the compression window is moved right by 2 characters, then the dictionary area contains the character ac. The area to be encoded includes the character aaca. There is no matching character in the dictionary, and the longest matching string in the dictionary area and the area to be encoded is 0, then the offset is 0, the matching length is 0, and the next character after the match is a. Therefore, the output triple is <0,0,a>.

[0107] like Figure 7 As shown in (d) in the figure, the compression window is moved right by len (matching length 0) + 1 characters, that is, the compression window is moved right by 1 character, and the dictionary area contains the character ca. The area to be encoded includes the characters acba. The longest matching string in the dictionary area and the area to be encoded is a. If the matching string is a, the offset is 1, the matching length is 1, and the next character after the match is c. Therefore, the output triple is <1,1,c>.

[0108] like Figure 7 As shown in (e), the compression window is moved right by len (match length 1) + 1 characters, that is, the compression window is moved right by 2 characters, then the dictionary area contains the characters ac. The area to be encoded includes the characters abca. There is no matching character in the dictionary, and the longest matching string in the dictionary area and the area to be encoded is 0, then the offset is 0, the matching length is 0, and the next character after the match is a. Therefore, the output triple is <0,0,a>.

[0109] The character bcaba after aacacac has no matching character in the dictionary. The longest matching string in the dictionary area and the area to be encoded is 0, the offset is 0, the matching length is 0, and the output triple contains <0,0,b>,<0,0,c>,<0,0,a>,<0,0,b>,<0,0,a>.

[0110] like Figure 7 As shown in (f), the compression window is moved right by len (matching length 0) + 1 characters, that is, the compression window is moved right by 1 character, and the dictionary area contains the character ba. The area to be encoded includes the character aac. The longest matching string in the dictionary area and the area to be encoded is a, and the matching string is aa, then the offset is 1, the matching length is 2, and the next character after the match is c. Therefore, the output triple is <1,2,c>.

[0111] like Figure 7 As shown in (g), the compression window is moved right by len (matching length 2) + 1 characters, that is, the compression window is moved right by 3 characters, and the dictionary area contains characters ac. The area to be encoded includes characters that are empty, and the encoding is completed.

[0112] Therefore, after the computing device compresses the encoded data aacaacabcabaaac using the dictionary compression algorithm, the resulting triple sequence includes <0,0,a>, <1,1,c>, <0,0,a>, <1,1,c>, <0,0,a>, <0,0,b>, <0,0,c>, <0,0,a>, <0,0,b>, <0,0,a> and <1,2,c>.

[0113] Depend on Figure 6 and Figure 7 It can be seen that the compressed data obtained by encoding the length of the dictionary area as 6 characters contains 5 triples, and the compressed data obtained by encoding the length of the dictionary area as 2 characters contains 11 triples. The smaller the dictionary area, the lower the compression rate. Figure 6 and Figure 7 The compression rate is , and the compression window is selected as 6 characters. The compressed data are <0,0,a>, <1,1,c>, <3,4,b>, <3,3,a> and <1,2,c>.

[0114] Understandably, if the dictionary area is smaller and the dictionary contains fewer characters, the computing device will search for fewer matching characters in the dictionary area in the area to be encoded, and the compression rate will be lower. If the dictionary area is larger and the dictionary contains more characters, the computing device may search for more matching characters in the dictionary area in the area to be encoded, and the compression rate will be higher. However, the computing device needs to spend more time searching for repeated characters, resulting in slower compression speed, which in turn affects system performance.

[0115] In some embodiments, the compression strategy may indicate at least two compression algorithms, so that the computing device selects the compression algorithm with the highest compression rate from the at least two compression algorithms to compress the data to be compressed. Alternatively, the compression strategy may indicate the values ​​of multiple parameters in the compression algorithm, so that the computing device selects the values ​​of the parameters of the compression algorithm with the highest compression rate from the values ​​of multiple parameters to compress the data to be compressed.

[0116] For example, the compression strategy may also include multiple compression windows. The compression window refers to the dictionary area in the dictionary compression algorithm. The compression strategy includes multiple dictionary areas of different sizes. The computing device compresses the data to be compressed according to the multiple compression windows to obtain multiple compressed data; compares the compression rates of the multiple compressed data, and selects the compressed data with the highest compression rate as the compressed data. Therefore, after the computing device compresses the data to be compressed using multiple compression windows of different sizes, the compressed data with the highest compression rate is used as the compressed data, thereby further improving the compression rate of the data.

[0117] In other embodiments, the computing device may also divide the data to be compressed into multiple data blocks to be compressed according to the data block size, and compress each data block to be compressed according to the data compression method provided in the embodiment of the present application, thereby further improving the data compression rate by compressing the data to be compressed as a whole.

[0118] In other embodiments, the computing device may divide the data to be compressed according to the size of the data block to obtain the data blocks to be compressed, and after performing a hash operation on the data blocks to be compressed, the hash value is stored in the dictionary area. The computing device matches the hash value of the dictionary area according to the hash value of the data block to be compressed, and performs compression if the match is successful, and stores a new hash value in the dictionary area if the match is unsuccessful. Since different data blocks to be compressed can use different hash algorithms for hash operations, the probability of hash collision is effectively reduced compared to using one hash algorithm to hash the compressed data.

[0119] Optionally, the compression strategy may further indicate whether to re-compress the compressed data. For example, the compression strategy includes a re-compression flag. The re-compression flag has a value of 1, indicating re-compression. The re-compression flag has a value of 0, indicating no re-compression. The computing device may perform entropy coding on the compressed data to further improve the compression rate of the data.

[0120] Optionally, after obtaining the compressed data, the computing device may also adjust the compression feature value and the parameters of the compression strategy in the corresponding relationship to improve the accuracy of the corresponding relationship. For example, the method further includes step 460 and step 470.

[0121] Step 460: The computing device determines the compression ratio of the compression strategy used when compressing data.

[0122] Compression rate is a name that describes the effect of compressing files. The compression rate can be the ratio of the size of the compressed data to the size of the data before compression. The compression rate of the compression strategy used by the computing device when compressing data can be the ratio of the amount of compressed data to the amount of data to be compressed. Generally, the smaller the compression rate, the smaller the amount of compressed data. For example, the size of the data before compression is 100 megabytes (MB), and the size of the data before compression is 90MB, and the compression rate is 90 / 100*100%=90%.

[0123] Step 470: The computing device adjusts the compression characteristic value and the parameters of the compression strategy according to the compression ratio.

[0124] If the compression rate of the compression strategy used when compressing the data to be compressed is still low, the computing device can also adjust the parameters of the compression strategy used when compressing the data according to the compression rate, so as to set the compression strategy more appropriately for the real-time characteristics of the scene, so that the compression strategy used when compressing the data can achieve the best compression rate. For example, if the processor occupancy rate is less than the threshold, the compression rate of the compression strategy is low, and the computing device can increase the size of the dictionary area. If the processor occupancy rate is greater than or equal to the threshold, the network bandwidth between the computing device and the storage device is large, the compression rate is high, and the computing device can reduce the size of the dictionary area.

[0125] The computing device can adjust the compression feature value in the corresponding relationship as a whole, that is, adjust each element in the compression feature vector. The computing device can also adjust some elements in the compression feature vector. The computing device can also retain the historical vector of the same element to form a chain structure, thereby improving the reliability of the compression strategy update. For example, the computing device averages the historical value and current value of the elements in the compression feature vector (such as: processor occupancy, network bandwidth between the computing device and the storage device, bandwidth of the storage device, available storage capacity of the storage device, data type, data block size, or distribution of characters contained in the data to be compressed), and determines the average value as the updated value.

[0126] The computing device compresses the data to be compressed to obtain compressed data, and stores the compressed data in a storage device. For example, the method further includes step 480.

[0127] Step 480: The computing device sends the compressed data and the compression window to the storage device.

[0128] The computing device may send a data packet to a storage device (eg, hard disk 105), the data packet including the compressed data and the compression window used to compress the data to be compressed. Figure 8 As shown, the data packet may include a header 810 and a payload 820. The header 810 includes a basic header, that is, an address required to transmit the data packet. The payload 820 includes an extended header and data to be transmitted. The computing device may incorporate the offset into the compression header. The compression header includes a triplet of compressed data and a compression window. The compression header may be located in the payload 820 of the data packet. For example, the computing device may include the following information: Figure 6 and Figure 7 The compression rate is , and the compression window is selected as 6 characters. The compressed data are <0,0,a>, <1,1,c>, <3,4,b>, <3,3,a> and <1,2,c>.

[0129] In some embodiments, the computing device can dynamically adjust the total length of the number of bits occupied by the compression header according to the length of the compressed data, that is, the offset, matching length, and the number of bits of each character support dynamic adjustment, and the bit length of each triple in the compression header will be dynamically adjusted under different data types (such as special characters, Chinese characters, and English characters, etc.). Compared with a fixed-length compression header, each byte in the compression header is more fully utilized, increasing the compression ratio of the data.

[0130] After the computing device reads the compressed header data from the storage device, it can determine the size of the compression window used by the computing device during compression according to the compression window contained in the compression header, and decode the triplet of the compressed data according to the compression window to obtain the original data to be compressed.

[0131] It is understandable that in order to implement the functions in the above embodiments, the computing device includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and method steps of each example described in the embodiments disclosed in this application, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application scenario and design constraints of the technical solution.

[0132] Combined with the above Figures 1 to 8 , describes in detail the data compression method provided by this embodiment, and will now be combined with Fig. 9 , describing the data compression device provided according to this embodiment.

[0133] Fig. 9 Schematic diagram of the structure of a possible data compression device provided in this embodiment. These data compression devices can be used to implement the functions of the computing device in the above method embodiment, and thus can also achieve the beneficial effects of the above method embodiment. In this embodiment, the data compression device can be as follows Figures 1 to 3 The computing node 110 or storage node 100 shown may also be a module (such as a chip) applied to a server.

[0134] like Fig. 9 As shown, the data compression device 900 includes a communication module 910, a preprocessing module 920, a strategy matching module 930, a compression module 940 and a storage module 950. The data compression device 900 is used to implement the above Figure 4 The method embodiments shown in FIG. 1 are functions of a computing device.

[0135] The communication module 910 is used to obtain the data to be compressed, and transmit the compressed data and the compression window to the hard disk 105. For example, the communication module 910 is used to perform Figure 4 Step 410 and step 480.

[0136] The preprocessing module 920 is used to obtain parameters that affect the compression result of the data to be compressed, and determine the compression characteristic value of the data to be compressed according to the parameters that affect the compression result of the data to be compressed. Figure 4 Step 420 and step 430.

[0137] The strategy matching module 930 is used to determine the compression strategy for compressing the data to be compressed according to the compression feature value. Figure 4 Step 440.

[0138] The parameters include parameters of hardware resources used when compressing the data to be compressed and / or parameters describing data characteristics of the data to be compressed. The parameters of data characteristics include at least one of data type, data block size, and distribution of characters contained in the data to be compressed. The parameters of hardware resources include at least one of the occupancy rate of the processor of the computing device, the network bandwidth between the computing device and the storage device when storing the compressed data in the storage device, and the available storage capacity of the storage device.

[0139] The compression module 940 is used to compress the data to be compressed according to the compression strategy to obtain compressed data. For example, the compression module 940 is used to perform Figure 4 Step 450.

[0140] The strategy matching module 930 is specifically used to determine the compression feature value in the corresponding relationship corresponding to the compression feature value of the data to be compressed; and determine the compression strategy corresponding to the compression feature value as the compression strategy for compressing the data to be compressed according to the compression feature value determined by the corresponding relationship.

[0141] The compression module 940 is specifically used to compress the data to be compressed according to the multiple compression windows to obtain multiple compressed data, compare the compression rates of the multiple compressed data, and select the compressed data with the highest compression rate as the compressed data. The compression strategy includes multiple compression windows.

[0142] The storage module 950 is used to store the correspondence between multiple compression feature values ​​and compression strategies.

[0143] The data compression device 900 further includes an updating module 960. The updating module 960 is used to determine the compression ratio of the compression strategy used when compressing data; adjust the compression characteristic value and the parameters of the compression strategy in the corresponding relationship according to the compression ratio to obtain the adjusted compression characteristic value and the parameters of the compression strategy, and store the adjusted compression characteristic value and the parameters of the compression strategy in the storage module 950. For example, the updating module 960 is used to execute Figure 4Step 460 and step 470.

[0144] It should be understood that the data compression device 900 of the embodiment of the present application can be implemented by ASIC, or programmable logic device (PLD), the PLD can be a complex programmable logical device (CPLD), field-programmable gate array (FPGA), generic array logic (GAL) or any combination thereof. It can also be implemented by software. Figure 4 When the data compression method is shown, the data compression device 900 and its various modules can also be software modules.

[0145] The data compression device 900 according to the embodiment of the present application may correspond to executing the method described in the embodiment of the present application, and the above and other operations and / or functions of each unit in the data compression device 900 are respectively to implement Figure 4 For the sake of brevity, the corresponding processes of each method in are not repeated here.

[0146] Fig.10 The present embodiment provides a schematic diagram of the structure of a computing device 1000. As shown in the figure, the computing device 1000 includes a processor 1010, a bus 1020, a memory 1030, a memory unit 1040 and a communication interface 1050.

[0147] It should be understood that in this embodiment, the processor 1010 may be a CPU, and the processor 1010 may also be other general-purpose processors, digital signal processors (digital signal processing, DSP), application specific integrated circuits (application specific integrated circuits, ASIC), FPGAs or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0148] The processor may also be a graphics processing unit (GPU), a neural network processing unit (NPU), a microprocessor, an ASIC, or one or more integrated circuits for controlling the execution of the program of the present application.

[0149] The communication interface 1050 is used to implement communication between the computing device 1000 and an external device or component. In this embodiment, the communication interface 1050 is used to receive data to be compressed and send compressed data.

[0150] The bus 1020 may include a path for transmitting information between the above components (such as the processor 1010, the memory unit 1040, and the memory 1030). In addition to the data bus, the bus 1020 may also include a power bus, a control bus, and a status signal bus. However, for the sake of clarity, various buses are labeled as bus 1020 in the figure. The bus 1020 may be a Peripheral Component Interconnect Express (PCIe) bus, or an extended industry standard architecture (EISA) bus, a unified bus (Ubus or UB), a compute express link (CXL), a cache coherent interconnect for accelerators (CCIX), etc.

[0151] As an example, the computing device 1000 may include multiple processors. The processor may be a multi-core (multi-CPU) processor. The processor here may refer to one or more devices, circuits, and / or computing units for processing data (e.g., computer program instructions). The processor 1010 may call the correspondence between multiple compression feature values ​​and compression strategies stored in the memory 1030, determine the compression strategy for compressing the data to be compressed according to the compression feature values, and compress the data to be compressed according to the compression strategy to obtain compressed data.

[0152] It is worth mentioning that Fig.10 In the example, a computing device 1000 including one processor 1010 and one memory 1030 is taken. Here, the processor 1010 and the memory 1030 are respectively used to indicate a type of device or equipment. In a specific embodiment, the number of each type of device or equipment can be determined according to business requirements.

[0153] The memory unit 1040 may correspond to a storage medium for storing information such as the correspondence between a plurality of compression feature values ​​and compression strategies in the above method embodiment. The memory unit 1040 may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0154] The memory 1030 is used to store data and may be a disk, such as a mechanical hard disk or a solid-state hard disk.

[0155] The computing device 1000 may be a general-purpose device or a dedicated device. For example, the computing device 1000 may be a mobile phone terminal, a tablet computer, a laptop computer, a VR device, an AR device, an MR device or an ER device, a vehicle terminal, etc. It may also be an edge device (e.g., a box carrying a chip with processing capabilities), etc. Optionally, the computing device 1000 may also be a server or other device with computing capabilities.

[0156] It should be understood that the computing device 1000 according to this embodiment may correspond to the data compression device 900 in this embodiment, and may correspond to the device executing the Figure 4 The corresponding subject in any method, and the above and other operations and / or functions of each module in the data compression device 900 are respectively to achieve Figure 4 For the sake of brevity, the corresponding processes of each method in are not repeated here.

[0157] The method steps in this embodiment can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable ROM (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC. In addition, the ASIC can be located in a network device or a terminal device. Of course, the processor and the storage medium can also exist in a network device or a terminal device as discrete components.

[0158] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instruction is loaded and executed on a computer, the process or function described in the embodiment of the present application is executed in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user device or other programmable device. The computer program or instruction may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer program or instruction may be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired or wireless means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium, for example, a floppy disk, a hard disk, a tape; it may also be an optical medium, for example, a digital video disc (DVD); it may also be a semiconductor medium, for example, a solid state drive (SSD).

[0159] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A data compression method, characterized in that: The method comprises: Determine a compression characteristic value of the data to be compressed according to a preset parameter, wherein the preset parameter is a parameter that will affect the compression result of the data to be compressed; Determine a compression strategy corresponding to the compression characteristic value from a plurality of compression strategies; The data to be compressed is compressed using the compression strategy, and the compressed data is stored.

2. The method according to claim 1, characterized in that: The preset parameters include parameters of hardware resources used when compressing the data to be compressed and / or parameters describing data characteristics of the data to be compressed.

3. The method according to claim 2, characterized in that The parameters of the data feature include at least one of a data type, a data block size, and a distribution of characters included in the data to be compressed.

4. The method according to claim 2, characterized in that: The parameters of the hardware resources include at least one of the occupancy rate of the processor of the computing device, the network bandwidth between the computing device and the storage device when storing the compressed data in the storage device, and the available storage capacity of the storage device.

5. The method according to any one of claims 1 to 4, characterized in that The computing device stores a correspondence between a plurality of compression feature values ​​and compression strategies; Determining a compression strategy corresponding to the compression characteristic value from a plurality of compression strategies includes: Determine the compression characteristic value in the corresponding relationship corresponding to the compression characteristic value of the data to be compressed; According to the compression characteristic value determined by the corresponding relationship, a compression strategy corresponding to the compression characteristic value is determined as the compression strategy for compressing the data to be compressed.

6. The method according to claim 5, characterized in that The correspondence between the plurality of compression feature values ​​and the compression strategy is obtained based on neural network training.

7. The method according to any one of claims 1 to 6, characterized in that The compression strategy includes multiple compression windows; Compressing the data to be compressed according to the compression strategy includes: Compressing the data to be compressed respectively according to the multiple compression windows to obtain multiple compressed data; The compression rates of the plurality of compressed data are compared, and the compressed data with the highest compression rate is selected as the compressed data.

8. The method according to any one of claims 1 to 7, characterized in that After obtaining the compressed data, the method further includes: Determining a compression ratio of the compression strategy used when compressing data; The compression characteristic value and the parameters of the compression strategy in the corresponding relationship are adjusted according to the compression rate.

9. A data compression device, characterized in that: The device comprises: A preprocessing module, used to determine the compression characteristic value of the data to be compressed according to a preset parameter, wherein the preset parameter is a parameter that will affect the compression result of the data to be compressed; A strategy matching module, used to determine a compression strategy corresponding to the compression characteristic value from a plurality of compression strategies; The compression module is used to compress the data to be compressed using the compression strategy and store the compressed data.

10. The device according to claim 9, characterized in that The preset parameters include parameters of hardware resources used when compressing the data to be compressed and / or parameters describing data characteristics of the data to be compressed.

11. The device according to claim 10, characterized in that The parameters of the data feature include at least one of a data type, a data block size, and a distribution of characters included in the data to be compressed.

12. The device according to claim 10, characterized in that The parameters of the hardware resources include at least one of the occupancy rate of the processor of the computing device, the network bandwidth between the computing device and the storage device when storing the compressed data in the storage device, and the available storage capacity of the storage device.

13. The device according to any one of claims 9 to 12, characterized in that The computing device stores a correspondence between a plurality of compression feature values ​​and compression strategies; When the strategy matching module determines the compression strategy for compressing the data to be compressed according to the compression feature value, it is specifically used to: Determine the compression characteristic value in the corresponding relationship corresponding to the compression characteristic value of the data to be compressed; According to the compression characteristic value determined by the corresponding relationship, a compression strategy corresponding to the compression characteristic value is determined as the compression strategy for compressing the data to be compressed.

14. The device according to claim 13, characterized in that The correspondence between the plurality of compression feature values ​​and the compression strategy is obtained based on neural network training.

15. The device according to any one of claims 9 to 14, characterized in that The compression strategy includes multiple compression windows; When the compression module compresses the data to be compressed according to the compression strategy, it is specifically used to: Compressing the data to be compressed respectively according to the multiple compression windows to obtain multiple compressed data; The compression rates of the plurality of compressed data are compared, and the compressed data with the highest compression rate is selected as the compressed data.

16. The device according to any one of claims 9 to 15, characterized in that After obtaining the compressed data, the device further comprises an updating module; The updating module is used to determine the compression ratio of the compression strategy used when compressing data; and to adjust the compression characteristic value and the parameters of the compression strategy in the corresponding relationship according to the compression ratio.

17. A computing device, characterized in that The computing device includes a memory and at least one processor, wherein the memory is used to store a set of computer instructions; when the processor executes the set of computer instructions, the operation steps of the method described in any one of claims 1 to 8 are performed.

18. A storage system, characterized in that: The storage system comprises at least one computing node and at least one storage node, the computing node is used to execute the operation steps of the method according to any one of claims 1 to 8 to obtain compressed data, and the storage node is used to store the compressed data; Alternatively, the computing node transmits the data to be compressed to the storage node, and the storage node executes the operation steps of the method described in any one of claims 1 to 8 to obtain compressed data and stores the compressed data.

Images