Data compression method and related system

Abstract

The invention provides a data compression method which comprises the following steps: acquiring a to-be-persisted data file which comprises a plurality of sub-files, and when a first sub-file in the plurality of sub-files is persisted, generating a dictionary corresponding to the first sub-file according to the first sub-file, and compressing the first sub-file by using the dictionary to obtain a first compressed file, generating a first persistent file according to the first compressed file and the dictionary, and writing the first persistent file into the persistent storage medium. A single persistent file independently maintains the dictionary corresponding to the sub-file, and the dictionary and the sub-file can realize the same life cycle management, that is, the data in the sub-file and the life cycle of the dictionary are bound in an affinity manner, so that the long tail problem that the common dictionary cannot be released due to the fact that a local data file quotes a global common dictionary is solved, and the user experience is improved. Therefore, the problem that the performance of a storage system is reduced due to metadata expansion caused by the fact that the common dictionary occupies the memory for a long time and cannot be released is solved.

Description

[0001] This application is a divisional application. The application number of the original application is 202310803573.7, and the original application date is June 30, 2023. The entire contents of the original application are incorporated by reference into this application. This application and the Chinese application with application number 202310803573.7 both claim the priority of the Chinese patent application with application number 202310473699.2 filed on April 26, 2023, and the invention name is “A Data Compression Method”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of storage technology, and in particular to a data compression method, device, storage system, computer-readable storage medium, and computer program product. Background Art

[0003] The widespread use of cloud computing, the fifth generation (5G) communication system, and the rapid popularization of big data are accelerating the production and flow of data, promoting the continuous release of data value, and posing greater challenges to data storage capacity. Larger data volumes and longer storage cycles mean higher storage costs, which puts higher cost-effective demands on data-centric infrastructure and memory-centric system architecture. Solving the problem of massive data storage has become a key factor affecting the procurement and maintenance costs of data center infrastructure.

[0004] All major data storage vendors have introduced corresponding data reduction features, which significantly increase the user's available capacity under the same physical capacity and reduce the total cost of ownership (TCO). Currently, the industry has introduced data compression algorithms such as LZ4, ZSTD, and Snappy to achieve data reduction features. The above data compression algorithms can provide multiple levels of data compression rate. However, due to the exponential growth trend of data volume, the above data compression algorithms are gradually unable to fully match the production, storage, and application requirements of massive data.

[0005] Data storage vendors try to use public dictionary (PD) compression to break through the boundaries of compression algorithms such as LZ4 and Snappy. Public dictionary compression can further improve the data reduction rate by mining a wider range of richer data features. However, the public dictionary provides global information independent of the user's data files. Different data files (such as data files corresponding to different columns of a database table) can reference the same public dictionary. The difference in the life cycle of local data files and the global public dictionary leads to a long-tail problem in which the public dictionary occupies memory for a long time and cannot be released. The memory occupied by the public dictionary will increase sharply, affecting the performance of the storage system. Summary of the invention

[0006] The present application provides a data compression method, which independently maintains the dictionary corresponding to the sub-file in a single persistent file, and the dictionary and the sub-file can achieve the same life cycle management, that is, the life cycle affinity binding of the data in the sub-file and the dictionary solves the long tail problem that the local data file references the global public dictionary, resulting in the public dictionary cannot be released, and ensures the performance of the storage system. The present application also provides a device, a storage system, a computer-readable storage medium and a computer program product corresponding to the above method.

[0007] In a first aspect, the present application provides a data compression method. The method may be performed by a storage system. The storage system may be a centralized storage system, including a centralized storage system with integrated disk and controller, or a centralized storage system with separate disk and controller, or the storage system may be a distributed storage system, including but not limited to a distributed storage system with integrated storage and computing, or a distributed storage system with separate storage and computing.

[0008] Specifically, the storage system obtains a data file to be persisted, which includes multiple sub-files. When persisting the first sub-file among the multiple sub-files: a dictionary corresponding to the first sub-file is generated according to the first sub-file, and the first sub-file is compressed using the dictionary to obtain a first compressed file, and then a first persistent file is generated according to the first compressed file and the dictionary, and the first persistent file is written to a persistent storage medium.

[0009] In this method, the storage system persists the dictionary and the compressed file obtained by compressing the sub-file using the dictionary in the same persistent file. The dictionary and the sub-file can implement the same life cycle management, that is, the data in the sub-file and the life cycle of the dictionary are affinity-bound, which solves the long-tail problem that the local data file references the global public dictionary, resulting in the public dictionary cannot be released. This avoids the public dictionary occupying the memory for a long time and cannot be released, causing metadata expansion (metadata overflow), resulting in the public dictionary that needs to be used cannot be hit in the memory, and needs to be frequently swapped in and out, which in turn leads to the problem of storage system performance degradation.

[0010] In some possible implementations, the first persistent file includes a pre-allocated data area and a metadata area, the data area stores the first compressed file, and a specified position of the metadata area stores a dictionary corresponding to the first sub-file. The specified position is determined according to configuration information for pre-allocating the metadata area. The configuration information for pre-allocating the metadata area includes an offset of the metadata area in the first persistent file and an offset of the dictionary in the metadata area.

[0011] This method adopts a separate space reservation strategy, divides the data area and metadata area in the persistent file, and completes the space reservation in advance when generating the persistent file, so that after the dictionary is generated, the corresponding space reserved in the metadata area is pre-filled with the dictionary (that is, the dictionary is filled into the reserved metadata area). The data area and the metadata area have a natural mapping relationship, which can achieve non-intrusive metadata and zero-overhead maintenance. There is no need to develop additional persistence and mapping update functions for mapping metadata, which avoids the impact on the existing storage system architecture and reduces the development difficulty and system complexity.

[0012] In some possible implementations, the storage system may receive a read request, which is used to read a first sub-file. The storage system then reads a dictionary corresponding to the first sub-file from a specified position in the metadata area of ​​the first persistent file based on the configuration information of the pre-allocated metadata area, and reads the first compressed file from the data area of ​​the first persistent file based on the configuration information of the pre-allocated data area, wherein the configuration information of the pre-allocated data area includes an offset of the data area in the first persistent file. The storage system may use the dictionary corresponding to the first sub-file to decompress the first compressed file to obtain the first sub-file.

[0013] For upper-layer read requests, the storage system can combine the fixed index information (such as offset) of each persistent file for the dictionary space to perform precise pre-fetching, improve the cache hit rate, and provide subsequent read request hits, greatly reducing the input and output (IO) path and improving read performance.

[0014] In some possible implementations, as data is written and overwritten, invalid data (garbage data) may be generated in a persistent file, where invalid data and garbage data refer to data that has been rewritten, such as data that has been rewritten in a subfile corresponding to a compressed file stored in a data area of ​​a persistent file. In contrast to invalid data is valid data, which refers to data that has not been rewritten. The storage system can also recycle garbage data to free up space. Specifically, the storage system can detect whether a merge condition is met; when the merge condition is met, the storage system merges multiple persistent files that have been written to the persistent storage medium to obtain a new persistent file, and deletes the multiple persistent files.

[0015] In this way, junk data of persistent files can be recycled, storage space waste can be avoided, and storage resource utilization can be improved.

[0016] In some possible implementations, the storage system may decompress multiple persistent files written to the persistent storage medium, obtain valid data from the decompressed sub-files respectively, and merge the valid data to obtain a merged sub-file. The valid data is data that has not been rewritten. The storage system may then generate a dictionary corresponding to the merged sub-file based on the merged sub-file, and compress the merged sub-file using the dictionary corresponding to the merged sub-file to obtain a merged compressed file, and generate a new persistent file based on the merged compressed file and the dictionary corresponding to the merged sub-file.

[0017] On the one hand, it can achieve lifecycle affinity binding between data (such as sub-files of data files) and dictionaries, achieve zero overlap between data and dictionaries, no metadata amplification, and avoid performance problems caused by metadata overflow. On the other hand, regenerating a new dictionary that meets the characteristics of the merged sub-files can achieve a better compression rate.

[0018] In some possible implementations, the merging condition includes at least one of the following:

[0019] The interval between the current time and the last merge time reaches the merge period; or,

[0020] The current load is less than the preset value; or,

[0021] The effective data ratio of at least one persistent file among the multiple persistent files written into the persistent storage medium is less than a preset ratio, where the effective data ratio is the ratio of the data that has not been rewritten in the subfile corresponding to the persistent file; or

[0022] A merge request reported when a user-triggered merge operation is detected.

[0023] This method supports multiple merging modes, such as periodic merging, off-peak merging, merging based on the proportion of valid data, or manually triggered merging, which can meet the needs of different businesses and has high availability.

[0024] In some possible implementations, the storage system may determine multiple persistent files to be merged according to the attribute information of the persistent files written into the persistent storage medium, and merge the multiple persistent files to obtain a new persistent file.

[0025] This method screens persistent files written to a persistent storage medium according to the attribute information of the persistent files to determine multiple persistent files to be merged, rather than merging any persistent files. This can achieve a merging benefit greater than the merging cost and meet business needs.

[0026] In some possible implementations, the attribute information of the persistent file includes a write time or a valid data ratio, wherein the valid data ratio is the ratio of data that has not been rewritten in the subfile corresponding to the persistent file. Accordingly, the storage system may sort the persistent files that have been written to the persistent storage medium according to the write time of the persistent files that have been written to the persistent storage medium, and determine a plurality of persistent files to be merged according to the sorting result. Alternatively, the storage system may determine a plurality of persistent files to be merged according to the valid data ratio of the persistent files that have been written to the persistent storage medium, and the plurality of persistent files to be merged include persistent files whose valid data ratio is less than a preset ratio.

[0027] In this method, the storage system can determine the persistent files to be merged based on the first-in-first-out principle, or based on the principle of giving priority to merging persistent files with a high proportion of junk data, and then merge the above persistent files to obtain the highest possible merging benefit.

[0028] In some possible implementations, the storage system may also update the physical addresses in the address mapping relationship of the multiple persistent files to the physical addresses of the new persistent file. In this way, when the storage system receives a read request to read valid data in a subfile corresponding to the deleted persistent file, it can index to the new persistent file according to the updated address mapping relationship, thereby achieving accurate reading and ensuring read performance.

[0029] In some possible implementations, when receiving a write request, the storage system can also write the data in the write request to the log first, obtain the data file to be persisted from the log for persistence, and generate a dictionary during the log flushing process. If a failure occurs at this stage, the data can be replayed because the log is still there, so the dictionary can be regenerated. In this way, data write protection can be implemented to ensure high reliability and high availability of data.

[0030] In a second aspect, the present application provides a data compression device. The device comprises:

[0031] An acquisition module, used for acquiring a data file to be persisted, wherein the data file to be persisted includes a plurality of sub-files;

[0032] A persistence module is used for persisting the first sub-file among the multiple sub-files:

[0033] generating a dictionary corresponding to the first subfile according to the first subfile, and compressing the first subfile using the dictionary to obtain a first compressed file;

[0034] generating a first persistent file according to the first compressed file and the dictionary; and

[0035] The first persistent file is written to the persistent storage medium.

[0036] In some possible implementations, the first persistent file includes a pre-allocated data area and a metadata area, the data area stores the first compressed file, a specified position of the metadata area stores a dictionary corresponding to the first sub-file, the specified position is determined based on configuration information for pre-allocating the metadata area, the configuration information for pre-allocating the metadata area includes the offset of the metadata area in the first persistent file and the offset of the dictionary in the metadata area.

[0037] In some possible implementations, the acquisition module is further configured to:

[0038] receiving a read request, the read request being used to read the first sub-file;

[0039] The device also includes:

[0040] a data reading module, configured to read a dictionary corresponding to the first sub-file from a specified position of the metadata area of ​​the first persistent file according to configuration information for pre-allocating the metadata area, and read the first compressed file from the data area of ​​the first persistent file according to configuration information for pre-allocating the data area, wherein the configuration information for pre-allocating the data area includes an offset of the data area in the first persistent file;

[0041] The first compressed file is decompressed using a dictionary corresponding to the first subfile to obtain the first subfile.

[0042] In some possible implementations, the device further includes:

[0043] A detection module, used to detect whether the merging condition is met;

[0044] The merging module is used to merge the multiple persistent files written into the persistent storage medium to obtain a new persistent file and delete the multiple persistent files when the merging condition is met.

[0045] In some possible implementations, the merging module is specifically used to:

[0046] Decompressing multiple persistent files written to the persistent storage medium, respectively acquiring valid data from the decompressed sub-files and merging the valid data to obtain a merged sub-file, wherein the valid data is data that has not been rewritten;

[0047] Generate a dictionary corresponding to the merged sub-file according to the merged sub-file, and compress the merged sub-file using the dictionary corresponding to the merged sub-file to obtain a merged compressed file;

[0048] A new persistent file is generated according to the merged compressed file and the dictionary corresponding to the merged sub-file.

[0049] In some possible implementations, the merging condition includes at least one of the following:

[0050] The interval between the current time and the last merge time reaches the merge period; or,

[0051] The current load is less than the preset value; or,

[0052] The effective data ratio of at least one persistent file among the multiple persistent files written into the persistent storage medium is less than a preset ratio, where the effective data ratio is the ratio of the data that has not been rewritten in the subfile corresponding to the persistent file; or

[0053] A merge request reported when a user-triggered merge operation is detected.

[0054] In some possible implementations, the merging module is specifically used to:

[0055] Determining multiple persistent files to be merged according to the attribute information of the persistent files written into the persistent storage medium;

[0056] The multiple persistent files are merged to obtain a new persistent file.

[0057] In some possible implementations, the attribute information of the persistent file includes a write time or a valid data ratio, where the valid data ratio is a ratio of data that has not been rewritten in a subfile corresponding to the persistent file;

[0058] The merging module is specifically used for:

[0059] sorting the persistent files written into the persistent storage medium according to the writing time of the persistent files written into the persistent storage medium, and determining the plurality of persistent files to be merged according to the sorting result; or,

[0060] According to the effective data ratio of the persistent files written into the persistent storage medium, a plurality of persistent files to be merged are determined, wherein the plurality of persistent files to be merged include persistent files whose effective data ratio is less than a preset ratio.

[0061] In some possible implementations, the device further includes:

[0062] The address mapping update module is used to update the physical address in the address mapping relationship of multiple persistent files to the physical address of the new persistent file.

[0063] In a third aspect, the present application provides a storage system. The storage system includes a controller and a hard disk. The controller and the hard disk communicate with each other. The controller is used to execute computer-readable instructions so that the storage system performs the data compression method described in the first aspect or any implementation of the first aspect.

[0064] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, wherein the instructions instruct a storage system to execute the data compression method described in the first aspect or any one of the implementations of the first aspect.

[0065] In a fifth aspect, the present application provides a computer program product comprising instructions, which, when executed on a storage system, enables the storage system to execute the data compression method described in the first aspect or any one of the implementations of the first aspect.

[0066] Based on the implementations provided in the above aspects, this application can also be further combined to provide more implementations. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the technical method of the embodiments of the present application, the drawings required for use in the embodiments are briefly introduced below.

[0068] Figure 1A A schematic diagram of the architecture of a centralized storage system with disk and controller separation provided in an embodiment of the present application;

[0069] Figure 1B A schematic diagram of the architecture of a centralized storage system with integrated disk and controller provided in an embodiment of the present application;

[0070] Figure 1C A schematic diagram of the architecture of a distributed storage system integrating storage and computing provided in an embodiment of the present application;

[0071] Figure 1D A schematic diagram of the architecture of a distributed storage system with storage and computing separation provided in an embodiment of the present application;

[0072] Figure 2 A flowchart of a data compression method provided in an embodiment of the present application;

[0073] Figure 3 A flowchart of a dictionary generation process provided in an embodiment of the present application;

[0074] Figure 4A schematic diagram of a persistent file recovery process provided in an embodiment of the present application;

[0075] Figure 5 A schematic diagram of a flow chart of a method for compressing data in a database provided in an embodiment of the present application;

[0076] Figure 6 A schematic diagram of a flow chart of a data compression method in a distributed storage system provided in an embodiment of the present application;

[0077] Figure 7 A schematic diagram of a data reading and writing process provided in an embodiment of the present application;

[0078] Figure 8 A schematic diagram of the structure of a computer cluster provided in an embodiment of the present application. DETAILED DESCRIPTION

[0079] The terms "first" and "second" in the embodiments of the present application are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features.

[0080] First, some technical terms involved in the embodiments of the present application are introduced.

[0081] Data compression (DC) is the process of representing information using fewer bits (or other information-related units) than the uncoded information according to a specific coding mechanism. Data compression can be applied to compress text, audio, video or images to reduce storage space and storage costs.

[0082] The data compression method provided in the present application can be applied to a storage system. The above storage system can be divided into the following categories according to logical functions: primary storage and backup storage. The primary storage serves as a storage pool or hard disk for upper-layer applications and can be accessed by upper-layer applications such as applications deployed in the form of virtual machines (VMs). The backup storage serves as a warehouse for storing images, backup disks or snapshots, which are usually not directly accessible to VMs. The primary storage and backup storage can be physically separated storage systems, or the same storage system can serve as both primary storage and backup storage. The storage system can also be divided into file storage, database storage (including row-based databases or column-based databases), and big data storage according to the application scenario.

[0083] The above storage system (including primary storage and backup storage) can adopt local storage mode or cloud storage mode. Among them, local storage refers to the mode of storing data in one's own physical storage device, and cloud storage refers to the mode of storing data in a cloud environment to support access to data through the network. The cloud environment refers to the cluster built by the cloud service provider (also known as cloud vendor) to provide computing resources, storage resources, and network resources.

[0084] The storage system may adopt a centralized storage architecture or a distributed storage architecture. In other words, the storage system may be a centralized storage system or a distributed storage system. Examples are given below to illustrate each.

[0085] A centralized storage system is a central node composed of one or more main devices. Data is stored in this central node, and all data processing services of the entire system are centrally deployed on this central node. In other words, in a centralized storage system, the terminal or client is only responsible for data entry and output, while the storage and control processing of data are completely completed by the central node. The biggest feature of a centralized system is its simple deployment structure. There is no need to consider how to deploy services on multiple nodes, nor is there any need to consider the distributed collaboration between multiple nodes.

[0086] For ease of understanding, Figure 1A A centralized storage system architecture is shown. Figure 1A In the application scenario shown, users access data through applications. Computers that run these applications are called "application servers". Application server 100 can be a physical machine or a virtual machine VM. Physical machines (physical application servers) include but are not limited to desktop computers, servers, laptops, and mobile devices. The application server accesses the storage system through the fiber optic switch 110 to access data. However, the switch 110 is only an optional device, and the application server 100 can also communicate directly with the storage system 120 through the network. Alternatively, the fiber optic switch 110 can also be replaced with an Ethernet switch, an InfiniBand (IB) switch, an RDMA (RDMA over Converged Ethernet, RoCE) switch based on converged Ethernet, etc.

[0087] Figure 1A The storage system 120 shown is a centralized storage system. The centralized storage system is characterized by having a unified entrance through which all data from external devices must pass, and this entrance is the engine 121 of the centralized storage system. The engine 121 is the core component of the centralized storage system, and many advanced functions of the storage system are implemented in it.

[0088] like Figure 1AAs shown, the engine 121 has one or more controllers. Figure 1A Take the example of an engine containing two controllers. There is a mirror channel between controller 0 and controller 1. Then, when controller 0 writes a copy of data into its memory 124, it can send a copy of the data to controller 1 through the mirror channel, and controller 1 stores the copy in its local memory 124. In this way, controller 0 and controller 1 back up each other. When controller 0 fails, controller 1 can take over the business of controller 0, and when controller 1 fails, controller 0 can take over the business of controller 1, thereby avoiding the unavailability of the entire storage system 120 due to hardware failure. When 4 controllers are deployed in the engine 121, there is a mirror channel between any two controllers, so any two controllers back up each other.

[0089] The engine 121 also includes a front-end interface 125 and a back-end interface 126, wherein the front-end interface 125 is used to communicate with the application server 100, thereby providing storage services for the application server 100. The back-end interface 126 is used to communicate with the hard disk 134 to expand the capacity of the storage system. Through the back-end interface 126, the engine 121 can connect more hard disks 134, thereby forming a very large storage resource pool.

[0090] In hardware, such as Figure 1A As shown, the controller 0 includes at least a processor 123 and a memory 124. The processor 123 is a central processing unit (CPU) used to process data access requests from outside the storage system (server or other storage system), and also used to process requests generated inside the storage system. Exemplarily, when the processor 123 receives a write data request sent by the application server 100 through the front-end port 125, the data in these write data requests will be temporarily stored in the memory 124. When the total amount of data in the memory 124 reaches a certain threshold, the processor 123 sends the data stored in the memory 124 to the hard disk 134 through the back-end port for persistent storage.

[0091] The memory 124 refers to an internal memory that directly exchanges data with the processor. It can read and write data at any time and at a high speed, and serves as a temporary data storage for the operating system or other running programs. The memory includes at least two types of memories, for example, the memory can be either a random access memory or a read-only memory (ROM). For example, the random access memory is a dynamic random access memory (DRAM) or a storage class memory (SCM), and the read-only memory can be a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), etc.

[0092] The memory 124 stores software programs, and the processor 123 runs the software programs in the memory 124 to manage the hard disk. For example, the hard disk is abstracted into a storage resource pool, and then divided into LUNs for use by the server. The LUN here is actually the hard disk seen on the server. Of course, some centralized storage systems are also file servers themselves, which can provide shared file services for the server.

[0093] Controller 1 (and others Figure 1A The hardware components and software structure of the controller (not shown in the figure) are similar to those of the controller 0 and will not be repeated here.

[0094] Figure 1A The illustrated embodiment is a centralized storage system with a disk-controller separation. In the system, the engine 121 may not have a hard disk slot, and the hard disk 134 needs to be placed in the hard disk frame 130, and the back-end interface 126 communicates with the hard disk frame 130. The back-end interface 126 exists in the engine 121 in the form of an adapter card, and two or more back-end interfaces 126 can be used simultaneously on one engine 121 to connect multiple hard disk frames. Alternatively, the adapter card can also be integrated on the motherboard, and the adapter card can communicate with the processor 123 via the PCIE bus.

[0095] It should be noted that Figure 1A Only one engine 121 is shown in the figure. However, in actual applications, the storage system may include two or more engines 121, and redundancy or load balancing is performed between the multiple engines 121.

[0096] The hard disk frame 130 includes a control unit 131 and a plurality of hard disks 134. The control unit 131 may have various forms. Generally, the number of control units 131 may be one, two or more. When the hard disk frame 130 includes at least two control units 131, the hard disk 134 and the control unit 131 may have an ownership relationship. The function of the control unit 131 may be offloaded to the network card 104. In this embodiment, the hard disk frame 130 does not have a control unit 131 inside, but the network card 104 completes data reading and writing, address conversion and other computing functions. At this time, the network card 104 is an intelligent network card. It may include a CPU and memory. In some application scenarios, the network card 104 may also have a persistent memory medium, such as a persistent memory (PM), or a non-volatile random access memory (NVRAM), or a phase change memory (PCM). The CPU is used to perform operations such as address conversion and reading and writing data. The memory is used to temporarily store data to be written to the hard disk 134, or data read from the hard disk 134 to be sent to the controller.

[0097] The centralized storage system can also be a disk-control integrated architecture. In the disk-control integrated architecture, the engine 121 has a hard disk slot, and the hard disk 134 can be directly deployed in the engine 121. The back-end interface 126 is an optional configuration. When the system storage space is insufficient, more hard disks or hard disk frames can be connected through the back-end interface 126. Figure 1B As shown, the engine 121 can be connected to more hard disks 134 through the backend interface 126 .

[0098] A distributed storage system is a system that stores data in multiple independent storage nodes. The distributed storage system adopts an extensible system structure and uses multiple storage nodes to share the storage load. It not only improves the reliability, availability and access efficiency of the system, but also makes it easy to expand.

[0099] Distributed storage systems can adopt a storage-computing integrated architecture or a storage-computing separated architecture. The following describes each of them.

[0100] like Figure 1C As shown, the distributed storage system integrating storage and computing includes a storage cluster. The storage cluster includes one or more servers 110 ( Figure 1CThree servers 110 are shown in the figure, but not limited to three servers 110), and each server 110 can communicate with each other. Server 110 is a device that has both computing and storage capabilities, such as a server, a desktop computer, etc. For example, an ARM server or an X86 server can be used as the server 110 here. In terms of hardware, server 110 includes at least a processor 112, a memory 113, a network card 114, and a hard disk 105. The processor 112, the memory 113, the network card 114, and the hard disk 105 are connected through a bus. Processor 112 is a central processing unit (CPU), which is used to process data access requests from outside the server 110 (application server or other server 110), and is also used to process requests generated inside the server 110. In addition, processor 112 is also used for data calculation or processing, such as metadata management, deduplication, data compression, data verification, virtualized storage space, and address conversion.

[0101] like Figure 1D As shown in FIG. 1 , the distributed storage cluster with storage and computing separation includes a computing cluster (also called a computing node cluster) and a storage cluster (also called a storage node cluster). Each computing node 110 can communicate with each other. The computing node 110 is a computing device, such as a server, a desktop computer, or a controller of a storage array. In terms of hardware, Figure 1D As shown, the computing node 110 includes at least a processor 112, a memory 113 and a network card 114. Among them, the processor 112 is a central processing unit (CPU), which is used to process data access requests from outside the computing node 110, or requests generated inside the computing node 110. When the total amount of data in the memory 113 reaches a certain threshold, the processor 112 sends the data stored in the memory 113 to the storage node 100 for persistent storage. In addition, the processor 112 is also used for data calculation or processing, such as metadata management, deduplication, data compression, virtualized storage space, and address translation.

[0102] Any computing node 110 can access any storage node 100 in the storage node cluster through the network. The storage node cluster includes multiple storage nodes 100 ( Figure 1DThree storage nodes 100 are shown in the figure, but are not limited to three storage nodes 100). A storage node 100 includes one or more controllers 101, a network card 104 and multiple hard disks 105. The network card 104 is used to communicate with the computing node 110. The hard disk 105 is used to store data and can be a disk or other types of storage media, such as a solid-state hard disk or a shingled magnetic recording hard disk. The controller 101 is used to write data to the hard disk 105 or read data from the hard disk 105 according to the read / write data request sent by the computing node 110. In the process of reading and writing data, the controller 101 needs to convert the address carried in the read / write data request into an address that can be recognized by the hard disk. It can be seen that the controller 101 also has some simple computing functions.

[0103] It should be noted that the above is merely an example description of the storage system. In other possible implementations of the embodiments of the present application, the storage system may also be a distributed storage system or a centralized storage system of other architectures. For example, the storage system may also be a distributed storage system of a fully integrated architecture.

[0104] When storing data, the storage system can compress the data to reduce the storage space occupied. Different data compression algorithms can have different data compression rates. For example, algorithms such as LZ4, ZSTD, and Snappy can provide data compression rates of multiples. However, the amount of data is growing exponentially, and traditional compression algorithms represented by LZ4, ZSTD, and Snappy cannot fully match the production, storage, and application requirements of massive data. The industry has also proposed a new lossless data compression method that breaks through the boundaries of traditional compression algorithms. Public Dictionary (PD) compression is a typical representative of this. Public dictionary compression is a new data compression solution that breaks through local limitations and further improves data reduction rates by mining a wider range and richer data features.

[0105] The following example uses a common dictionary to apply to a column-based database. A column-based database uses a table partition mechanism, which is also called a column cluster. The data of each table partition (column cluster) is stored in a separate file. For a column cluster, every K rows of data form a row group (each row group includes a fixed number of rows). In each row group, each column of data is continuously stored in a basic data processing unit such as a data block (denoted as a block). Since each block has the same data type, it can support efficient data compression.

[0106] To speed up queries, the storage system can build a metadata for each column and save it in a separate file. The metadata mainly includes the following four parts:

[0107] ①Header: includes version number (which can be represented by timestamp, for example), file length and some statistical information;

[0108] ② Column statistics: including the number of rows, the number of NULL values, cardinality (the number of different values), SUM (for sum), MAX (for maximum), and MIN (for minimum) values;

[0109] ③ Dictionary reference relationship: Public dictionaries belong to metadata. The dictionary number of the public dictionary is stored in the data file that references the public dictionary, thereby establishing a reference relationship with the public dictionary;

[0110] ④Block address information: saves the mapping relationship between block number and data file starting address and length.

[0111] Since the public dictionary and local data files can be a one-to-many mapping relationship, that is, a public dictionary can be referenced by multiple data files, there is a difference in the life cycle between the public dictionary and the local data files. Due to the difference in life cycle, after the public dictionary is loaded into the memory during the query process, if the multiple data files that reference the public dictionary are not completely deleted, the public dictionary cannot be released. This operation will inevitably lead to the long-tail problem that the public dictionary occupies the memory for a long time and cannot be released, and then metadata expansion will occur, resulting in a sharp increase in the memory occupied by the public dictionary, resulting in dictionary data swapping in and out, and the public dictionary that needs to be used cannot be hit in the memory, which can lead to performance degradation.

[0112] In view of this, the present application provides a data compression method. The method generates a dictionary corresponding to the first sub-file according to the first sub-file when persisting the first sub-file in the data file to be persisted, compresses the first sub-file using the dictionary to obtain a first compressed file, and then persists the first compressed file and the dictionary in the same persistent file, for example, pre-partitioning the persistent file to include a data area and a metadata area, pre-filling the dictionary corresponding to the first sub-file in the metadata area reserved in the persistent file, and filling the first compressed file in the data area reserved in the persistent file, thereby achieving persistence of the first sub-file and the dictionary corresponding to the first sub-file together.

[0113] In this way, a single persistent file independently maintains the dictionary corresponding to the sub-file, and the dictionary and the sub-file can implement the same life cycle management, that is, the data in the sub-file and the dictionary life cycle are affinity bound, which solves the long-tail problem that the local data file references the global public dictionary, resulting in the public dictionary cannot be released. This avoids the public dictionary occupying the memory for a long time and cannot be released, causing metadata expansion (metadata overflow) and resulting in the public dictionary that needs to be used cannot be hit in the memory, and needs to be frequently swapped in and out, which in turn leads to the problem of storage system performance degradation.

[0114] In addition, the method can reuse the existing persistence engine to persist the compressed file after the sub-file is compressed and the dictionary corresponding to the sub-file, without the need to develop a proprietary persistence engine to persist the dictionary. In addition, there is a natural mapping relationship between the compressed file and the dictionary in the persistence file, and there is no need to add additional functions such as persistence and mapping update of mapping metadata, which reduces the impact on the existing storage system architecture, and reduces the development difficulty and system complexity.

[0115] In order to make the technical solution of the present application clearer and easier to understand, the data compression method provided by the present application will be introduced below by taking the storage system of the above embodiment as an example.

[0116] See also Figure 2 A flow chart of a data compression method is shown, the method comprising:

[0117] S202: The storage system receives a write request.

[0118] A write request, also called write input output (IO), is a request for writing data. Specifically, the client can generate a write request in response to a write operation triggered by a user, and the storage system receives the write request sent by the client. The write request includes the data to be written.

[0119] S204: The storage system writes the data in the write request into a log.

[0120] Specifically, during the data writing phase, the storage system can write the data in the write request to the log by appending (Append-only), for example, recording it to the write ahead log (WAL), also known as the pre-write log, and subsequently transfer the data in the log to a persistent storage medium, for example, it can be periodically transferred to the disk, thereby ensuring high data reliability and high availability.

[0121] It should be noted that S204 is an optional step of the embodiment of the present application, and the method of the present application may not execute S204. For example, the storage system may directly refresh the data in the write request to the persistent storage medium.

[0122] S206: The storage system obtains the data file to be persisted from the log.

[0123] The data file to be persisted refers to a data file that needs to be refreshed to a persistent storage medium. The data file to be persisted may include multiple sub-files. Taking a column-based database application as an example, the data file to be persisted may be a database table, which may include multiple table partitions (column clusters). The data in the table partitions (column clusters) may form sub-files in the data file to be persisted.

[0124] Specifically, the storage system can update the log to the memory structure, which refers to the data structure in the memory, such as a memory table (referred to as Memtable). When the memory structure such as Memtable meets the persistence condition, the storage system can trigger persistence. Among them, the persistence condition can be that the capacity of the memory structure reaches the target value, and / or the time from the last data write reaches the target time.

[0125] When the storage system triggers persistence, the data updated by the log to the memory structure can be obtained to obtain the data file to be persisted. In some examples, when the storage system triggers persistence, the current memory table can be set to read-only (or the current memory table can be frozen) to obtain an immutable memory table (immutable Memtable). The data file formed by the data in the immutableMemtable can be a subfile in the data file to be persisted, such as the first subfile.

[0126] Furthermore, the storage system can create a new memory table to update other data in the log to the new memory table. When the new memory table meets the persistence conditions, the new memory table can be set to read-only, and the data in the new memory table forms a sub-file in the data file to be persisted, such as the second sub-file.

[0127] It should be noted that the above S202 to S206 are a specific implementation method for the storage system to obtain the data file to be persisted. In other possible implementation methods of the embodiment of the present application, the data file to be persisted can also be obtained by other methods. For example, the storage system can directly determine the data in the write request as the data file to be persisted. The present application does not limit the method for obtaining the data file to be persisted.

[0128] S208: When persisting the first sub-file in the data file to be persisted, the storage system generates a dictionary corresponding to the first sub-file according to the first sub-file.

[0129] The storage system may generate a dictionary corresponding to the first sub-file based on the first sub-file during the process of transferring the log to a persistent storage medium (such as a log flush disk). The dictionary may be a public dictionary. Specifically, the first sub-file may include one or more data blocks, and the storage system may shard the data blocks to obtain training samples and test samples. The storage system may generate a dictionary based on the training samples, and then perform tests based on the test samples in the test set, for example, by compressing the test samples using the dictionary and then calculating the compression rate. When the compression rate meets the requirements, for example, the compression rate is greater than the target compression rate, the dictionary is determined as the final dictionary. When the compression rate does not meet the requirements, the dictionary is regenerated.

[0130] The following is a detailed description of the dictionary generation process with an example. Figure 3 As shown in the flowchart of dictionary generation, the storage system obtains the data block of the first sub-file, slices the data block, and obtains the training set and the test set. Among them, the storage system can use a variable-length slicing mechanism to slice the data block according to the frequency of the string corresponding to the window in the data block to obtain the slices of the data block. For example, the storage system can determine the strings with high correlation according to the frequency of the strings, divide the strings with high correlation into the same slice, and divide the strings with low correlation into different slices. In this way, different strings in the same slice are regarded as a whole to participate in the subsequent dictionary training process, which improves the efficiency of processing strings and reduces the frequency of loading interference information into the dictionary.

[0131] Furthermore, the training samples in the training set include shards of different categories, such as shards of different languages. The storage system can cluster the shards in the training samples to obtain training sets of different categories. The training set of each category includes the training samples of that category, and the training samples of each category include the shards of that category. The storage system trains the dictionary of each category through the training sets of each category. Accordingly, the storage system can use the dictionaries of different categories to compress the strings of different data types in the data blocks, which can further improve the compression rate and ensure the compression effect.

[0132] The following is an example of a dictionary of one of the categories. The set of slices of the target category can form a training set, such as training set T. The dictionary trained by the training set is denoted as D T . The training process is described below. The storage system can divide the training set T into multiple units, and then set the window according to the window length in the dictionary parameter, slide the window on each unit of the training set T, and determine the score of the window when it slides to different positions. For each unit, when the score of the window when it slides to a certain position meets the set conditions, the storage system can load the string corresponding to the window into the dictionary of the target category, thereby generating a dictionary of the target category.

[0133] In some embodiments, the sizes of the multiple units may be fixed. A string corresponding to a window is selected in each unit, so the storage system can store the string according to the compression dictionary D T The size of the storage system and the window length determine the number of units. Further, the storage system can determine the size of each unit according to the number of units. Specifically, the storage system can divide the units according to the following formula:

[0134]

[0135] Where k represents the window length.

[0136] In specific implementation, the storage system can obtain the scores of different strings in the window according to the frequencies of different strings in the window (for example, the number of units when the string appears in different units can be determined as the frequency of the string). Then, the storage system can obtain the score of the window when it slides to a certain position according to the scores of different strings in the window. Specifically, the storage system can determine the score of the window when it slides to each position according to the following formula:

[0137]

[0138] The window whose score meets the set condition may be the window with the highest score, or the window with a score greater than a preset score. This embodiment uses the window with the highest score as an example. The storage system may output the string corresponding to the window with the highest score in each unit to the dictionary D T middle.

[0139] Similarly, the storage system can generate dictionaries of other categories based on training sets of other categories. The storage system can merge dictionaries of various categories.

[0140] The storage system can use the trained dictionary to compress the test samples, and count the compression rate and compression time of each category, and then optimize the dictionary based on the compression rate and compression time. For example, the storage system can reduce the dictionary to obtain a final dictionary when the compression rate drops to an acceptable range (for example, greater than or equal to the minimum compression rate). When reducing the segments, the storage system can select segments with smaller compression rate contributions for reduction. For example, the storage system can reduce segments whose compression rate contributions are less than a first preset value. For another example, the storage system can add segments to the dictionary to obtain a final dictionary. Specifically, the dictionary can increase the segments whose compression rate contributions are greater than a second preset value, specifically by loading more character strings to improve the overall compression rate.

[0141] If a failure occurs during the process of generating the dictionary, since the log is still there, the storage system can replay the data according to the log, thereby regenerating the dictionary corresponding to the first sub-file.

[0142] It should be noted that when persisting the second sub-file in the persistent data file, the storage system can generate a dictionary corresponding to the second sub-file based on the second sub-file. The specific implementation of the storage system generating a dictionary corresponding to the second sub-file based on the second sub-file can refer to the description of generating a dictionary corresponding to the first sub-file based on the first sub-file, which will not be repeated here.

[0143] S209: The storage system compresses the first sub-file using the dictionary to obtain a first compressed file.

[0144] S210: The storage system generates a first persistent file according to the first compressed file and a dictionary corresponding to the first sub-file.

[0145] The first persistent file includes a dictionary corresponding to the first subfile and a first compressed file. Similarly, the storage system can compress the second subfile according to the dictionary corresponding to the second subfile to obtain a second compressed file, and then generate a second persistent file according to the second compressed file and the dictionary corresponding to the second subfile. The second persistent file includes a dictionary corresponding to the second subfile and a second compressed file.

[0146] In some possible implementations, persistent files can be pre-partitioned. Figure 3 A partition diagram of a persistent file, the persistent file includes a pre-allocated data area (data area) and a metadata area (meta area). The data area is used to store compressed files, and the metadata area is used to store metadata including dictionaries. In order to facilitate the rapid and accurate acquisition of dictionaries when reading data subsequently, dictionaries can usually be stored in a specified location in the metadata area. The specified location can be determined based on the configuration information of the pre-allocated metadata area. The configuration information of the pre-allocated metadata area includes the offset of the metadata area in the persistent file and the offset of the dictionary in the metadata area.

[0147] exist Figure 3 In the example, the dictionary can be stored in the metadata area in the form of a dictionary block. The metadata area can also store other metadata besides the dictionary, such as index metadata, and the index metadata can be stored in the metadata area in the form of an index block. Taking the first persistent file as an example, the data area of ​​the first persistent file stores the first compressed file, and the metadata area of ​​the first persistent file stores the dictionary corresponding to the first sub-file. For example, the starting address of the dictionary storage can be determined based on the first address of the first persistent file plus the offset of the metadata area in the first persistent file and the offset of the dictionary in the metadata area, and the storage system can store the dictionary at the location of the starting address (the specified location of the metadata area).

[0148] Furthermore, if Figure 3 As shown, the persistent file may also include a header data area (head area). The head area stores footer metadata, including but not limited to version number, file length or statistical information.

[0149] The storage system can adopt a separate space reservation strategy to divide the persistent file into a data area and a meta area (further, a head area can also be divided), and complete the space reservation in advance when generating the persistent file. After the dictionary is generated, the corresponding space reserved in the meta area is used to pre-fill the dictionary (that is, fill the dictionary into the reserved meta area).

[0150] Wherein, the storage system may receive the configuration information of the partition, and the configuration information of the partition includes the configuration information of the data area and the configuration information of the meta area. Wherein, the configuration information of the data area may include the offset (or offset address) of the data area in the persistent file, and the configuration information of the meta area may include the offset (or offset address) of the meta area in the persistent file. Accordingly, when generating the persistent file, the persistent file may be partitioned according to the above offset. Wherein, the meta area may be further partitioned, for example, the configuration information of the meta area may also include the offset of the dictionary in the meta area, so that the meta area may be further partitioned, so as to store the dictionary at the specified position of the meta area. Accordingly, when reading the dictionary, the storage system may read the dictionary from the specified position of the meta area of ​​the persistent file based on the first address of the persistent file, the offset of the meta area in the persistent file, and the offset of the dictionary in the meta area, so as to realize efficient reading of the dictionary and improve the reading performance. Similarly, the storage system may read the compressed file from the data area of ​​the persistent file based on the first address of the persistent file and the offset of the data area in the persistent file, thereby realizing efficient data reading and improving the reading performance.

[0151] In some possible implementations, a head area may be further divided in the persistent file. In some examples, the storage system may determine the area between the first address and the offset address of the data area as the head area. In other examples, the partition configuration information may further include configuration information of the head area, and the configuration information of the head area may include the offset of the head area in the persistent file.

[0152] Among them, the head area can store the offset of the data area in the persistent file and the offset of the meta area in the persistent file. When performing a read operation on the persistent file, the data area can be determined according to the offset of the data area in the persistent file stored in the head area, and the compressed file can be read from the data area, or the meta area can be determined according to the offset of the meta area in the persistent file stored in the head area, and the metadata can be read from the meta area. The head area of ​​the persistent file can also store the offset of the dictionary in the meta area, so that the dictionary can be accurately read from the specified position in the meta area. Since information such as offsets is configurable, in order to avoid affecting other partitions by modifying the format of the meta area, a head area can also be set in the meta area. For example, the meta area can be divided into a head area, a dictionary area, and an index area. Among them, the head area of ​​the meta area can store the offset of other partitions in the meta area (such as the dictionary area) in the meta area.

[0153] S212: The storage system writes the first persistent file to the persistent storage medium.

[0154] The persistent storage medium may be a hard disk, including but not limited to a mechanical hard disk HDD or a solid-state disk SSD. The storage system may write a first persistent file to the persistent storage medium. In other words, the first compressed file obtained by compressing the first sub-file and the dictionary corresponding to the first sub-file are stored in the persistent storage medium in the form of a first persistent file, thereby achieving persistence of data and metadata including dictionaries. Similarly, the storage system may also write a second persistent file to the persistent storage medium. In other words, the second compressed file obtained by compressing the second sub-file and the dictionary corresponding to the second sub-file are stored in the persistent storage medium in the form of a second persistent file.

[0155] Furthermore, when the persistence of the first sub-file fails, the storage system can also restore the first sub-file based on the log. For example, the storage system can replay the log to restore the first sub-file, so that the first sub-file can be recompressed to obtain the first compressed file, and the first compressed file and the dictionary corresponding to the first sub-file can be rewritten into the persistent storage medium, thereby achieving data and metadata persistence.

[0156] In some possible implementations, the storage system may also receive a read request for reading the first sub-file. Accordingly, the storage system may read the dictionary corresponding to the first sub-file from the specified position of the metadata area of ​​the first persistent file according to the configuration information of the pre-allocated meta area, read the first compressed file from the data area of ​​the first persistent file according to the configuration information of the pre-allocated data area, and then decompress the first compressed file using the dictionary corresponding to the first sub-file to obtain the first sub-file. Considering the read performance, the storage system may also load the dictionary into the cache to improve the decompression efficiency. During the decompression process, the storage system can directly find the unique dictionary corresponding to the sub-file in the data file, thereby achieving zero-bite between the data and the dictionary.

[0157] Based on the above description, the data compression method of the embodiment of the present application generates a dictionary corresponding to the sub-file in real time when persisting the sub-file in the data file to be persisted, and persists the compressed file obtained by compressing the sub-file using the dictionary and the dictionary corresponding to the sub-file together, without providing a dedicated persistence engine responsible for the persistence of the dictionary, and without adding mapping metadata persistence and mapping update functions to the metadata persistence engine, thereby reducing the development difficulty and system complexity. Moreover, the persistent file reserves a metadata area to pre-fill the dictionary, which can achieve non-intrusive metadata and zero-overhead maintenance.

[0158] For storage systems that use append-write mode, data that has been persisted can be rewritten. In other words, invalid data or junk data can be generated in persistent files. Invalid data and junk data refer to data that has been rewritten, such as data that has been rewritten in a subfile corresponding to a compressed file stored in the data area of ​​a persistent file. In contrast to invalid data is valid data, which refers to data that has not been rewritten, such as data that has not been rewritten in a subfile corresponding to a compressed file stored in the data area of ​​a persistent file.

[0159] As data is written and overwritten, the effective data ratio of persistent files gradually decreases, and the storage system can recycle to free up space. Recycling persistent files can be done by merging persistent files to generate new persistent files that include effective data, and regenerating a new dictionary that conforms to the characteristics of effective data in the new persistent file, ensuring that data and dictionaries have a high affinity throughout the entire cycle.

[0160] In some possible implementations, the storage system may set a merge condition. The merge condition may be set according to business requirements. For example, in order to save storage space, the merge condition may be set to a condition related to the effective data ratio. For another example, in order to reduce resource waste, the merge condition may be set to a condition related to the load so that the merge can be performed when idle. Based on this, the merge condition may include one or more of the following:

[0161] ① The interval between the current time and the last merge time reaches the merge cycle; ② The current load is less than the preset value; ③ The proportion of valid data of at least one of the multiple persistent files written to the persistent storage medium is less than the preset proportion, where the valid data is the data that has not been rewritten; ④ The merge request reported when the user triggers the merge operation is detected.

[0162] In specific implementation, the storage system can detect whether the merge condition is met. For example, the storage system can set a timer when the persistent file is merged last time. When the timer reaches the merge period, it is determined that the interval between the current time and the last merge time reaches the merge period. In other words, the storage system can determine that the merge condition ① is met.

[0163] When the merging condition is met, the storage system may merge multiple persistent files written to the persistent storage medium to obtain a new persistent file, and the storage system deletes the merged multiple persistent files (also referred to as old persistent files) to reclaim space.

[0164] In some possible implementations, the storage system may determine multiple persistent files to be merged according to the attribute information of the persistent files written into the persistent storage medium, and merge the multiple persistent files to obtain a new persistent file.

[0165] In some examples, the attribute information of the persistent file may include a write time. The storage system may sort the persistent files written to the persistent storage medium according to the write time of the persistent files written to the persistent storage medium, and determine the multiple persistent files to be merged according to the sorting result. For example, the storage system may determine the multiple persistent files to be merged according to the first-in-first-out principle, based on which the storage system may determine the persistent files with the top sorting results as the multiple persistent files to be merged. Among them, when the number of persistent files written to the persistent storage medium is 2n or 2n+1, the storage system may determine the persistent file with the sorting result of top i (1<i<n) as the persistent file to be merged.

[0166] In other examples, the attribute information of the persistent file may include a valid data ratio. The storage system may determine the multiple persistent files to be merged according to the valid data ratio of the persistent files written to the persistent storage medium, and the multiple persistent files to be merged include persistent files whose valid data ratio is less than a preset ratio. The preset ratio may be set according to an empirical value, for example, may be set to 50%.

[0167] The persistent file merging process is described below. Specifically, the storage system can decompress multiple persistent files (for example, the multiple persistent files to be merged determined above) that have been written to the persistent storage medium, obtain valid data from the decompressed sub-files respectively and merge the valid data to obtain a merged sub-file. The storage system can then generate a dictionary corresponding to the merged sub-file based on the merged sub-file, and compress the merged sub-file using the dictionary corresponding to the merged sub-file to obtain a merged compressed file. The storage system can then generate a new persistent file based on the merged compressed file and the dictionary corresponding to the merged sub-file.

[0168] Among them, the storage system generates a dictionary corresponding to the merged subfile, compresses the merged subfile using the dictionary corresponding to the merged subfile, and generates a new persistent file according to the merged compressed file and the dictionary corresponding to the merged subfile. The process can refer to the persistence process of the first subfile in the multiple subfiles by the storage system, which will not be repeated here. Considering that after the multiple merged persistent files are deleted, if the storage system still reads the data according to the original address mapping relationship, it may cause a read failure. For this reason, the storage system can also update the physical address in the address mapping relationship of the multiple persistent files to the physical address of the new persistent file. When a read request is received, the storage request can be indexed to the physical address of the new persistent file through the updated address mapping relationship according to the logical address (logic block address, LBA) in the read request. For example, the address mapping relationship of multiple persistent files can be LBA1→PBA1, LBA2→PBA2. When multiple persistent files are merged into a new persistent file, the address mapping relationship of multiple persistent files can be updated to LBA1→PBA3, LBA2-PBA3.

[0169] The following is an example to illustrate the persistent file merging process and the persistent file reading process.

[0170] See also Figure 4 A flow chart of persistent file merging and reading is shown. In this example, persistent files are stored in a structured merge tree (Log Structured Merge Tree, LSM Tree). When the merge conditions are met, multiple persistent files in the LSM tree can be merged. For example, if the effective data ratio of the persistent files in the L0 layer is less than 50%, the persistent files in the L0 layer can be merged to generate a new persistent file and written to the L1 layer. When the effective data ratio of the persistent files in the L1 layer is less than 50%, the persistent files in the L1 layer can be merged to generate a new persistent file and written to the L2 layer.

[0171] Since persistent files are divided into the head area, data area, and meta area, when merging persistent files, the storage system not only obtains compressed files from the data area, decompresses the compressed files using the dictionary in the meta area, extracts valid data from the decompressed files, and merges the valid data to obtain merged sub-files, but also regenerates a dictionary that meets the characteristics of the merged sub-files based on the merged sub-files. A new persistent file can be generated based on the merged sub-files and the regenerated dictionary.

[0172] In this example, when reading persistent files, the storage system can accurately pre-fetch metadata through a metadata cache such as MetaCache. For example, the storage system can accurately pre-fetch dictionary 1 (denoted as dictionary1, dict1 for short) from persistent file 1 (denoted as file1), and accurately pre-fetch dictionary 2 (denoted as dictionary2, dict2 for short) from persistent file 2 (denoted as file2). This can improve the MetaCache hit rate and maximize system performance.

[0173] In this embodiment, when a sub-file in a data file is persisted (downloaded to disk), the sub-file and the dictionary are persisted and recycled at the same time, thus achieving lifecycle affinity binding between the data (e.g., the sub-file in the data file) and the dictionary, zero overlap between the data and the dictionary, no metadata amplification, and avoiding performance issues caused by metadata overflow.

[0174] In order to make the technical solution of the present application clearer and easier to understand, the data compression method of the present application is introduced below in conjunction with specific application scenarios.

[0175] See also Figure 5 A schematic diagram of a data compression method in a database is shown in FIG. Figure 5 As shown in the figure, the database storage engine includes the following components: WAL files that play a role similar to transaction logs, memory structures Memtable, and persistent static sorted table (SST) files. Data is first recorded in the log, then updated to the memory structure Memtable. After the capacity / duration conditions are met, the persistence (Minor Compaction) operation can be triggered and persisted in the SST file. The SST file is divided into the Head area, Data area, and Meta area. The WAL file can be used to recover the Memtable that has not been written to the disk in the event of a failure.

[0176] Specifically, when the storage system receives a write request from the upper layer, it can write-protect the data in the write request based on the WAL write protection mechanism provided in the LogStructured Merge Tree (LSM) layered architecture. The write protection process can be that the storage system writes the data to the log, such as writing it to a WAL file, and then writes it to the memory structure Memtable, then serializes it, and persists the serialized data to the disk. Figure 5 As shown in step ①, when a failure occurs, the storage system can replay the WAL file to restore the data and make it persistent again.

[0177] like Figure 5 As shown in step ②, when persisting, the storage system can pre-set dictionary filling space (pre-partition) in the Meta area of ​​the SST file, so that no additional metadata management overhead is introduced. Based on the current unified metadata service, efficient and highly reliable metadata management can be achieved, avoiding the repeated development workload introduced by redeveloping the metadata persistence engine, and avoiding the complexity brought by the management of multiple types of metadata.

[0178] In addition, to address the metadata overflow problem caused by the long tail of the dictionary, this embodiment affinity binds the life cycle of data (such as file objects such as the first sub-file) and the dictionary (such as the dictionary corresponding to the first sub-file). Specifically, a single SST file independently maintains a dictionary, and the dictionary scope is naturally isolated between SST files. At the same time, the database synchronously updates the dictionary during the recycling and merging (Major Compaction) process, which can ensure zero overlap between data and dictionary, no metadata amplification, and avoid performance degradation caused by metadata overflow.

[0179] For upper-layer read requests, see Figure 5 In step ③, the storage system can combine the fixed index information of each SST file for the dictionary space to perform accurate pre-fetching, improve the cache hit rate, and provide subsequent read request hits, which greatly reduces the IO path and improves the read performance.

[0180] See also Figure 6 A schematic diagram of a data compression method in a distributed storage system is shown, and the distributed storage system may be a Ceph system with storage and computing separation, and the Ceph system may be logically divided into a cache layer and a storage layer. The cache layer may be a global cache (GC), which may specifically include a cache foreground and a cache background. The storage layer may include a Ceph cluster, which includes a hard disk.

[0181] Specifically, the client can deploy client plug-ins such as Librados to accelerate the reading and writing of the Ceph system. Librados provides a client adapter, which is used to perform semantic conversion (also called escape) so that the server can identify read IO and write IO. The cache front end includes a server adapter, which is used to perform semantic conversion, send write IO to the write cache, and send read IO to the read cache (including L1 read cache or L2 read cache). The index system in the cache background performs public dictionary compression on the sub-files in the data file, and writes them to the persistence log (plog) after IO aggregation. In order to solve the mapping management problem between data and the public dictionary, in this embodiment, the public dictionary is persisted along with the data (for example, the first subfile in the data file to be persisted). The implementation of plog append writing requires that the public dictionary be aggregated at the tail. The native plog design requires distributed negotiation of the write position of the public dictionary, and the negotiated position is monotonically increasing. In this embodiment, the plog format can be modified to pre-partition the plog. Specifically, the plog is divided into a data area, a meta area, and a header area, and the dictionary is pre-filled in the meta area, so that the public dictionary is persisted along with the data.

[0182] See also Figure 7 The traditional method requires the development of a proprietary dictionary persistence engine for dictionary persistence, which makes the storage system architecture complex and introduces additional development workload. In addition, the dictionary and data are usually a one-to-many mapping relationship. When the data is queried for the first time, the corresponding metadata file needs to be loaded into the memory and cached. As long as the table data is not completely deleted, the dictionary cannot be released. This operation will inevitably have a long-tail problem in which the dictionary occupies the memory for a long time and cannot be released, resulting in metadata expansion, which leads to a sharp increase in the occupied memory, resulting in dictionary data swapping in and out, and the dictionary cannot be hit in the memory, resulting in performance degradation.

[0183] The data compression method of the present application adopts a separate space reservation strategy, divides the persistent file into a head area, a data area, a meta area and other partitions, and completes the space reservation in advance while generating the file object, so that after the dictionary is generated, the corresponding space is reserved in the meta area to pre-fill the dictionary (fill the dictionary into the reserved meta area), and the filling space is fully reserved and configurable.

[0184] Moreover, in order to solve the problem of public dictionary management and persistence, in the data compression method of this application, the storage system can first write logs (such as WAL), then update the memory, and finally serialize and persist the data to the disk, so that if a failure occurs during the WAL writing process, the data can be replayed and rewritten, and the dictionary can be regenerated in this process. Since the failure scenario accounts for a small proportion (daily failure rate 0.1%), failure retry will not affect performance.

[0185] To address the problem of metadata expansion and overflow, a single persistent file independently maintains a dictionary, and the two can correspond one-to-one. The life cycle of data and dictionary are affinity-bound. When the persistent file is recycled or merged, the dictionary is also updated synchronously, thereby achieving a one-to-one mapping for the entire life cycle. Data and dictionary are persisted and recycled at the same time, avoiding metadata expansion and overflow.

[0186] Based on the data compression method of the above embodiment, the present application also provides a data compression device. The device can be deployed in Figures 1A to 1D The storage system shown in the figure runs the data compression device to perform Figure 2 The data compression method shown.

[0187] The data cache device provided by the present application is introduced below from the perspective of functional modularization in combination with the embodiments.

[0188] See also Figure 8 The data compression device 800 is a schematic structural diagram of a data compression device, wherein the data compression device 800 comprises:

[0189] An acquisition module 802 is used to acquire a data file to be persisted, wherein the data file to be persisted includes a plurality of sub-files;

[0190] The persistence module 804 is configured to: when persisting the first sub-file among the multiple sub-files:

[0191] generating a dictionary corresponding to the first subfile according to the first subfile, and compressing the first subfile using the dictionary to obtain a first compressed file;

[0192] generating a first persistent file according to the first compressed file and the dictionary; and

[0193] The first persistent file is written to the persistent storage medium.

[0194] The acquisition module 802 and the persistence module 804 can be implemented by software. Figure 2In the embodiment shown, the method steps S202 to S206 are used to obtain the data file to be persisted. The persistence module 804 can be used to execute the method steps S208 to S212. The specific implementation process of the acquisition module 802 and the persistence module 804 can be found in Figure 2 The description of the related contents of the illustrated embodiment will not be repeated here.

[0195] In some possible implementations, the first persistent file includes a pre-allocated data area and a metadata area, the data area stores the first compressed file, a specified position of the metadata area stores a dictionary corresponding to the first sub-file, the specified position is determined based on configuration information for pre-allocating the metadata area, the configuration information for pre-allocating the metadata area includes the offset of the metadata area in the first persistent file and the offset of the dictionary in the metadata area.

[0196] The pre-partitioning of the first persistent file may be implemented when the first persistent file is generated. The partitioning method of the first persistent file may refer to the description of the relevant contents of S210, which will not be described in detail here.

[0197] In some possible implementations, the acquisition module 802 is further configured to:

[0198] receiving a read request, the read request being used to read the first sub-file;

[0199] The device 800 further includes:

[0200] The data reading module 806 is used to read the dictionary corresponding to the first sub-file from the specified position of the metadata area of ​​the first persistent file according to the configuration information of pre-allocating the metadata area, and read the first compressed file from the data area of ​​the first persistent file according to the configuration information of pre-allocating the data area, the configuration information of pre-allocating the data area including the offset of the data area in the first persistent file; and decompress the first compressed file using the dictionary corresponding to the first sub-file to obtain the first sub-file.

[0201] The specific implementation of the data reading module 806 can be found in Figure 2 The description of the related contents of the illustrated embodiment will not be repeated here.

[0202] In some possible implementations, the apparatus 800 further includes:

[0203] A detection module 808, used to detect whether the merging condition is met;

[0204] The merging module 810 is configured to merge the multiple persistent files written into the persistent storage medium to obtain a new persistent file and delete the multiple persistent files when the merging condition is met.

[0205] The specific implementation of the detection module 808 and the merging module 810 can refer to Figure 4 The description of the related contents of the illustrated embodiment will not be repeated here.

[0206] In some possible implementations, the merging module 810 is specifically configured to:

[0207] Decompressing multiple persistent files written to the persistent storage medium, respectively acquiring valid data from the decompressed sub-files and merging the valid data to obtain a merged sub-file, wherein the valid data is data that has not been rewritten;

[0208] Generate a dictionary corresponding to the merged sub-file according to the merged sub-file, and compress the merged sub-file using the dictionary corresponding to the merged sub-file to obtain a merged compressed file;

[0209] A new persistent file is generated according to the merged compressed file and the dictionary corresponding to the merged sub-file.

[0210] In some possible implementations, the merging condition includes at least one of the following:

[0211] The interval between the current time and the last merge time reaches the merge period; or,

[0212] The current load is less than the preset value; or,

[0213] The effective data ratio of at least one persistent file among the multiple persistent files written into the persistent storage medium is less than a preset ratio, where the effective data ratio is the ratio of the data that has not been rewritten in the subfile corresponding to the persistent file; or

[0214] A merge request reported when a user-triggered merge operation is detected.

[0215] In some possible implementations, the merging module 810 is specifically configured to:

[0216] Determining multiple persistent files to be merged according to the attribute information of the persistent files written into the persistent storage medium;

[0217] The multiple persistent files are merged to obtain a new persistent file.

[0218] In some possible implementations, the attribute information of the persistent file includes a write time or a valid data ratio, where the valid data ratio is a ratio of data that has not been rewritten in a subfile corresponding to the persistent file;

[0219] The merging module 810 is specifically used for:

[0220] sorting the persistent files written into the persistent storage medium according to the writing time of the persistent files written into the persistent storage medium, and determining the plurality of persistent files to be merged according to the sorting result; or,

[0221] According to the effective data ratio of the persistent files written into the persistent storage medium, a plurality of persistent files to be merged are determined, wherein the plurality of persistent files to be merged include persistent files whose effective data ratio is less than a preset ratio.

[0222] In some possible implementations, the apparatus 800 further includes:

[0223] The address mapping update module 812 is used to update the physical address in the address mapping relationship of multiple persistent files to the physical address of the new persistent file.

[0224] The specific implementation of the address mapping update module 812 can be referenced Figure 4 The description of the related contents of the illustrated embodiment will not be repeated here.

[0225] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center that contains one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions that instruct the storage system to perform the aforementioned data compression method.

[0226] The embodiment of the present application also provides a computer program product including instructions. The computer program product may be a software or program product including instructions that can be run on a storage system or stored in any available medium. When the computer program product is run on a storage system, the storage system executes the above data compression method.

[0227] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data compression method, characterized in that: When persisting the first sub-file, the method includes: generating a dictionary corresponding to the first sub-file according to the first sub-file, where the first sub-file is a data file to be persisted; Compressing the first sub-file using the dictionary to obtain a first compressed file; A first persistent file is generated according to the first compressed file and the dictionary, and the first persistent file is written into a persistent storage medium.

2. The method according to claim 1, characterized in that The first persistent file includes a pre-allocated data area and a metadata area, the data area stores the first compressed file, a designated position of the metadata area stores a dictionary corresponding to the first sub-file, the designated position is determined according to configuration information of pre-allocating the metadata area, the configuration information of pre-allocating the metadata area includes an offset of the metadata area in the first persistent file and an offset of the dictionary in the metadata area.

3. The method according to claim 2, characterized in that The method further comprises: receiving a read request, the read request being used to read the first sub-file; Reading a dictionary corresponding to the first subfile from a specified position of the metadata area of ​​the first persistent file according to the configuration information of pre-allocating the metadata area, and reading the first compressed file from the data area of ​​the first persistent file according to the configuration information of pre-allocating the data area, wherein the configuration information of pre-allocating the data area includes an offset of the data area in the first persistent file; The first compressed file is decompressed using a dictionary corresponding to the first subfile to obtain the first subfile.

4. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: Check whether the merge conditions are met; When the merging condition is met, the multiple persistent files written into the persistent storage medium are merged to obtain a new persistent file, and the multiple persistent files are deleted.

5. The method according to claim 4, characterized in that The step of merging the multiple persistent files written into the persistent storage medium to obtain a new persistent file includes: Decompressing multiple persistent files written to the persistent storage medium, respectively acquiring valid data from the decompressed sub-files and merging the valid data to obtain a merged sub-file, wherein the valid data is data that has not been rewritten; generating a dictionary corresponding to the merged sub-file according to the merged sub-file, and compressing the merged sub-file using the dictionary corresponding to the merged sub-file to obtain a merged compressed file; A new persistent file is generated according to the merged compressed file and the dictionary corresponding to the merged sub-file.

6. The method according to claim 4, characterized in that The merging conditions include at least one of the following: The interval between the current time and the last merge time reaches the merge period; or, The current load is less than the preset value; or, The effective data ratio of at least one persistent file among the multiple persistent files written into the persistent storage medium is less than a preset ratio, where the effective data ratio is the ratio of the data that has not been rewritten in the subfile corresponding to the persistent file; or A merge request reported when a user-triggered merge operation is detected.

7. The method according to any one of claims 4 to 6, characterized in that: The step of merging the multiple persistent files written into the persistent storage medium to obtain a new persistent file includes: Determining multiple persistent files to be merged according to the attribute information of the persistent files written into the persistent storage medium; The multiple persistent files are merged to obtain a new persistent file.

8. The method according to claim 7, characterized in that The attribute information of the persistent file includes the writing time or the effective data ratio, where the effective data ratio is the ratio of the data that has not been rewritten in the sub-file corresponding to the persistent file; The step of determining the plurality of persistent files to be merged according to the attribute information of the persistent files written into the persistent storage medium comprises: sorting the persistent files written into the persistent storage medium according to the writing time of the persistent files written into the persistent storage medium, and determining a plurality of persistent files to be merged according to the sorting result; or, According to the effective data ratio of the persistent files written into the persistent storage medium, a plurality of persistent files to be merged are determined, wherein the plurality of persistent files to be merged include persistent files whose effective data ratio is less than a preset ratio.

9. The method according to any one of claims 4 to 8, characterized in that: The method further comprises: The physical addresses in the address mapping relationship of the multiple persistent files are updated to the physical addresses of the new persistent file.

10. A data compression device, characterized in that: The device comprises: The persistence module is used to persist the first sub-file: generating a dictionary corresponding to the first sub-file according to the first sub-file, where the first sub-file is a data file to be persisted; Compressing the first sub-file using the dictionary to obtain a first compressed file; A first persistent file is generated according to the first compressed file and the dictionary, and the first persistent file is written into a persistent storage medium.

11. The device according to claim 10, characterized in that The first persistent file includes a pre-allocated data area and a metadata area, the data area stores the first compressed file, a designated position of the metadata area stores a dictionary corresponding to the first sub-file, the designated position is determined according to configuration information of pre-allocating the metadata area, the configuration information of pre-allocating the metadata area includes an offset of the metadata area in the first persistent file and an offset of the dictionary in the metadata area.

12. The device according to claim 11, characterized in that The acquisition module is also used for: receiving a read request, the read request being used to read the first sub-file; The device also includes: a data reading module, configured to read a dictionary corresponding to the first sub-file from a specified position of the metadata area of ​​the first persistent file according to configuration information for pre-allocating the metadata area, and read the first compressed file from the data area of ​​the first persistent file according to configuration information for pre-allocating the data area, wherein the configuration information for pre-allocating the data area includes an offset of the data area in the first persistent file; The first compressed file is decompressed using a dictionary corresponding to the first subfile to obtain the first subfile.

13. The device according to any one of claims 10 to 12, characterized in that The device also includes: A detection module, used to detect whether the merging condition is met; The merging module is used to merge the multiple persistent files written into the persistent storage medium to obtain a new persistent file and delete the multiple persistent files when the merging condition is met.

14. The device according to claim 13, characterized in that The merging module is specifically used for: Decompressing multiple persistent files written to the persistent storage medium, respectively acquiring valid data from the decompressed sub-files and merging the valid data to obtain a merged sub-file, wherein the valid data is data that has not been rewritten; generating a dictionary corresponding to the merged sub-file according to the merged sub-file, and compressing the merged sub-file using the dictionary corresponding to the merged sub-file to obtain a merged compressed file; A new persistent file is generated according to the merged compressed file and the dictionary corresponding to the merged sub-file.

15. The device according to claim 13, characterized in that The merging conditions include at least one of the following: The interval between the current time and the last merge time reaches the merge period; or, The current load is less than the preset value; or, The effective data ratio of at least one persistent file among the multiple persistent files written into the persistent storage medium is less than a preset ratio, where the effective data ratio is the ratio of the data that has not been rewritten in the subfile corresponding to the persistent file; or A merge request reported when a user-triggered merge operation is detected.

16. The device according to any one of claims 13 to 15, characterized in that The merging module is specifically used for: Determining multiple persistent files to be merged according to the attribute information of the persistent files written into the persistent storage medium; The multiple persistent files are merged to obtain a new persistent file.

17. The device according to claim 16, characterized in that The attribute information of the persistent file includes the writing time or the effective data ratio, where the effective data ratio is the ratio of the data that has not been rewritten in the sub-file corresponding to the persistent file; The merging module is specifically used for: sorting the persistent files written into the persistent storage medium according to the writing time of the persistent files written into the persistent storage medium, and determining a plurality of persistent files to be merged according to the sorting result; or, According to the effective data ratio of the persistent files written into the persistent storage medium, a plurality of persistent files to be merged are determined, wherein the plurality of persistent files to be merged include persistent files whose effective data ratio is less than a preset ratio.

18. The device according to any one of claims 13 to 17, characterized in that The device also includes: The address mapping update module is used to update the physical address in the address mapping relationship of multiple persistent files to the physical address of the new persistent file.

19. A storage system, characterized in that: The storage system comprises a controller and a hard disk, wherein the controller executes computer-readable instructions so that the storage system performs the method according to any one of claims 1 to 9.

20. A computer-readable storage medium, characterized in that: The method comprises computer-readable instructions; the computer-readable instructions are used to implement the method according to any one of claims 1 to 9.

Images