A data compression method and related system

Abstract

The present application provides a data compression method, including: obtaining a data file to be persisted, the data file to be persisted includes multiple sub-files, and when persisting the first sub-file in the multiple sub-files: generating a dictionary corresponding to the first sub-file based on the first sub-file, and compressing the first sub-file using the dictionary to obtain a first compressed file, generating a first persistent file based on the first compressed file and the dictionary, and writing the first persistent file to a persistent storage medium. A single persistent file independently maintains the dictionary corresponding to the sub-file, and the dictionary and the sub-file can implement the same lifecycle management, that is, the lifecycle of the data in the sub-file and the dictionary are affinity-bound, solving the long-tail problem of local data files referencing the global public dictionary, resulting in the public dictionary being unable to be released. This avoids the public dictionary occupying memory for a long time and being unable to be released, causing metadata expansion, and then causing the storage system performance to degrade.

Description

[0001] This application is a divisional application. The original application has application number 202310803573.7 and an original filing date of June 30, 2023. The entire contents of the original application are incorporated herein by reference. This application and the Chinese application with application number 202310803573.7 both claim priority to Chinese patent application number 202310473699.2, filed on April 26, 2023, entitled “A Data Compression Method,” the entire contents of which are incorporated herein by reference. Technical Field

[0002] The present application relates to the field of storage technology, and in particular to a data compression method, device, storage system, computer-readable storage medium, and computer program product. Background Art

[0003] The widespread adoption of cloud computing, fifth-generation (5G) communications systems, and big data are accelerating data production and flow, unlocking the value of data and posing greater challenges to data storage capacity. Larger data volumes and longer storage cycles translate into higher storage costs, placing greater demands on data-centric infrastructure and memory-centric system architectures for greater cost-effectiveness. Addressing the issue of massive data storage has become a key factor impacting the procurement and maintenance costs of data center infrastructure.

[0004] All major data storage vendors have introduced corresponding data reduction features, significantly increasing user available capacity for the same physical capacity and reducing total cost of ownership (TCO). Currently, the industry has introduced data compression algorithms such as LZ4, ZSTD, and Snappy to implement data reduction features. These algorithms can provide data compression ratios of up to several times the original rate. However, due to the exponential growth of data volumes, these algorithms are gradually failing to fully meet the production, storage, and application needs of massive data.

[0005] Data storage vendors are attempting to use Public Dictionary (PD) compression to break through the limitations of compression algorithms like LZ4 and Snappy. Public dictionary compression can further improve data reduction rates by mining a wider range of richer data features. However, public dictionaries provide global information independent of user data files. Different data files (e.g., data files corresponding to different columns in a database table) can reference the same public dictionary. The difference in the lifecycles of local data files and the global public dictionary leads to a long-tail problem where the public dictionary occupies memory for a long time and cannot be released. The memory occupied by the public dictionary increases dramatically, affecting the performance of the storage system. Summary of the Invention

[0006] This application provides a data compression method that independently maintains the dictionary corresponding to a subfile in a single persistent file. This method allows the dictionary and the subfile to implement the same lifecycle management. Specifically, the lifecycles of the data in the subfile and the dictionary are affinity-bound. This method solves the long-tail problem of local data files referencing a global public dictionary, preventing the public dictionary from being released, and ensures the performance of the storage system. This application also provides a device, storage system, computer-readable storage medium, and computer program product corresponding to the above method.

[0007] In a first aspect, the present application provides a data compression method. The method can be performed by a storage system. The storage system can be a centralized storage system, including a centralized storage system with integrated disk and controller, or a centralized storage system with separate disk and controller. Alternatively, the storage system can be a distributed storage system, including but not limited to a distributed storage system with integrated storage and computing, or a distributed storage system with separate storage and computing.

[0008] Specifically, the storage system obtains a data file to be persisted, which includes multiple sub-files. When persisting the first sub-file among the multiple sub-files: a dictionary corresponding to the first sub-file is generated based on the first sub-file, and the dictionary is used to compress the first sub-file to obtain a first compressed file, and then a first persistent file is generated based on the first compressed file and the dictionary, and the first persistent file is written to a persistent storage medium.

[0009] In this method, the storage system persists the dictionary and the compressed file obtained by compressing the sub-file using the dictionary in the same persistent file. The dictionary and the sub-file can implement the same lifecycle management, that is, the data in the sub-file and the lifecycle of the dictionary are affinity-bound, which solves the long-tail problem of local data files referencing the global public dictionary, resulting in the public dictionary being unable to be released. This avoids the public dictionary occupying the memory for a long time and being unable to be released, causing metadata expansion (metadata overflow), resulting in the public dictionary that needs to be used not being able to hit the memory, and the need for frequent swapping in and out, which in turn leads to the problem of storage system performance degradation.

[0010] In some possible implementations, the first persistent file includes a pre-allocated data area and a metadata area. The data area stores the first compressed file, and a dictionary corresponding to the first sub-file is stored at a designated location in the metadata area. The designated location is determined based on configuration information for pre-allocating the metadata area. The configuration information for pre-allocating the metadata area includes an offset of the metadata area in the first persistent file and an offset of the dictionary in the metadata area.

[0011] This method uses a separate space reservation strategy, dividing the persistent file into a data area and a metadata area. Space reservation is completed in advance when the persistent file is generated. After the dictionary is generated, the corresponding space reserved in the metadata area is used to pre-populate the dictionary (that is, the dictionary is filled into the reserved metadata area). The data area and metadata area have a natural mapping relationship, which enables non-intrusive metadata and zero-overhead maintenance. There is no need to develop additional metadata persistence and mapping update functions, thus avoiding impact on the existing storage system architecture and reducing development difficulty and system complexity.

[0012] In some possible implementations, the storage system may receive a read request for reading a first sub-file, and then the storage system may read a dictionary corresponding to the first sub-file from a specified location in the metadata area of ​​the first persistent file based on the configuration information of the pre-allocated metadata area, and read a first compressed file from the data area of ​​the first persistent file based on the configuration information of the pre-allocated data area, where the configuration information of the pre-allocated data area includes the offset of the data area in the first persistent file. The storage system may use the dictionary corresponding to the first sub-file to decompress the first compressed file to obtain the first sub-file.

[0013] For upper-layer read requests, the storage system can combine the fixed index information (such as offset) of each persistent file in the dictionary space to perform precise prefetching, improve the cache hit rate, and facilitate subsequent read requests, significantly reducing the input and output (IO) path and improving read performance.

[0014] In some possible implementations, as data is written and overwritten, invalid data (garbage data) may be generated in the persistent file, where invalid data and garbage data refer to data that has been rewritten, such as data that has been rewritten in a subfile corresponding to a compressed file stored in the data area of ​​the persistent file. In contrast to invalid data is valid data, which refers to data that has not been rewritten. The storage system can also recycle garbage data to free up space. Specifically, the storage system can detect whether a merge condition is met; when the merge condition is met, the storage system merges multiple persistent files that have been written to the persistent storage medium to obtain a new persistent file, and deletes the multiple persistent files.

[0015] In this way, garbage data of persistent files can be recycled, storage space waste can be avoided, and storage resource utilization can be improved.

[0016] In some possible implementations, the storage system may decompress multiple persistent files written to the persistent storage medium, obtain valid data from each of the decompressed subfiles, and merge the valid data to obtain a merged subfile. The valid data is data that has not been overwritten. The storage system may then generate a dictionary corresponding to the merged subfile based on the merged subfile, compress the merged subfile using the dictionary corresponding to the merged subfile, obtain a merged compressed file, and generate a new persistent file based on the merged compressed file and the dictionary corresponding to the merged subfile.

[0017] On the one hand, this allows for lifecycle-friendly binding of data (e.g., subfiles of a data file) to the dictionary, achieving zero overlap between data and the dictionary, eliminating metadata amplification and avoiding performance issues caused by metadata overflow. On the other hand, regenerating a new dictionary that matches the characteristics of the merged subfiles can achieve better compression rates.

[0018] In some possible implementations, the merging condition includes at least one of the following:

[0019] The interval between the current time and the last merge time reaches the merge period; or,

[0020] The current load is less than the preset value; or,

[0021] The effective data ratio of at least one persistent file among the multiple persistent files written to the persistent storage medium is less than a preset ratio, where the effective data ratio is the ratio of the unrewritten data in the subfile corresponding to the persistent file; or

[0022] A merge request reported when a user-triggered merge operation is detected.

[0023] This method supports multiple merging methods, such as periodic merging, off-peak merging, merging based on the proportion of valid data, or manually triggered merging, which can meet the needs of different businesses and has high availability.

[0024] In some possible implementations, the storage system may determine multiple persistent files to be merged based on the attribute information of the persistent files written to the persistent storage medium, and merge the multiple persistent files to obtain a new persistent file.

[0025] This method screens persistent files written to persistent storage media based on their attribute information to determine multiple persistent files to be merged, rather than merging any persistent files. This allows the merging benefit to be greater than the merging cost, thus meeting business needs.

[0026] In some possible implementations, the attribute information of the persistent file includes a write time or a valid data ratio, wherein the valid data ratio is the ratio of data that has not been rewritten in the subfile corresponding to the persistent file. Accordingly, the storage system can sort the persistent files written to the persistent storage medium according to the write time of the persistent files written to the persistent storage medium, and determine a plurality of persistent files to be merged according to the sorting result. Alternatively, the storage system can determine a plurality of persistent files to be merged according to the valid data ratio of the persistent files written to the persistent storage medium, and the plurality of persistent files to be merged include persistent files whose valid data ratio is less than a preset ratio.

[0027] In this method, the storage system can determine the persistent files to be merged based on the first-in-first-out principle, or based on the principle of prioritizing the merging of persistent files with a high proportion of junk data, and then merge the above persistent files to obtain the highest possible merging benefits.

[0028] In some possible implementations, the storage system can also update the physical addresses in the address mappings of the multiple persistent files to the physical addresses of the new persistent file. This allows the storage system to index to the new persistent file based on the updated address mappings when it receives a read request for valid data in a subfile corresponding to the deleted persistent file, ensuring accurate reading and high read performance.

[0029] In some possible implementations, when a storage system receives a write request, it can first write the data in the write request to a log, retrieve the data files to be persisted from the log, and generate a dictionary during the log flushing process. If a failure occurs during this stage, the log is still available, allowing data to be replayed and the dictionary to be regenerated. This achieves data write protection and ensures high data reliability and availability.

[0030] In a second aspect, the present application provides a data compression device. The device comprises:

[0031] An acquisition module, configured to acquire a data file to be persisted, wherein the data file to be persisted includes a plurality of sub-files;

[0032] A persistence module is configured to: when persisting the first sub-file among the multiple sub-files:

[0033] generating a dictionary corresponding to the first subfile according to the first subfile, and compressing the first subfile using the dictionary to obtain a first compressed file;

[0034] generating a first persistent file according to the first compressed file and the dictionary; and

[0035] The first persistent file is written to a persistent storage medium.

[0036] In some possible implementations, the first persistent file includes a pre-allocated data area and a metadata area, the data area stores the first compressed file, and a specified location of the metadata area stores a dictionary corresponding to the first sub-file, the specified location is determined based on the configuration information of the pre-allocated metadata area, and the configuration information of the pre-allocated metadata area includes the offset of the metadata area in the first persistent file and the offset of the dictionary in the metadata area.

[0037] In some possible implementations, the acquisition module is further configured to:

[0038] receiving a read request for reading the first sub-file;

[0039] The device further comprises:

[0040] a data reading module, configured to read a dictionary corresponding to the first subfile from a specified position in the metadata area of ​​the first persistent file according to configuration information for pre-allocating the metadata area, and read the first compressed file from the data area of ​​the first persistent file according to configuration information for pre-allocating the data area, the configuration information for pre-allocating the data area including an offset of the data area in the first persistent file;

[0041] The first compressed file is decompressed using a dictionary corresponding to the first sub-file to obtain the first sub-file.

[0042] In some possible implementations, the apparatus further includes:

[0043] A detection module, used to detect whether the merging conditions are met;

[0044] The merging module is configured to merge the multiple persistent files written into the persistent storage medium to obtain a new persistent file and delete the multiple persistent files when the merging condition is met.

[0045] In some possible implementations, the merging module is specifically configured to:

[0046] decompressing the plurality of persistent files written to the persistent storage medium, obtaining valid data from each of the decompressed sub-files and merging the valid data to obtain a merged sub-file, wherein the valid data is data that has not been rewritten;

[0047] generating a dictionary corresponding to the merged sub-file according to the merged sub-file, and compressing the merged sub-file using the dictionary corresponding to the merged sub-file to obtain a merged compressed file;

[0048] A new persistent file is generated according to the merged compressed file and the dictionary corresponding to the merged sub-file.

[0049] In some possible implementations, the merging condition includes at least one of the following:

[0050] The interval between the current time and the last merge time reaches the merge period; or,

[0051] The current load is less than the preset value; or,

[0052] The effective data ratio of at least one persistent file among the multiple persistent files written to the persistent storage medium is less than a preset ratio, where the effective data ratio is the ratio of the unrewritten data in the subfile corresponding to the persistent file; or

[0053] A merge request reported when a user-triggered merge operation is detected.

[0054] In some possible implementations, the merging module is specifically configured to:

[0055] Determining multiple persistent files to be merged according to attribute information of the persistent files written to the persistent storage medium;

[0056] The multiple persistent files are merged to obtain a new persistent file.

[0057] In some possible implementations, the attribute information of the persistent file includes a write time or a valid data ratio, where the valid data ratio is a ratio of unrewritten data in a subfile corresponding to the persistent file;

[0058] The merging module is specifically used for:

[0059] sorting the persistent files written to the persistent storage medium according to the writing time of the persistent files written to the persistent storage medium, and determining the plurality of persistent files to be merged according to the sorting result; or

[0060] A plurality of persistent files to be merged are determined according to a valid data ratio of the persistent files written to the persistent storage medium, wherein the plurality of persistent files to be merged include persistent files whose valid data ratio is less than a preset ratio.

[0061] In some possible implementations, the apparatus further includes:

[0062] The address mapping update module is used to update the physical addresses in the address mapping relationship of multiple persistent files to the physical addresses of the new persistent file.

[0063] In a third aspect, the present application provides a storage system. The storage system includes a controller and a hard disk. The controller and the hard disk communicate with each other. The controller is configured to execute computer-readable instructions to cause the storage system to perform the data compression method described in the first aspect or any implementation of the first aspect.

[0064] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, wherein the instructions instruct a storage system to execute the data compression method described in the first aspect or any implementation of the first aspect.

[0065] In a fifth aspect, the present application provides a computer program product comprising instructions, which, when executed on a storage system, enables the storage system to execute the data compression method described in the first aspect or any one of the implementations of the first aspect.

[0066] Based on the implementation methods provided in the above aspects, this application can also be further combined to provide more implementation methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the technical methods of the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments.

[0068] Figure 1A A schematic diagram of the architecture of a centralized storage system with disk and controller separation provided in an embodiment of the present application;

[0069] Figure 1B A schematic diagram of the architecture of a centralized storage system with integrated disk and controller provided in an embodiment of the present application;

[0070] Figure 1C A schematic diagram of the architecture of a distributed storage system integrating storage and computing provided in an embodiment of the present application;

[0071] Figure 1D A schematic diagram of the architecture of a distributed storage system with storage and computing separation provided in an embodiment of the present application;

[0072] Figure 2 A flowchart of a data compression method provided in an embodiment of the present application;

[0073] Figure 3 A flowchart of a dictionary generation process provided in an embodiment of the present application;

[0074] Figure 4A schematic diagram of a persistent file recovery process provided in an embodiment of the present application;

[0075] Figure 5 A flowchart of a data compression method in a database provided in an embodiment of the present application;

[0076] Figure 6 A schematic diagram of a flow chart of a data compression method in a distributed storage system provided in an embodiment of the present application;

[0077] Figure 7 A schematic diagram of a data reading and writing process provided in an embodiment of the present application;

[0078] Figure 8 A schematic diagram of the structure of a computer cluster provided in an embodiment of the present application. DETAILED DESCRIPTION

[0079] The terms "first" and "second" in the embodiments of this application are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Therefore, features specified as "first" or "second" may explicitly or implicitly include one or more of the features.

[0080] First, some technical terms involved in the embodiments of this application are introduced.

[0081] Data compression (DC) is the process of representing information using fewer bits (or other information-related units) than the original encoding scheme, using a specific encoding scheme. Data compression can be applied to text, audio, video, or images to reduce storage space and storage costs.

[0082] The data compression method provided in this application can be applied to storage systems. The above-mentioned storage systems can be divided into the following categories according to logical functions: primary storage and backup storage. Primary storage serves as a storage pool or hard disk for upper-layer applications and can be accessed by upper-layer applications such as applications deployed in the form of virtual machines (VMs). Backup storage serves as a warehouse for storing images, backup disks or snapshots, which are usually not directly accessible to VMs. Primary storage and backup storage can be physically separated storage systems, or the same storage system can serve as both primary storage and backup storage. Storage systems can also be divided into file storage, database storage (including row-based databases or column-based databases), and big data storage according to application scenarios.

[0083] The above storage systems (including primary storage and backup storage) can adopt local storage mode or cloud storage mode. Local storage refers to the mode of storing data on one's own physical storage devices, while cloud storage refers to the mode of storing data in a cloud environment to support network access. The cloud environment refers to the cluster built by cloud service providers (also known as cloud vendors) to provide computing resources, storage resources, and network resources.

[0084] The storage system may adopt a centralized storage architecture or a distributed storage architecture. In other words, the storage system may be a centralized storage system or a distributed storage system.

[0085] A centralized storage system consists of one or more master devices forming a central node. Data is centrally stored in this central node, and all data processing operations for the entire system are centrally deployed on this central node. In other words, in a centralized storage system, terminals or clients are only responsible for data input and output, while data storage and control processing are completely handled by the central node. The most significant feature of a centralized system is its simple deployment structure. There is no need to consider how to deploy services on multiple nodes, nor does it require the consideration of distributed collaboration between multiple nodes.

[0086] For ease of understanding, Figure 1A Shows an architecture diagram of a centralized storage system. Figure 1A In the application scenario shown, users access data through applications. The computers that run these applications are called "application servers." The application server 100 can be a physical machine or a virtual machine VM. Physical machines (physical application servers) include but are not limited to desktop computers, servers, laptops, and mobile devices. The application server accesses the storage system through the fiber optic switch 140 to access data. However, the switch 140 is only an optional device, and the application server 100 can also communicate directly with the storage system 120 through the network. Alternatively, the fiber optic switch 140 can also be replaced with an Ethernet switch, an InfiniBand (IB) switch, an RDMA over Converged Ethernet (RDMA over Converged Ethernet, RoCE) switch, etc.

[0087] Figure 1A The storage system 120 shown is a centralized storage system. A centralized storage system is characterized by a unified entry point through which all data from external devices must pass. This entry point is the centralized storage system's engine 121. Engine 121 is the core component of the centralized storage system, implementing many of the system's advanced functions.

[0088] like Figure 1AAs shown, the engine 121 has one or more controllers. Figure 1A Take the example of an engine containing two controllers. There is a mirror channel between controller 0 and controller 1. Then, after controller 0 writes a copy of data to its memory 124, it can send a copy of the data to controller 1 through the mirror channel, and controller 1 stores the copy in its own local memory 124. In this way, controller 0 and controller 1 back up each other. When controller 0 fails, controller 1 can take over the business of controller 0. When controller 1 fails, controller 0 can take over the business of controller 1, thereby avoiding the unavailability of the entire storage system 120 due to hardware failure. When there are 4 controllers deployed in the engine 121, there is a mirror channel between any two controllers, so any two controllers back up each other.

[0089] The engine 121 also includes a front-end interface 125 and a back-end interface 126. The front-end interface 125 is used to communicate with the application server 100, thereby providing storage services for the application server 100. The back-end interface 126 is used to communicate with the hard disk 134 to expand the storage system capacity. Through the back-end interface 126, the engine 121 can connect to more hard disks 134, thereby forming a very large storage resource pool.

[0090] In hardware, such as Figure 1A As shown, the controller 0 includes at least a processor 123 and a memory 124. The processor 123 is a central processing unit (CPU) that is used to process data access requests from outside the storage system (servers or other storage systems), and is also used to process requests generated within the storage system. For example, when the processor 123 receives write data requests sent by the application server 100 through the front-end port 125, it temporarily stores the data in these write data requests in the memory 124. When the total amount of data in the memory 124 reaches a certain threshold, the processor 123 sends the data stored in the memory 124 to the hard disk 134 through the back-end port for persistent storage.

[0091] Memory 124 refers to internal memory that directly exchanges data with the processor. It can read and write data at any time and at a high speed, and serves as temporary data storage for the operating system or other running programs. Memory includes at least two types of memory. For example, memory can be either random access memory (RAM) or read-only memory (ROM). For example, RAM is dynamic random access memory (DRAM) or storage class memory (SCM), and ROM can be programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), etc.

[0092] Memory 124 stores software programs, and processor 123 executes the software programs in memory 124 to manage the hard disks. For example, hard disks can be abstracted into a storage resource pool, which is then divided into LUNs for server use. The LUNs here are essentially the hard disks seen on the server. Of course, some centralized storage systems are also file servers, providing shared file services to servers.

[0093] Controller 1 (and others Figure 1A The hardware components and software structure of the controller (not shown in the figure) are similar to those of the controller 0 and will not be repeated here.

[0094] Figure 1A The illustrated embodiment shows a centralized storage system with separate disk and controller architectures. In this system, the engine 121 may not have a hard drive slot; the hard drive 134 must be placed in the hard drive enclosure 130, with the back-end interface 126 communicating with the hard drive enclosure 130. The back-end interface 126 exists in the form of an adapter card within the engine 121. Two or more back-end interfaces 126 can be used simultaneously on one engine 121 to connect to multiple hard drive enclosures. Alternatively, the adapter card can be integrated into the motherboard, in which case the adapter card can communicate with the processor 123 via the PCIE bus.

[0095] It should be noted that Figure 1A Only one engine 121 is shown in the figure. However, in actual applications, the storage system may include two or more engines 121, and redundancy or load balancing is performed between the multiple engines 121.

[0096] The disk enclosure 130 includes a control unit 131 and several hard disks 134. The control unit 131 can have various forms. Typically, the number of control units 131 can be one, two, or more. When the disk enclosure 130 includes at least two control units 131, the hard disks 134 and the control units 131 may have an ownership relationship. The functions of the control unit 131 can be offloaded to the network card 104. In this embodiment, the disk enclosure 130 does not have a control unit 131. Instead, the network card 104 performs data reading and writing, address translation, and other computing functions. In this case, the network card 104 is an intelligent network card. It can include a CPU and memory. In some application scenarios, the network card 104 may also include persistent memory media, such as persistent memory (PM), non-volatile random access memory (NVRAM), or phase change memory (PCM). The CPU is used to perform operations such as address translation and reading and writing data. The memory is used to temporarily store data to be written to the hard disk 134 or data read from the hard disk 134 to be sent to the controller.

[0097] The centralized storage system can also be a disk-control integrated architecture. In the disk-control integrated architecture, the engine 121 has a hard disk slot, and the hard disk 134 can be directly deployed in the engine 121. The back-end interface 126 is an optional configuration. When the system storage space is insufficient, more hard disks or hard disk frames can be connected through the back-end interface 126. Figure 1B As shown, the engine 121 can be connected to more hard disks 134 through the backend interface 126 .

[0098] A distributed storage system stores data across multiple independent storage nodes. This system utilizes a scalable architecture, sharing the storage load across multiple storage nodes. This not only improves system reliability, availability, and access efficiency, but also facilitates scalability.

[0099] Distributed storage systems can adopt either a storage-computing integrated architecture or a storage-computing separated architecture. Each is described below.

[0100] like Figure 1C As shown, the distributed storage system with integrated storage and computing includes a storage cluster. The storage cluster includes one or more servers 110 ( Figure 1CThree servers 110 are shown in the figure, but are not limited to three servers 110. Each server 110 can communicate with each other. The server 110 is a device that has both computing and storage capabilities, such as a server, desktop computer, etc. For example, an ARM server or an X86 server can be used as the server 110 here. In terms of hardware, the server 110 includes at least a processor 112, a memory 113, a network card 114, and a hard disk 105. The processor 112, the memory 113, the network card 114, and the hard disk 105 are connected by a bus. The processor 112 is a central processing unit (CPU) that is used to process data access requests from outside the server 110 (application servers or other servers 110), and is also used to process requests generated within the server 110. In addition, the processor 112 is also used to perform calculations or processing on data, such as metadata management, deduplication, data compression, data verification, virtualized storage space, and address translation.

[0101] like Figure 1D As shown in FIG, a distributed storage cluster with storage and computing separation includes a computing cluster (also called a computing node cluster) and a storage cluster (also called a storage node cluster). Each computing node 110 can communicate with each other. A computing node 110 is a computing device, such as a server, a desktop computer, or a controller of a storage array. In terms of hardware, Figure 1D As shown, the computing node 110 includes at least a processor 112, a memory 113, and a network card 114. The processor 112 is a central processing unit (CPU) that processes data access requests from outside the computing node 110 or requests generated within the computing node 110. When the total amount of data in the memory 113 reaches a certain threshold, the processor 112 sends the data stored in the memory 113 to the storage node 100 for persistent storage. In addition, the processor 112 is also used for data calculation or processing, such as metadata management, data deduplication, data compression, virtualized storage space, and address translation.

[0102] Any computing node 110 can access any storage node 100 in the storage node cluster through the network. The storage node cluster includes multiple storage nodes 100 ( Figure 1DThree storage nodes 100 are shown in the figure, but are not limited to three storage nodes 100). A storage node 100 includes one or more controllers 101, a network card 104 and multiple hard disks 105. The network card 104 is used to communicate with the computing node 110. The hard disk 105 is used to store data and can be a disk or other type of storage medium, such as a solid-state drive or a shingled magnetic recording hard disk. The controller 101 is used to write data to the hard disk 105 or read data from the hard disk 105 according to the read / write data request sent by the computing node 110. In the process of reading and writing data, the controller 101 needs to convert the address carried in the read / write data request into an address that can be recognized by the hard disk. It can be seen that the controller 101 also has some simple computing functions.

[0103] It should be noted that the above is merely an example description of the storage system. In other possible implementations of the embodiments of the present application, the storage system may also be a distributed storage system or a centralized storage system of other architectures. For example, the storage system may also be a distributed storage system of a fully converged architecture.

[0104] When storing data, the storage system can compress the data to reduce the storage space occupied. Different data compression algorithms can have different data compression rates. For example, algorithms such as LZ4, ZSTD, and Snappy can provide data compression rates of multiples. However, the amount of data is growing exponentially, and traditional compression algorithms represented by LZ4, ZSTD, and Snappy cannot fully match the production, storage, and application requirements of massive data. The industry has also proposed new lossless data compression methods that break through the boundaries of traditional compression algorithms. Public Dictionary (PD) compression is a typical example. Public Dictionary Compression is a new data compression solution that breaks through local limitations and further improves the data reduction rate by mining a wider range and richer data features.

[0105] The following example illustrates the application of a public dictionary to a column-based database. Column-based databases use table partitioning, also known as column clusters. The data for each table partition (column cluster) is stored in a separate file. Within a column cluster, every k rows of data form a row group (each row group contains a fixed number of rows). Within each row group, each column of data is stored contiguously in a basic data processing unit, called a data block (block). Because each block has the same data type, it supports efficient data compression.

[0106] To speed up queries, the storage system can create metadata for each column and save it in a separate file. The metadata mainly includes the following four parts:

[0107] ①Header: includes version number (which can be represented by a timestamp, for example), file length, and some statistical information;

[0108] ② Column statistics: including the number of rows, the number of NULL values, cardinality (the number of different values), SUM (for sum), MAX (for maximum), and MIN (for minimum) values;

[0109] ③ Dictionary reference relationship: The public dictionary belongs to metadata. The dictionary number of the public dictionary is stored in the data file that references the public dictionary, thereby establishing a reference relationship with the public dictionary;

[0110] ④ Block address information: saves the mapping relationship between block number and data file starting address and length.

[0111] Because public dictionaries and local data files can have a one-to-many mapping relationship—that is, a public dictionary can be referenced by multiple data files—there are differences in the lifecycles of public dictionaries and local data files. This difference in lifecycles means that during queries, after a public dictionary is loaded into memory, if the multiple data files referencing it are not completely deleted, the public dictionary cannot be released. This inevitably leads to a long-tail problem, where the public dictionary occupies memory for a long time and cannot be released. This leads to metadata bloat, which causes a sharp increase in the memory occupied by the public dictionary, resulting in dictionary data being swapped in and out. This can cause the required public dictionary to be unable to be found in memory, which can lead to performance degradation.

[0112] In view of this, the present application provides a data compression method. The method generates a dictionary corresponding to the first sub-file based on the first sub-file when persisting the first sub-file in a data file to be persisted, compresses the first sub-file using the dictionary to obtain a first compressed file, and then persists the first compressed file and the dictionary in the same persistent file. For example, the persistent file is pre-partitioned to include a data area and a metadata area, the metadata area reserved in the persistent file is pre-populated with the dictionary corresponding to the first sub-file, and the data area reserved in the persistent file is populated with the first compressed file, thereby achieving persistence of the first sub-file and the dictionary corresponding to the first sub-file.

[0113] In this way, a single persistent file independently maintains the dictionary corresponding to the sub-file. The dictionary and the sub-file can implement the same lifecycle management, that is, the lifecycle of the data in the sub-file and the dictionary are affinity-bound. This solves the long-tail problem of local data files referencing the global public dictionary, which makes the public dictionary unable to be released. This avoids the public dictionary occupying memory for a long time and being unable to be released, causing metadata expansion (metadata overflow), resulting in the public dictionary needed to be used not being able to hit the memory, and the need for frequent swapping in and out, which in turn leads to the problem of storage system performance degradation.

[0114] In addition, this method can reuse the existing persistence engine to persist the compressed file after the sub-file is compressed and the dictionary corresponding to the sub-file, without the need to develop a proprietary persistence engine to persist the dictionary. In addition, there is a natural mapping relationship between the compressed file and the dictionary in the persistence file, and there is no need to add additional mapping metadata persistence, mapping update and other functions, which reduces the impact on the existing storage system architecture, reduces development difficulty and system complexity.

[0115] In order to make the technical solution of the present application clearer and easier to understand, the data compression method provided by the present application will be introduced below by taking the storage system of the above embodiment as an example.

[0116] See also Figure 2 A flow chart of a data compression method is shown, the method comprising:

[0117] S202: The storage system receives a write request.

[0118] A write request, also known as an input / output (IO) write, is a request for writing data. Specifically, a client can generate a write request in response to a user-triggered write operation, and the storage system receives the write request from the client. The write request includes the data to be written.

[0119] S204: The storage system writes the data in the write request into the log.

[0120] Specifically, during the data writing phase, the storage system can write the data in the write request to the log through append-only writing, for example, recording it to the write-ahead log (WAL), also known as the pre-write log, and subsequently transfer the data in the log to a persistent storage medium, for example, it can be periodically transferred to the disk, thereby ensuring high data reliability and high availability.

[0121] It should be noted that S204 is an optional step in the embodiment of the present application, and the method of the present application may also be executed without executing S204. For example, the storage system may also directly refresh the data in the write request to the persistent storage medium.

[0122] S206: The storage system obtains the data file to be persisted from the log.

[0123] A data file to be persisted refers to a data file that needs to be flushed to a persistent storage medium. A data file to be persisted can include multiple subfiles. For example, in a column-oriented database application, a data file to be persisted can be a database table, which can include multiple table partitions (column clusters). The data in the table partitions (column clusters) can form subfiles within the data file to be persisted.

[0124] Specifically, the storage system can update the log to a memory structure, which refers to a data structure in memory, such as a memory table (referred to as a Memtable). When a memory structure such as a Memtable meets the persistence conditions, the storage system can trigger persistence. The persistence conditions can be that the capacity of the memory structure reaches a target value and / or the time since the last data write reaches a target duration.

[0125] When the storage system triggers persistence, it can retrieve the data updated from the log to the memory structure and obtain the data file to be persisted. In some examples, when the storage system triggers persistence, it can set the current memory table to read-only (or freeze the current memory table) to obtain an immutable memory table (immutable memtable). The data file formed by the data in the immutable memtable can be a subfile of the data file to be persisted, such as the first subfile.

[0126] Furthermore, the storage system can create a new memory table to update other data in the log to the new memory table. When the new memory table meets the persistence conditions, the new memory table can be set to read-only, and the data in the new memory table forms a sub-file in the data file to be persisted, such as the second sub-file.

[0127] It should be noted that S202 to S206 above are a specific implementation method for the storage system to obtain the data file to be persisted. In other possible implementations of the embodiments of the present application, the data file to be persisted may also be obtained through other methods. For example, the storage system may directly determine the data in the write request as the data file to be persisted. This application does not limit the method for obtaining the data file to be persisted.

[0128] S208 : When persisting the first sub-file in the data file to be persisted, the storage system generates a dictionary corresponding to the first sub-file based on the first sub-file.

[0129] The storage system may generate a dictionary corresponding to the first sub-file based on the first sub-file during the process of transferring the log to a persistent storage medium (such as a log flush disk). The dictionary may be a public dictionary. Specifically, the first sub-file may include one or more data blocks, and the storage system may shard the data blocks to obtain training samples and test samples. The storage system may generate a dictionary based on the training samples, and then perform tests based on the test samples in the test set, for example, by compressing the test samples using the dictionary and then calculating the compression rate. When the compression rate meets the requirements, for example, the compression rate is greater than the target compression rate, the dictionary is determined as the final dictionary. When the compression rate does not meet the requirements, the dictionary is regenerated.

[0130] The following is a detailed description of the dictionary generation process with an example. Figure 3 As shown in the flowchart of dictionary generation, the storage system obtains the data block of the first sub-file, slices the data block, and obtains a training set and a test set. Among them, the storage system can adopt a variable-length sharding mechanism to slice the data block according to the frequency of the character string corresponding to the window in the data block to obtain the shards of the data block. For example, the storage system can determine the character strings with higher correlation based on the frequency of the character strings, divide the character strings with higher correlation into the same shard, and divide the character strings with lower correlation into different shards. In this way, different character strings in the same shard are regarded as a whole to participate in the subsequent dictionary training process, which improves the efficiency of processing character strings and reduces the frequency of loading interference information into the dictionary.

[0131] Furthermore, the training samples in the training set include shards of different categories, such as shards of different languages. The storage system can cluster the shards in the training samples to obtain training sets of different categories. Each category's training set includes training samples of that category, and each category's training samples includes shards of that category. The storage system trains dictionaries for each category using the training sets of each category. Accordingly, the storage system can use dictionaries of different categories to compress strings of different data types in the data blocks, thereby further improving the compression rate and ensuring the compression effect.

[0132] The following is an example of a dictionary of one of the categories. The set of slices of the target category can form a training set, such as training set T. The dictionary trained by the training set is denoted as D T . The training process is explained below. The storage system can divide the training set T into multiple units, and then set the window according to the window length in the dictionary parameter, slide the window on each unit of the training set T, and determine the score of the window when it slides to different positions. For each unit, when the score of the window when it slides to a certain position meets the set conditions, the storage system can load the string corresponding to the window into the dictionary of the target category, thereby generating a dictionary of the target category.

[0133] In some embodiments, the size of the multiple units may be fixed. A string corresponding to a window is selected in each unit, so the storage system can store the string according to the compression dictionary D. T The size of the unit and the window length determine the number of units. Further, the storage system can determine the size of each unit based on the number of units. Specifically, the storage system can divide the units according to the following formula:

[0134]

[0135] Where k represents the window length.

[0136] In a specific implementation, the storage system can obtain the scores of different strings in the window based on the frequency of different strings in the window (for example, the number of cells in which the string appears can be determined as the frequency of the string). Then, the storage system can obtain the score of the window when it slides to a certain position based on the scores of different strings in the window. Specifically, the storage system can determine the score of the window when it slides to each position according to the following formula:

[0137]

[0138] The window whose score meets the set conditions can be the window with the highest score, or the window with a score greater than a preset score. This embodiment uses the window with the highest score as an example. The storage system can output the string corresponding to the window with the highest score in each unit to the dictionary D T middle.

[0139] Similarly, the storage system can generate dictionaries for other categories based on training sets for other categories. The storage system can also merge dictionaries for different categories.

[0140] The storage system can use the trained dictionary to compress the test samples, and calculate the compression rate and compression time of each category, and then optimize the dictionary based on the compression rate and compression time. For example, the storage system can reduce the dictionary segments when the compression rate drops to an acceptable range (for example, greater than or equal to the minimum compression rate) to obtain the final dictionary. When reducing the segments, the storage system can select segments with smaller compression rate contributions for reduction. For example, the storage system can reduce segments whose compression rate contributions are less than a first preset value. For another example, the storage system can increase the segments of the dictionary to obtain the final dictionary. Specifically, the dictionary can increase the segments whose compression rate contributions are greater than a second preset value, specifically by loading more character strings to improve the overall compression rate.

[0141] If a failure occurs during the dictionary generation process, since the log is still available, the storage system can replay the data according to the log, thereby regenerating the dictionary corresponding to the first sub-file.

[0142] It should be noted that when persisting the second sub-file in a persistent data file, the storage system can generate a dictionary corresponding to the second sub-file based on the second sub-file. The specific implementation of the storage system generating the dictionary corresponding to the second sub-file based on the second sub-file can be referenced with the description of generating the dictionary corresponding to the first sub-file based on the first sub-file, and is not further described here.

[0143] S209: The storage system compresses the first sub-file using the dictionary to obtain a first compressed file.

[0144] S210: The storage system generates a first persistent file according to the first compressed file and a dictionary corresponding to the first sub-file.

[0145] The first persistent file includes a dictionary corresponding to the first sub-file and a first compressed file. Similarly, the storage system can compress the second sub-file based on the dictionary corresponding to the second sub-file to obtain a second compressed file, and then generate a second persistent file based on the second compressed file and the dictionary corresponding to the second sub-file. The second persistent file includes a dictionary corresponding to the second sub-file and a second compressed file.

[0146] In some possible implementations, persistent files can be pre-partitioned. Figure 3 A partition diagram of a persistent file is shown, wherein the persistent file includes a pre-allocated data area (data area) and a metadata area (meta area). The data area is used to store compressed files, and the metadata area is used to store metadata, including dictionaries. To facilitate rapid and accurate retrieval of dictionaries when subsequently reading data, dictionaries can typically be stored in a specified location in the metadata area. The specified location can be determined based on configuration information for pre-allocating the metadata area. The configuration information for pre-allocating the metadata area includes the offset of the metadata area in the persistent file and the offset of the dictionary in the metadata area.

[0147] exist Figure 3 In the example, the dictionary can be stored in the metadata area in the form of a dictionary block. The metadata area can also store other metadata besides the dictionary, such as index metadata, which can be stored in the metadata area in the form of an index block. Taking the first persistent file as an example, the data area of ​​the first persistent file stores the first compressed file, and the metadata area of ​​the first persistent file stores the dictionary corresponding to the first sub-file. For example, the starting address of the dictionary storage can be determined based on the first address of the first persistent file plus the offset of the metadata area in the first persistent file and the offset of the dictionary in the metadata area, and the storage system can store the dictionary at the location of the starting address (the specified location of the metadata area).

[0148] Further, if Figure 3 As shown, the persistent file may also include a header data area (head area). The head area stores footer metadata, including but not limited to version number, file length or statistical information.

[0149] The storage system can adopt a separate space reservation strategy to divide the persistent file into a data area and a meta area (further, a head area can also be divided). Space reservation is completed in advance when the persistent file is generated. After the dictionary is generated, the corresponding space reserved in the meta area is used to pre-fill the dictionary (that is, the dictionary is filled into the reserved meta area).

[0150] In which, the storage system can receive the configuration information of the partition, and the configuration information of the partition includes the configuration information of the data area and the configuration information of the meta area. In which, the configuration information of the data area can include the offset (or offset address) of the data area in the persistent file, and the configuration information of the meta area can include the offset (or offset address) of the meta area in the persistent file. Accordingly, when generating a persistent file, the persistent file can be partitioned according to the above offsets. In which, the meta area can be further partitioned. For example, the configuration information of the meta area can also include the offset of the dictionary in the meta area. In this way, the meta area can be further partitioned to facilitate the storage of the dictionary at a specified location in the meta area. Accordingly, when reading the dictionary, the storage system can read the dictionary from the specified location in the meta area of ​​the persistent file based on the first address of the persistent file, the offset of the meta area in the persistent file, and the offset of the dictionary in the meta area, thereby achieving efficient dictionary reading and improving read performance. Similarly, the storage system can read a compressed file from the data area of ​​the persistent file based on the first address of the persistent file and the offset of the data area in the persistent file, thereby achieving efficient data reading and improving read performance.

[0151] In some possible implementations, a head region may be further partitioned within the persistent file. In some examples, the storage system may define the region between the first address and the offset address of the data region as the head region. In other examples, the partition configuration information may further include configuration information for the head region, which may include the offset of the head region within the persistent file.

[0152] Among them, the head area can store the offset of the data area in the persistent file and the offset of the meta area in the persistent file. When performing a read operation on the persistent file, the data area can be determined based on the offset of the data area in the persistent file stored in the head area, and the compressed file can be read from the data area, or the meta area can be determined based on the offset of the meta area in the persistent file stored in the head area, and metadata can be read from the meta area. The head area of ​​the persistent file can also store the offset of the dictionary in the meta area, so that the dictionary can be accurately read from the specified position in the meta area. Since information such as the offset is configurable, in order to avoid affecting other partitions by modifying the format of the meta area, a head area can also be set in the meta area. For example, the meta area can be divided into a head area, a dictionary area, and an index area. Among them, the head area of ​​the meta area can store the offset of other partitions in the meta area (such as the dictionary area) in the meta area.

[0153] S212: The storage system writes the first persistent file to the persistent storage medium.

[0154] The persistent storage medium may be a hard disk, including but not limited to a mechanical hard disk HDD or a solid-state drive SSD. The storage system may write a first persistent file to the persistent storage medium. In other words, the first compressed file obtained by compressing the first sub-file and the dictionary corresponding to the first sub-file are stored in the persistent storage medium in the form of a first persistent file, thereby achieving persistence of data and metadata including dictionaries. Similarly, the storage system may also write a second persistent file to the persistent storage medium. In other words, the second compressed file obtained by compressing the second sub-file and the dictionary corresponding to the second sub-file are stored in the persistent storage medium in the form of a second persistent file.

[0155] Furthermore, when the persistence of the first sub-file fails, the storage system can also restore the first sub-file based on the log. For example, the storage system can replay the log to restore the first sub-file, so that the first sub-file can be recompressed to obtain the first compressed file, and the first compressed file and the dictionary corresponding to the first sub-file can be rewritten into the persistent storage medium, thereby achieving data and metadata persistence.

[0156] In some possible implementations, the storage system may also receive a read request for reading the first sub-file. Accordingly, the storage system may read the dictionary corresponding to the first sub-file from a specified location in the metadata area of ​​the first persistent file according to the configuration information of the pre-allocated meta area, and read the first compressed file from the data area of ​​the first persistent file according to the configuration information of the pre-allocated data area. Then, the storage system may decompress the first compressed file using the dictionary corresponding to the first sub-file to obtain the first sub-file. Considering the read performance, the storage system may also load the dictionary into the cache to improve the decompression efficiency. During the decompression process, the storage system may directly find the unique dictionary corresponding to the sub-file in the data file, thereby achieving zero overlap between the data and the dictionary.

[0157] Based on the above description, the data compression method of the embodiment of the present application generates a dictionary corresponding to a subfile in a persistent data file in real time when persisting the subfile. This method persists the compressed file obtained by compressing the subfile using the dictionary, along with the dictionary corresponding to the subfile. This eliminates the need for a dedicated persistence engine responsible for dictionary persistence, nor does it require the metadata persistence engine to add mapping metadata persistence and mapping update functions, thereby reducing development difficulty and system complexity. Furthermore, the persistent file reserves a metadata area for pre-populating the dictionary, enabling non-intrusive, zero-overhead metadata maintenance.

[0158] For storage systems that use append-write methods, persistent data can be overwritten. In other words, invalid data or garbage data can be generated in persistent files. Invalid data and garbage data refer to data that has been overwritten, such as data that has been overwritten in a subfile corresponding to a compressed file stored in the data area of ​​a persistent file. The opposite of invalid data is valid data, which refers to data that has not been overwritten, such as data that has not been overwritten in a subfile corresponding to a compressed file stored in the data area of ​​a persistent file.

[0159] As data is written and overwritten, the proportion of valid data in persistent files gradually decreases, allowing the storage system to reclaim the data and free up space. Reclaiming persistent files involves merging them to create new persistent files containing valid data. A new dictionary is then regenerated that matches the characteristics of the valid data in the new persistent files, ensuring high compatibility between the data and the dictionary throughout the entire lifecycle.

[0160] In some possible implementations, the storage system can set merging conditions. These merging conditions can be set based on business needs. For example, to save storage space, the merging conditions can be set to conditions related to the proportion of valid data. For another example, to reduce resource waste, the merging conditions can be set to conditions related to load, so that merging can be performed during idle time. Based on this, the merging conditions can include one or more of the following:

[0161] ① The interval between the current time and the last merge time reaches the merge period; ② The current load is less than the preset value; ③ The valid data ratio of at least one persistent file among multiple persistent files written to the persistent storage medium is less than the preset ratio, where valid data is data that has not been rewritten; ④ A merge request reported when a user triggers a merge operation is detected.

[0162] In specific implementations, the storage system can detect whether the merge condition is met. For example, the storage system can set a timer when the persistent file was last merged. When the timer reaches the merge period, the storage system determines that the interval between the current time and the last merge time has reached the merge period. In other words, the storage system can determine that merge condition ① is met.

[0163] When the merging condition is met, the storage system may merge multiple persistent files written to the persistent storage medium to obtain a new persistent file, and the storage system deletes the merged multiple persistent files (also called old persistent files) to reclaim space.

[0164] In some possible implementations, the storage system may determine multiple persistent files to be merged based on attribute information of persistent files written to the persistent storage medium, and merge the multiple persistent files to obtain a new persistent file.

[0165] In some examples, the attribute information of the persistent file may include the write time. The storage system may sort the persistent files written to the persistent storage medium according to the write time of the persistent files written to the persistent storage medium, and determine the multiple persistent files to be merged according to the sorting result. For example, the storage system may determine the multiple persistent files to be merged according to the first-in-first-out principle, based on which the storage system may determine the persistent files with the highest sorting result as the multiple persistent files to be merged. Among them, when the number of persistent files written to the persistent storage medium is 2n or 2n+1, the storage system may determine the persistent file with the sorting result of top i (1<i<n) as the persistent file to be merged.

[0166] In other examples, the attribute information of a persistent file may include a valid data ratio. The storage system may determine multiple persistent files to be merged based on the valid data ratio of the persistent files written to the persistent storage medium, where the multiple persistent files to be merged include persistent files whose valid data ratio is less than a preset ratio. The preset ratio may be set based on empirical values, such as 50%.

[0167] The persistent file merging process is described below. Specifically, the storage system can decompress multiple persistent files (for example, the multiple persistent files to be merged as determined above) that have been written to the persistent storage medium, obtain valid data from the decompressed sub-files respectively, and merge the valid data to obtain a merged sub-file. The storage system can then generate a dictionary corresponding to the merged sub-file based on the merged sub-file, and compress the merged sub-file using the dictionary corresponding to the merged sub-file to obtain a merged compressed file. The storage system can then generate a new persistent file based on the merged compressed file and the dictionary corresponding to the merged sub-file.

[0168] The storage system generates a dictionary corresponding to the merged subfiles, compresses the merged subfiles using the dictionary corresponding to the merged subfiles, and generates a new persistent file based on the merged compressed file and the dictionary corresponding to the merged subfiles. The process can refer to the storage system's persistence process for the first subfile among multiple subfiles, and will not be repeated here. Considering that after the multiple merged persistent files are deleted, if the storage system still reads data according to the original address mapping relationship, it may cause a read failure. To this end, the storage system can also update the physical addresses in the address mapping relationship of the multiple persistent files to the physical addresses of the new persistent files. When a read request is received, the storage request can be indexed to the physical address of the new persistent file through the updated address mapping relationship based on the logical block address (LBA) in the read request. For example, the address mapping relationship of the multiple persistent files can be LBA1→PBA1, LBA2→PBA2. When the multiple persistent files are merged into a new persistent file, the address mapping relationship of the multiple persistent files can be updated to LBA1→PBA3, LBA2→PBA3.

[0169] The following describes the persistent file merging process and the persistent file reading process with an example.

[0170] See also Figure 4 A flowchart of the process of merging and reading persistent files is shown. In this example, persistent files are stored in a Log Structured Merge Tree (LSM Tree). When the merge conditions are met, multiple persistent files in the LSM tree can be merged. For example, if the valid data ratio of the persistent files in the L0 layer is less than 50%, the persistent files in the L0 layer can be merged to generate a new persistent file and written to the L1 layer. When the valid data ratio of the persistent files in the L1 layer is less than 50%, the persistent files in the L1 layer can be merged to generate a new persistent file and written to the L2 layer.

[0171] Because persistent files are divided into the head area, data area, and meta area, when merging persistent files, the storage system not only obtains compressed files from the data area, decompresses the compressed files using the dictionary in the meta area, extracts valid data from the decompressed files, and merges the valid data to obtain a merged sub-file, but also regenerates a dictionary that meets the characteristics of the merged sub-file based on the merged sub-file. A new persistent file can be generated based on the merged sub-file and the regenerated dictionary.

[0172] In this example, when reading persistent files, the storage system can accurately pre-fetch metadata through metadata cache such as MetaCache. For example, the storage system can accurately pre-fetch dictionary 1 (denoted as dictionary1, abbreviated as dict1) from persistent file 1 (denoted as file1) and accurately pre-fetch dictionary 2 (denoted as dictionary2, abbreviated as dict2) from persistent file 2 (denoted as file2). This can improve the MetaCache hit rate and maximize system performance.

[0173] In this embodiment, when a subfile in a data file is persisted (downloaded from disk), the subfile and the dictionary are persisted and recycled at the same time, achieving lifecycle affinity binding between the data (for example, the subfile in the data file) and the dictionary. There is zero overlap between the data and the dictionary, no metadata amplification, and performance issues caused by metadata overflow are avoided.

[0174] In order to make the technical solution of the present application clearer and easier to understand, the data compression method of the present application is introduced below in conjunction with specific application scenarios.

[0175] See also Figure 5 A schematic diagram of a data compression method in a database is shown in FIG. Figure 5 As shown in Figure 1, the database storage engine consists of the following components: WAL files, which act like transaction logs, in-memory structures called Memtables, and persistent static sorted table (SST) files. Data is first recorded in the log and then updated to the in-memory structure called Memtables. Once capacity and duration conditions are met, a minor compaction operation is triggered, resulting in data being persisted in the SST files. The SST files are divided into the Head area, the Data area, and the Meta area. The WAL files can be used to recover Memtables that have not been written to disk in the event of a failure.

[0176] Specifically, when the storage system receives a write request from the upper layer, it can write-protect the data in the write request based on the WAL write protection mechanism provided in the LogStructured Merge Tree (LSM) layered architecture. The write protection process can be that the storage system writes the data to the log, such as writing it to a WAL file, then writing it to the memory structure Memtable, then serializing it, and persisting the serialized data to the disk. Figure 5 As shown in step ①, when a failure occurs, the storage system can replay the WAL file to recover the data and make it persistent again.

[0177] like Figure 5 As shown in step ②, when persisting, the storage system can pre-set dictionary filling space (pre-partition) in the Meta area of ​​the SST file. This does not introduce additional metadata management overhead. Based on the current unified metadata service, efficient and highly reliable metadata management can be achieved, avoiding the repeated development workload introduced by redeveloping the metadata persistence engine, while avoiding the complexity brought by managing multiple types of metadata.

[0178] In addition, to address the metadata overflow problem caused by the long tail of the dictionary, this embodiment affinity binds the life cycle of data (such as file objects such as the first sub-file) and the dictionary (such as the dictionary corresponding to the first sub-file). Specifically, a single SST file independently maintains a dictionary, and the dictionary scope is naturally isolated between SST files. At the same time, the database synchronously updates the dictionary during the recycling and merging (Major Compaction) process, which can ensure zero overlap between data and the dictionary, no metadata amplification, and avoid performance degradation caused by metadata overflow.

[0179] For upper-layer read requests, see Figure 5 In step ③, the storage system can combine the fixed index information of each SST file for the dictionary space to perform precise prefetching, improve the cache hit rate, and provide subsequent read requests with hits, significantly reducing the I / O path and improving read performance.

[0180] See also Figure 6 The diagram shows a data compression method in a distributed storage system. The distributed storage system may be a Ceph system with separated storage and computing. The Ceph system can be logically divided into a cache layer and a storage layer. The cache layer may be a global cache (GC), specifically including a cache frontend and a cache backend. The storage layer may include a Ceph cluster, which includes hard disks.

[0181] Specifically, clients can deploy client plug-ins such as Librados to accelerate reading and writing in the Ceph system. Librados provides a client adapter, which is used to perform semantic conversion (also known as escape) so that the server can identify read IO and write IO. The cache front-end includes a server adapter, which is used to perform semantic conversion, sending write IO to the write cache and sending read IO to the read cache (including L1 read cache or L2 read cache). The index system in the cache back-end performs common dictionary compression on sub-files in the data file and writes them to the persistence log (plog) after IO aggregation. To address the mapping management issues between data and the public dictionary, in this embodiment, the public dictionary is persisted along with the data (for example, the first subfile in the data file to be persisted). The implementation of append-write in plog requires that the public dictionary be aggregated at the tail. The native plog design requires distributed negotiation of the public dictionary's write position, and the negotiated position is monotonically increasing. In this embodiment, the plog format can be modified to pre-partition the plog. Specifically, the plog is divided into a data area, a meta area, and a header area, and the dictionary is pre-filled into the meta area, thereby enabling the public dictionary to be persisted along with the data.

[0182] See also Figure 7 Traditional methods require the development of a proprietary dictionary persistence engine for dictionary persistence, which complicates the storage system architecture and introduces additional development workload. Furthermore, dictionaries and data typically have a one-to-many mapping relationship. When data is first queried, the corresponding metadata file needs to be loaded into memory and cached. Furthermore, the dictionary cannot be released as long as the table data is not completely deleted. This operation inevitably leads to a long-tail problem where the dictionary occupies memory for a long time and cannot be released. This leads to metadata expansion, which in turn causes a sharp increase in memory usage. This results in dictionary data being swapped in and out, and the dictionary cannot be found in memory, resulting in performance degradation.

[0183] The data compression method of the present application adopts a separate space reservation strategy, dividing the persistent file into partitions such as the head area, data area, and meta area, and completing space reservation in advance when generating the file object. Therefore, after the dictionary is generated, the corresponding space is reserved in the meta area to pre-fill the dictionary (the dictionary is filled into the reserved meta area). The filling space is fully reserved and configurable.

[0184] Furthermore, to address the management and persistence of public dictionaries, the data compression method of this application allows the storage system to first write to a log (e.g., WAL), then update the memory, and finally serialize and persist the data to disk. This allows for data to be replayed and rewritten if a failure occurs during the WAL writing process, regenerating the dictionary during this process. Since failure scenarios are rare (daily failure rate 0.1%), retrying a failure does not impact performance.

[0185] To address the problem of metadata expansion and overflow, a single persistent file independently maintains a dictionary, and the two can correspond one-to-one. The lifecycles of data and dictionary are affinity-bound. When the persistent file is recycled or merged, the dictionary is also updated synchronously, thus achieving a one-to-one mapping throughout the entire lifecycle. Data and dictionary are persisted and recycled at the same time, avoiding metadata expansion and overflow.

[0186] Based on the data compression method of the above embodiment, the present application also provides a data compression device. The device can be deployed in Figures 1A to 1D The storage system shown in FIG. 1 operates the data compression device to perform Figure 2 The data compression method shown.

[0187] The data cache device provided by the present application is introduced below from the perspective of functional modularization in conjunction with the embodiments.

[0188] See also Figure 8 The data compression device 800 shown in FIG. 1 includes:

[0189] An acquisition module 802 is configured to acquire a data file to be persisted, wherein the data file to be persisted includes multiple sub-files;

[0190] The persistence module 804 is configured to: when persisting the first sub-file among the multiple sub-files:

[0191] generating a dictionary corresponding to the first subfile according to the first subfile, and compressing the first subfile using the dictionary to obtain a first compressed file;

[0192] generating a first persistent file according to the first compressed file and the dictionary; and

[0193] The first persistent file is written to a persistent storage medium.

[0194] The acquisition module 802 and the persistence module 804 can be implemented by software. Figure 2The method steps S202 to S206 in the embodiment shown are used to obtain the data file to be persisted. The persistence module 804 can be used to execute the method steps S208 to S212. The specific implementation process of the acquisition module 802 and the persistence module 804 can be found in Figure 2 The description of the relevant contents of the illustrated embodiment will not be repeated here.

[0195] In some possible implementations, the first persistent file includes a pre-allocated data area and a metadata area, the data area stores the first compressed file, and a specified location of the metadata area stores a dictionary corresponding to the first sub-file, the specified location is determined based on the configuration information of the pre-allocated metadata area, and the configuration information of the pre-allocated metadata area includes the offset of the metadata area in the first persistent file and the offset of the dictionary in the metadata area.

[0196] The pre-partitioning of the first persistent file may be implemented when the first persistent file is generated. The partitioning method of the first persistent file may refer to the description of the relevant contents of S210 and will not be described in detail here.

[0197] In some possible implementations, the obtaining module 802 is further configured to:

[0198] receiving a read request for reading the first sub-file;

[0199] The apparatus 800 further includes:

[0200] The data reading module 806 is used to read the dictionary corresponding to the first sub-file from the specified position of the metadata area of ​​the first persistent file according to the configuration information of the pre-allocated metadata area, and read the first compressed file from the data area of ​​the first persistent file according to the configuration information of the pre-allocated data area, the configuration information of the pre-allocated data area including the offset of the data area in the first persistent file; and decompress the first compressed file using the dictionary corresponding to the first sub-file to obtain the first sub-file.

[0201] The specific implementation of the data reading module 806 can be found in Figure 2 The description of the relevant contents of the illustrated embodiment will not be repeated here.

[0202] In some possible implementations, the apparatus 800 further includes:

[0203] A detection module 808 is used to detect whether the merging condition is met;

[0204] The merging module 810 is configured to merge the multiple persistent files written into the persistent storage medium to obtain a new persistent file and delete the multiple persistent files when the merging condition is met.

[0205] The specific implementation of the detection module 808 and the merging module 810 can be referred to Figure 4 The description of the relevant contents of the illustrated embodiment will not be repeated here.

[0206] In some possible implementations, the merging module 810 is specifically configured to:

[0207] decompressing the plurality of persistent files written to the persistent storage medium, obtaining valid data from each of the decompressed sub-files and merging the valid data to obtain a merged sub-file, wherein the valid data is data that has not been rewritten;

[0208] generating a dictionary corresponding to the merged sub-file according to the merged sub-file, and compressing the merged sub-file using the dictionary corresponding to the merged sub-file to obtain a merged compressed file;

[0209] A new persistent file is generated according to the merged compressed file and the dictionary corresponding to the merged sub-file.

[0210] In some possible implementations, the merging condition includes at least one of the following:

[0211] The interval between the current time and the last merge time reaches the merge period; or,

[0212] The current load is less than the preset value; or,

[0213] The effective data ratio of at least one persistent file among the multiple persistent files written to the persistent storage medium is less than a preset ratio, where the effective data ratio is the ratio of the unrewritten data in the subfile corresponding to the persistent file; or

[0214] A merge request reported when a user-triggered merge operation is detected.

[0215] In some possible implementations, the merging module 810 is specifically configured to:

[0216] Determining multiple persistent files to be merged according to attribute information of the persistent files written to the persistent storage medium;

[0217] The multiple persistent files are merged to obtain a new persistent file.

[0218] In some possible implementations, the attribute information of the persistent file includes a write time or a valid data ratio, where the valid data ratio is a ratio of unrewritten data in a subfile corresponding to the persistent file;

[0219] The merging module 810 is specifically used for:

[0220] sorting the persistent files written to the persistent storage medium according to the writing time of the persistent files written to the persistent storage medium, and determining the plurality of persistent files to be merged according to the sorting result; or

[0221] A plurality of persistent files to be merged are determined according to a valid data ratio of the persistent files written to the persistent storage medium, wherein the plurality of persistent files to be merged include persistent files whose valid data ratio is less than a preset ratio.

[0222] In some possible implementations, the apparatus 800 further includes:

[0223] The address mapping update module 812 is configured to update the physical addresses in the address mapping relationship of the plurality of persistent files to the physical addresses of the new persistent file.

[0224] The specific implementation of the address mapping update module 812 can be referenced Figure 4 The description of the relevant contents of the illustrated embodiment will not be repeated here.

[0225] The present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device, or a data storage device such as a data center that includes one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive). The computer-readable storage medium includes instructions that instruct a storage system to execute the aforementioned data compression method.

[0226] The present application also provides a computer program product including instructions. The computer program product may be software or a program product including instructions that can be run on a storage system or stored on any available medium. When the computer program product is run on a storage system, the storage system executes the above-described data compression method.

[0227] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the protection scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A data compression method, characterized in that: When persisting the first sub-file, the method includes: generating a dictionary corresponding to the first subfile according to the first subfile, where the first subfile is a subfile in the data file to be persisted; compressing the first sub-file using the dictionary to obtain a first compressed file; A first persistent file is generated according to the first compressed file and the dictionary, and the first persistent file is written into a persistent storage medium.

2. The method according to claim 1, characterized in that The first persistent file includes a pre-allocated data area and a metadata area. The data area stores the first compressed file. The specified position of the metadata area stores the dictionary corresponding to the first sub-file. The specified position is determined based on the configuration information of the pre-allocated metadata area. The configuration information of the pre-allocated metadata area includes the offset of the metadata area in the first persistent file and the offset of the dictionary in the metadata area.

3. The method according to claim 2, characterized in that The method further comprises: receiving a read request for reading the first sub-file; Reading a dictionary corresponding to the first subfile from a specified location in the metadata area of ​​the first persistent file according to configuration information for pre-allocating the metadata area, and reading the first compressed file from the data area of ​​the first persistent file according to configuration information for pre-allocating the data area, the configuration information for pre-allocating the data area including an offset of the data area in the first persistent file; The first compressed file is decompressed using a dictionary corresponding to the first subfile to obtain the first subfile.

4. The method according to any one of claims 1 to 3, characterized in that The method further comprises: Check whether the merge conditions are met; When the merging condition is met, the multiple persistent files written into the persistent storage medium are merged to obtain a new persistent file, and the multiple persistent files are deleted.

5. The method according to claim 4, characterized in that The merging of the multiple persistent files written into the persistent storage medium to obtain a new persistent file includes: decompressing the plurality of persistent files written to the persistent storage medium, obtaining valid data from each of the decompressed sub-files and merging the valid data to obtain a merged sub-file, wherein the valid data is data that has not been rewritten; generating a dictionary corresponding to the merged sub-file according to the merged sub-file, and compressing the merged sub-file using the dictionary corresponding to the merged sub-file to obtain a merged compressed file; A new persistent file is generated according to the merged compressed file and the dictionary corresponding to the merged sub-file.

6. The method according to claim 4, characterized in that The merging conditions include at least one of the following: The interval between the current time and the last merge time reaches the merge period; or, The current load is less than the preset value; or, The effective data ratio of at least one persistent file among the multiple persistent files written to the persistent storage medium is less than a preset ratio, where the effective data ratio is the ratio of the unrewritten data in the subfile corresponding to the persistent file; or A merge request reported when a user-triggered merge operation is detected.

7. The method according to claim 4, characterized in that The merging of the multiple persistent files written into the persistent storage medium to obtain a new persistent file includes: Determining multiple persistent files to be merged according to attribute information of the persistent files written to the persistent storage medium; The multiple persistent files are merged to obtain a new persistent file.

8. The method according to claim 7, characterized in that The attribute information of the persistent file includes a write time or a valid data ratio, where the valid data ratio is the ratio of the data that has not been rewritten in the sub-file corresponding to the persistent file; The determining of the plurality of persistent files to be merged according to the attribute information of the persistent files written into the persistent storage medium includes: sorting the persistent files written to the persistent storage medium according to the writing time of the persistent files written to the persistent storage medium, and determining a plurality of persistent files to be merged according to the sorting result; or, A plurality of persistent files to be merged are determined according to a valid data ratio of the persistent files written to the persistent storage medium, wherein the plurality of persistent files to be merged include persistent files whose valid data ratio is less than a preset ratio.

9. The method according to claim 4, characterized in that The method further comprises: The physical addresses in the address mapping relationship of the multiple persistent files are updated to the physical addresses of the new persistent file.

10. A data compression device, characterized in that: The device comprises: The persistence module is used to persist the first sub-file: generating a dictionary corresponding to the first subfile according to the first subfile, where the first subfile is a subfile in the data file to be persisted; compressing the first sub-file using the dictionary to obtain a first compressed file; A first persistent file is generated according to the first compressed file and the dictionary, and the first persistent file is written into a persistent storage medium.

11. The device according to claim 10, characterized in that The first persistent file includes a pre-allocated data area and a metadata area. The data area stores the first compressed file. The specified position of the metadata area stores the dictionary corresponding to the first sub-file. The specified position is determined based on the configuration information of the pre-allocated metadata area. The configuration information of the pre-allocated metadata area includes the offset of the metadata area in the first persistent file and the offset of the dictionary in the metadata area.

12. The device according to claim 11, characterized in that The device further comprises: an acquisition module, configured to receive a read request for reading the first sub-file; a data reading module, configured to read a dictionary corresponding to the first subfile from a specified position in the metadata area of ​​the first persistent file according to configuration information for pre-allocating the metadata area, and read the first compressed file from the data area of ​​the first persistent file according to configuration information for pre-allocating the data area, the configuration information for pre-allocating the data area including an offset of the data area in the first persistent file; The first compressed file is decompressed using a dictionary corresponding to the first sub-file to obtain the first sub-file.

13. The device according to any one of claims 10 to 12, characterized in that The device further comprises: A detection module, used to detect whether the merging conditions are met; The merging module is configured to merge the multiple persistent files written into the persistent storage medium to obtain a new persistent file and delete the multiple persistent files when the merging condition is met.

14. The device according to claim 13, characterized in that The merging module is specifically used for: decompressing the plurality of persistent files written to the persistent storage medium, obtaining valid data from each of the decompressed sub-files and merging the valid data to obtain a merged sub-file, wherein the valid data is data that has not been rewritten; generating a dictionary corresponding to the merged sub-file according to the merged sub-file, and compressing the merged sub-file using the dictionary corresponding to the merged sub-file to obtain a merged compressed file; A new persistent file is generated according to the merged compressed file and the dictionary corresponding to the merged sub-file.

15. The device according to claim 13, characterized in that The merging conditions include at least one of the following: The interval between the current time and the last merge time reaches the merge period; or, The current load is less than the preset value; or, The effective data ratio of at least one persistent file among the multiple persistent files written to the persistent storage medium is less than a preset ratio, where the effective data ratio is the ratio of the unrewritten data in the subfile corresponding to the persistent file; or A merge request reported when a user-triggered merge operation is detected.

16. The device according to claim 13, characterized in that The merging module is specifically used for: Determining multiple persistent files to be merged according to attribute information of the persistent files written to the persistent storage medium; The multiple persistent files are merged to obtain a new persistent file.

17. The device according to claim 16, characterized in that The attribute information of the persistent file includes a write time or a valid data ratio, where the valid data ratio is the ratio of the data that has not been rewritten in the sub-file corresponding to the persistent file; The merging module is specifically used for: sorting the persistent files written to the persistent storage medium according to the writing time of the persistent files written to the persistent storage medium, and determining a plurality of persistent files to be merged according to the sorting result; or, A plurality of persistent files to be merged are determined according to a valid data ratio of the persistent files written to the persistent storage medium, wherein the plurality of persistent files to be merged include persistent files whose valid data ratio is less than a preset ratio.

18. The device according to claim 13, characterized in that The device further comprises: The address mapping update module is used to update the physical addresses in the address mapping relationship of multiple persistent files to the physical addresses of the new persistent file.

19. A storage system, characterized in that: The storage system includes a controller and a hard disk, and the controller executes computer-readable instructions to enable the storage system to perform the method according to any one of claims 1 to 9.

20. A computer-readable storage medium, characterized in that The method comprises computer-readable instructions; the computer-readable instructions are used to implement the method according to any one of claims 1 to 9.

Images