A data compression method and apparatus

Abstract

The application discloses a data compression method and device, the method comprises the following steps: acquiring m data blocks of a data area in a readable and writable file system, compressing the m data blocks by using a preset compression algorithm, and sequentially obtaining n compressed data blocks, wherein the first capacity of each compressed data block is the same, and the first capacity represents the number of bytes of compressed data that the compressed data block can contain. The first index of each data block in the j data blocks corresponding to the i-th compressed data block in the n compressed data is established, and the mapping relationship between the first index and the j data blocks is recorded. Wherein, the first index is used to identify the storage location of each data block in the j data blocks in the storage medium, and the attribute information contained in each data block in the j data blocks. When reading the data block, the reading efficiency can be effectively improved, and the random reading scene can be completed with a small read amplification coefficient.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data compression, and particularly relates to a data compression method and device. BACKGROUND

[0002] In order to improve the overall input output (IO) read-write performance of a storage system, it is necessary to compress files in the memory. At present, the read-write file systems of Linux, such as F2FS, JFFS2, BTRFS, etc., and the read-write file systems of Windows, such as NTFS, etc., have the problem of random read amplification, and the read efficiency is low.

[0003] The existing data compression method usually compresses original file data (or source data) to be compressed according to a fixed minimum compressible unit, and the compressed file data (or compressed data) can include header data and compressed data. The header data is used to represent the attribute information of the file data, and the compressed data is used to represent the content of the file data. Then, the compressed file data is saved to a storage medium. However, the compression scheme of the existing read-write file system has the problem of random read amplification, and the read efficiency is low. SUMMARY

[0004] The embodiments of the present application provide a data compression method and device, which can solve the problem of random read amplification of the read-write file system and improve the read efficiency.

[0005] In a first aspect, an embodiment of the present application provides a data compression method. An execution subject of the method can be an electronic device or a component (for example, a chip, a chip system, or a processor) in the electronic device. The method includes: obtaining, by the electronic device, m data blocks in a data area of a read-write file system, where m is a positive integer greater than or equal to 1. The electronic device compresses the m data blocks by using a preset compression algorithm to obtain n compressed data blocks in sequence, where a first capacity of each compressed data block is the same, the first capacity represents a number of bytes of compressed data that can be contained in the compressed data block, and n is a positive integer greater than or equal to 1. The electronic device establishes a first index of each data block in j data blocks corresponding to an i th compressed data block in the n compressed data, and records a mapping relationship between the first index and the j data blocks. Where i is a positive integer greater than or equal to 1 and less than or equal to n, and j is a positive integer greater than or equal to 1 and less than or equal to m. The first index is used to identify a storage location of each data block in the j data blocks in a storage medium, and attribute information contained in each data block in the j data blocks.

[0006] Therefore, the data compression method provided by the embodiment of the present application can effectively improve the reading efficiency when reading the data blocks, and can ensure that the random reading scenario completes the reading of the data with a small read amplification factor. In addition, the attributes contained in the index of the data blocks can be modified, so that the compressed file on the storage device can be modified. It can be seen that the embodiment of the present application solves the problem of random read amplification of the existing read-write file system compression scheme, and solves the problem that the existing fixed output compression mode file system cannot support data and metadata updates.

[0007] In a specific implementation, the m data blocks are compressed by using the preset compression algorithm to obtain the n compressed data blocks in sequence. Specifically, each data block in the m data blocks is allocated to a first set in a preset order in sequence. When a data capacity of the j data blocks in the first set is equal to a rated capacity of the first set, a compression operation is performed on the j data blocks according to a set compression threshold, and the i th compressed data block is obtained.

[0008] In a specific implementation, the first index of each data block in the j data blocks corresponding to the i th compressed data block in the n compressed data is established. Specifically, when a sum of a head data of the i th compressed data block and a total data length of the compressed data is less than or equal to a total data length of the j data blocks, the first index of each data block in the j data blocks is established.

[0009] In a specific implementation, the attribute information includes at least one of the following: a first attribute, used to represent whether a storage location of the compressed data block after the data block is compressed is pre-allocated; a second attribute, used to represent whether the data page of the data block is valid; a third attribute, used to represent whether the data page of the data block is the first compressed page of the compressed data block of the data block; a fourth attribute, used to represent whether the data page of the data block is contained in the compressed data pages of two compressed blocks; a fifth attribute, used to represent whether the data page of the data block is the compressed page of the compressed data block after the data block is compressed; a sixth attribute, used to represent an index address of the compressed data block where the data page of the data block is located; and a seventh attribute, used to represent, when the data page of the data block belongs to the first compressed page of the compressed data block of the data block, an attribute value of the seventh attribute as an offset of the data block in a set corresponding to the compressed data block; and used to represent, when the data page of the data block does not belong to the first compressed page of the compressed data block of the data block, the attribute value of the seventh attribute as a distance of the data page of the data block from the first compressed page of the compressed data block.

[0010] In a specific implementation, the attribute information includes a third attribute, used to establish a first index of each of the j data blocks corresponding to the i-th compressed data block in the n compressed data, and specifically, when the data page of each of the j data blocks is the first compressed page of the i-th compressed data block, the attribute value of the third attribute is assigned as 1; and when the data page of each of the j data blocks is not the first compressed page of the i-th compressed data block, the attribute value of the third attribute is assigned as 0.

[0011] In a specific implementation, the attribute information includes the seventh attribute, and further includes: when the attribute value of the third attribute is 1, the attribute value of the seventh attribute is updated as an offset of the data block in a set corresponding to the compressed data block; and when the attribute value of the third attribute is 0, the attribute value of the seventh attribute is updated as a distance of the data page of the data block from the first compressed page of the compressed data block.

[0012] In a specific implementation, the attribute information includes a fourth attribute, used to establish a first index of each of the j data blocks corresponding to the i-th compressed data block in the n compressed data, and specifically, when the data page of each of the j data blocks is contained in the compressed data pages of two compressed blocks, the attribute value of the fourth attribute is assigned as 1; and when the data page of each of the j data blocks is not contained in the compressed data pages of two compressed blocks, the attribute value of the fourth attribute is assigned as 0.

[0013] In a specific implementation, the attribute information includes a second attribute, and the first index of each data block in the j data blocks corresponding to the i-th compressed data block is established, specifically: when the data page of each data block in the j data blocks is valid, the attribute value of the second attribute is assigned as 1. When the data page of each data block in the j data blocks is invalid, the attribute value of the second attribute is assigned as 0.

[0014] In some implementations, before the m data blocks are compressed by using the preset compression algorithm to obtain the n compressed data blocks in sequence, the method further includes: obtaining a second set of to-be- overwritten data, the second set including p compressed data blocks, p being a positive integer greater than or equal to 1. Obtaining a compressed page of a first target compressed data in the p compressed data blocks, and q data blocks corresponding to the compressed page of the first target compressed data block, q being a positive integer greater than or equal to 1. Determining a position offset of a first target data block in the q data blocks. Determining that the data page of the first target data block is a data page of the to-be-overwritten data.

[0015] In a specific implementation, the first index is used to identify the storage position of the i-th compressed data block in the storage medium, and attribute information contained in each data block in the j data blocks.

[0016] In some implementations, the method further includes: reading the first index of the first data block to obtain an index address of a first compressed data block corresponding to the first data block, wherein the first index includes attribute information of the first data block. Reading the index of the first compressed data block corresponding to the first data block. Decompressing the first compressed data block according to the index of the first compressed data block to obtain a plurality of data blocks corresponding to the first compressed data block, the plurality of data blocks including the first data block. Determining the offset of the first data block in the decompressed plurality of data blocks. According to the offset of the first data block in the decompressed plurality of data blocks, the data of the first data block can be obtained.

[0017] In a second aspect, an embodiment of the present application provides a data compression device, the device comprising: a first obtaining unit configured to obtain m data blocks in a data area of a read-write file system, m being a positive integer greater than or equal to 1; a compression unit configured to compress the m data blocks using a preset compression algorithm to obtain n compressed data blocks in sequence, wherein a first capacity of each compressed data block is the same, the first capacity representing a number of bytes of compressed data that the compressed data block can contain, and n is a positive integer greater than or equal to 1; and an updating unit configured to establish a first index of each data block in j data blocks corresponding to an i th compressed data block in the n compressed data blocks, and record a mapping relationship between the first index and the j data blocks, wherein i is a positive integer greater than or equal to 1 and less than or equal to n, j is a positive integer greater than or equal to 1 and less than or equal to m, and the first index is used to identify a storage location of each data block in the j data blocks in a storage medium and attribute information contained in each data block in the j data blocks.

[0018] Therefore, the data compression method provided by the embodiment of the present application can effectively improve the reading efficiency when reading data blocks, and can ensure that the random reading scenario completes the reading of data with a small read amplification factor. In addition, the attributes contained in the index of the data block can be modified, so that the compressed file on the storage device can be modified. It can be seen that the embodiment of the present application solves the problem of random read amplification of the existing read-write file system compression scheme, and solves the problem that the existing fixed output compression mode file system cannot support data and metadata updates.

[0019] In a specific implementation manner, the compression unit is configured to: allocate each data block in the m data blocks to the first set in a preset order in sequence. When a data capacity of the j data blocks in the first set is equal to a rated capacity of the first set, perform a compression operation on the j data blocks according to a set compression threshold, and obtain the i th compressed data block.

[0020] In a specific implementation manner, the updating unit is configured to: when a sum of a head data of the i th compressed data block and a total data length of the compressed data is less than or equal to a total data length of the j data blocks, establish the first index of each data block in the j data blocks.

[0021] In a specific implementation, the attribute information includes at least one of the following: a first attribute, used to represent whether a storage location of the compressed data block where the data block is located after being compressed is pre-allocated; a second attribute, used to represent whether the data page of the data block is valid; a third attribute, used to represent whether the data page of the data block is the first compressed page of the compressed data block of the data block; a fourth attribute, used to represent whether the data page of the data block is contained in the compressed data pages of two compressed blocks; a fifth attribute, used to represent whether the data page of the data block is the compressed page of the compressed data block after the data block is compressed; a sixth attribute, used to represent an index address of the compressed data block where the data page of the data block is located; and a seventh attribute, used to represent, when the data page of the data block belongs to the first compressed page of the compressed data block of the data block, an attribute value of the seventh attribute as an offset of the data block in a set corresponding to the compressed data block. When the data page of the data block does not belong to the first compressed page of the compressed data block of the data block, the attribute value of the seventh attribute is an offset of the data page of the data block from the first compressed page of the compressed data block.

[0022] In a specific implementation, the attribute information includes the third attribute, and the updating unit is further configured to assign the attribute value of the third attribute as 1 when the data page of each data block in the j data blocks is the first compressed page of the i-th compressed data block, and assign the attribute value of the third attribute as 0 when the data page of each data block in the j data blocks is not the first compressed page of the i-th compressed data block.

[0023] In a specific implementation, the attribute information includes the seventh attribute, and the updating unit is further configured to update the attribute value of the seventh attribute as an offset of the data block in a set corresponding to the compressed data block when the attribute value of the third attribute is 1, and update the attribute value of the seventh attribute as an offset of the data page of the data block from the first compressed page of the compressed data block when the attribute value of the third attribute is 0.

[0024] In a specific implementation, the attribute information includes the fourth attribute, and the updating unit is further configured to assign the attribute value of the fourth attribute as 1 when the data page of each data block in the j data blocks is contained in the compressed data pages of two compressed blocks, and assign the attribute value of the fourth attribute as 0 when the data page of each data block in the j data blocks is not contained in the compressed data pages of two compressed blocks.

[0025] In a specific implementation, the attribute information includes the second attribute, and the updating unit is further configured to assign the attribute value of the second attribute as 1 when the data page of each data block in the j data blocks is valid, and assign the attribute value of the second attribute as 0 when the data page of each data block in the j data blocks is invalid.

[0026] In some embodiments, the apparatus further comprises a second obtaining unit configured to obtain a second set of data to be overwritten, the second set comprising p compressed data blocks, p being a positive integer greater than or equal to 1. A third obtaining unit is configured to obtain a compressed page of a first target compressed data in the p compressed data blocks, and q data blocks corresponding to the compressed page of the first target compressed data block, q being a positive integer greater than or equal to 1. A first determining unit is configured to determine a position offset of a first target data block in the q data blocks. A second determining unit is configured to determine that a data page of the first target data block is a data page of the data to be overwritten.

[0027] In a specific embodiment, the first index is used to identify a storage location of the ith compressed data block in the storage medium, and attribute information contained in each of the j data blocks.

[0028] In some embodiments, the apparatus further comprises a first reading unit configured to read a first index of a first data block to obtain an index address of a first compressed data block corresponding to the first data block, wherein the first index comprises attribute information of the first data block. A second reading unit is configured to read an index of the first compressed data block corresponding to the first data block. A decompression unit is configured to decompress the first compressed data block according to the index of the first compressed data block to obtain a plurality of data blocks corresponding to the first compressed data block, the plurality of data blocks comprising the first data block. A third determining unit is configured to determine an offset of the first data block in the decompressed plurality of data blocks. A third obtaining unit is configured to obtain data of the first data block according to the offset of the first data block in the decompressed plurality of data blocks.

[0029] In a third aspect, an embodiment of the present application provides an apparatus, comprising: a processor configured to execute the data compression method of the first aspect.

[0030] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, comprising computer instructions, when the computer instructions are executed on an electronic device, the electronic device executes the data compression method of the first aspect.

[0031] In a fifth aspect, an embodiment of the present application provides a computer program, when the program is called by a processor, the data compression method of the first aspect is executed.

[0032] In a sixth aspect, an embodiment of the present application provides a chip system, comprising one or more processors, when the one or more processors execute instructions, the one or more processors execute the data compression method of the first aspect.

[0033] It should be understood that the descriptions of technical features, technical solutions, beneficial effects, or similar language in this application do not imply that all features and advantages can be achieved in any single embodiment. Rather, it is understood that the description of a feature or beneficial effect means that a specific technical feature, technical solution, or beneficial effect is included in at least one embodiment. Therefore, the descriptions of technical features, technical solutions, or beneficial effects in this specification do not necessarily refer to the same embodiment. Furthermore, the technical features, technical solutions, and beneficial effects described in this embodiment can be combined in any suitable manner. Those skilled in the art will understand that embodiments can be implemented without one or more specific technical features, technical solutions, or beneficial effects of a particular embodiment. In other embodiments, additional technical features and beneficial effects may be identified in specific embodiments that do not embody all embodiments. Attached Figure Description

[0034] Figure 1a A block diagram illustrating the composition of an operating system provided in an embodiment of this application;

[0035] Figure 1b This application provides a schematic diagram of the structure of a storage system according to an embodiment of the present application.

[0036] Figure 2 for Figure 1b A schematic diagram of the structure of a solid-state drive in a storage system;

[0037] Figure 3 for Figure 2 A schematic diagram of the flash memory chip structure in a solid-state drive (SSD);

[0038] Figure 4 for Figure 3 A schematic diagram of the flash translation layer corresponding to the flash memory chip in the middle;

[0039] Figure 5 This is a diagram illustrating a fixed input compression mode.

[0040] Figure 6 A schematic diagram of a fixed output compression mode provided in an embodiment of this application;

[0041] Figure 7 A schematic diagram of a data block index provided for an embodiment of this application;

[0042] Figure 8 This is a schematic diagram of a data block index in an existing scalable read-only file system;

[0043] Figure 9 A schematic flowchart illustrating a data compression method provided in an embodiment of this application;

[0044] Figure 10 A schematic diagram illustrating a data block index update process provided in an embodiment of this application;

[0045] Figure 11 This application provides a schematic diagram of the data block index relationship during data compression.

[0046] Figure 12 A flowchart illustrating yet another data compression method provided in this application embodiment;

[0047] Figure 13 A schematic diagram of the data block index relationship during an overwrite or read process is provided in an embodiment of this application;

[0048] Figure 14 A schematic diagram of a data reading process provided in an embodiment of this application;

[0049] Figure 15 This is a schematic diagram of a data compression device provided in an embodiment of this application. Detailed Implementation

[0050] The terms “comprising” and “having”, and any variations thereof, used in the description of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the steps or units listed, but may optionally include other steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus.

[0051] It should be noted that in the embodiments of this application, the words "exemplary" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of the words "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0052] In the description of this application, unless otherwise stated, "multiple" means two or more. "And / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone.

[0053] To facilitate understanding, the relevant terms and concepts that may be involved in the embodiments of this application will be introduced below.

[0054] like Figure 1a The diagram shows the components of an operating system.

[0055] An operation system (OS) is a computer program that manages computer hardware and software resources. For example, a unix operation system, a windows operation system, a linux operation system, etc. An operation system needs to handle basic transactions such as managing and configuring memory, deciding the priority of system resource supply and demand, controlling input and output devices, operating a network, and managing a file system. An operation system also provides an operation interface for a user to interact with a system.

[0056] An operation system kernel refers to the core part of most operation systems. It is composed of those parts in an operation system for managing memory, files, peripherals, and system resources, and is responsible for managing the processes, memory, device drivers, files, and network systems of a system, and determines the performance and stability of an operation system. An operation system kernel is a system software that provides a hardware abstraction layer, disk and file system control, multitasking, and other functions. It provides safe access to computer hardware for numerous application programs, and it can determine when an application program operates on a certain part of computer hardware for how long. Since direct operation on computer hardware is very complex, an operation system kernel can provide a set of hardware abstraction methods to complete these operations.

[0057] A file system is a core module, i.e., a main component, of an operation system kernel. A file system is a method of organizing files on a storage device, and is responsible for managing and storing file information, mainly for a user to create files, store, read out, modify, dump files, control file access, and delete files when a user no longer uses them.

[0058] A file system provides an abstract representation of a file in a kernel, completes mapping of a file to a physical storage device (for example, a disk, a hard disk, etc.), maps a physical address of a file on a storage device into a path name and a file name visible to a user, and facilitates quick reading, modification, and persistence of file data, etc.

[0059] File systems include read-write file systems and read-only file systems. A read-write file system is a file system that can both write files to a storage device and read files from the storage device, such as a file allocation table (FAT), a high performance file system (HPFS), a new technology file system (NTFS), a fourth extended file system (EXT4), a flash friendly file system (F2FS), and the like. A read-only file system is a file system that can only read files from a storage device but cannot write files to the storage device, such as an extendable read-only file system (EROFS).

[0060] In order to make the present application clearer, first, an application scenario of the present application is described.

[0061] Figure 1b Fig. 1 shows a structural diagram of a storage system.

[0062] In Figure 1b In the application scenario shown in Fig. 1, a user accesses data through an application program. A computer running the application program is referred to as an "application server". The application server 100 can be a physical machine or a virtual machine. The physical application server includes but is not limited to a desktop computer, a server, a notebook computer, and a mobile device. The application server accesses a storage system to access data through a fiber switch 110. However, the switch 110 is only an optional device, and the application server 100 can also directly communicate with the storage system 120 through a network. Alternatively, the fiber switch 110 can also be replaced by an Ethernet switch, an InfiniBand switch, a RoCE (RDMA over Converged Ethernet) switch, and the like.

[0063] Figure 1bThe storage system 120 shown is a centralized storage system. A centralized storage system refers to a central node composed of one or more master devices, and data is stored in the central node, and all data processing services of the entire system are deployed on the central node. In other words, in a centralized storage system, the terminal or client is only responsible for the input and output of data, and the storage and control processing of data are completely completed by the central node. The centralized storage system has the characteristics of a unified entrance, and all data from external devices pass through this entrance, which is the engine 121 of the centralized storage system. The engine 121 is the most core component in the centralized storage system, and many high-level functions of the storage system are implemented therein.

[0064] As shown in Figure 1b The engine 121 has one or more controllers, Figure 1b Taking an engine containing two controllers as an example, the controller 0 and the controller 1 have a mirror channel. When the controller 0 writes a piece of data into the memory 124 thereof, the controller 0 can send a copy of the data to the controller 1 through the mirror channel, and the controller 1 stores the copy in the local memory 124 thereof. Thus, the controller 0 and the controller 1 back up each other, and when the controller 0 fails, the controller 1 can take over the business of the controller 0, and when the controller 1 fails, the controller 0 can take over the business of the controller 1, thereby avoiding the hardware failure to cause the entire storage system 120 to be unavailable. When four controllers are deployed in the engine 121, there is a mirror channel between any two controllers, and thus any two controllers back up each other.

[0065] The engine 121 also contains a front-end interface 125 and a back-end interface 126, wherein the front-end interface 125 is used to communicate with the application server 100, thereby providing storage services for the application server 100. The back-end interface 126 is used to communicate with the hard disk 134 to expand the capacity of the storage system. Through the back-end interface 126, the engine 121 can connect more hard disks 134, thereby forming a very large storage resource pool.

[0066] According to the type of the communication protocol between the engine 121 and the hard disk frame 130, the hard disk frame 130 can be a SAS hard disk frame, an NVMe hard disk frame, an IP hard disk frame, and other types of hard disk frames. The SAS hard disk frame adopts the SAS3.0 protocol, and each frame supports 25 SAS hard disks. The engine 121 is connected with the hard disk frame 130 through the on-board SAS interface or the SAS interface module. The NVMe hard disk frame is more like a complete computer system, and the NVMe hard disk is inserted into the NVMe hard disk frame. The NVMe hard disk frame is connected with the engine 121 through the RDMA port.

[0067] In hardware, as Figure 1bAs shown, the controller 0 includes at least a processor 123 and a memory 124. The processor 123 is a central processing unit (CPU) configured to process data access requests from outside the storage system (e.g., a server or other storage system) and to process requests generated within the storage system. For example, when the processor 123 receives write data requests from the application server 100 via the front-end port 125, the processor 123 temporarily stores the data in the write data requests in the memory 124. When the total amount of data in the memory 124 reaches a certain threshold, the processor 123 sends the data stored in the memory 124 to the hard disk 134 via the back-end port for persistent storage.

[0068] The memory 124 is an internal memory that directly exchanges data with the processor. The memory 124 can read and write data at any time and at a high speed, and is used as a temporary data storage for an operating system or other programs that are running. The memory 124 includes at least two types of memory, such as a random access memory. For example, the random access memory can be a dynamic random access memory (DRAM) or a storage class memory (SCM). The DRAM is a semiconductor memory, and like most random access memories (RAMs), is a volatile memory device. The SCM is a hybrid storage technology that combines the characteristics of traditional storage devices and memories. The storage class memory can provide a faster read / write speed than a hard disk, but has a slower access speed than a DRAM and is less expensive than a DRAM. However, the DRAM and the SCM are only exemplary in this embodiment, and the memory can include other random access memories, such as a static random access memory (SRAM). In addition, the memory 124 can be a dual in-line memory module (DIMM) composed of DRAMs, or a solid state disk (SSD). In practical applications, multiple memories 124 and different types of memories 124 can be configured in the controller 0. The number and types of the memories 124 are not limited in this embodiment. In addition, the memory 124 can be configured to have a power retention function. The power retention function means that when the system is powered off and then powered on again, the data stored in the memory 124 will not be lost. The memory with the power retention function is referred to as a non-volatile memory.

[0069] Exemplarily, the memory 124 and the hard disk 134 can be solid-state drives (SSDs). As shown in Figure 2 The SSD 200 includes a NAND flash and a main controller (main control for short) 201. The NAND flash includes a plurality of flash chips 205 for storing data. The main control 201 is the brain of the SSD, responsible for some complex tasks, such as managing data storage, maintaining SSD performance and service life, etc. The main control 201 is an embedded microchip, which includes a processor 202, whose function is like a command center, issuing all operation requests of the SSD. For example, the processor 202 can execute read / write data, garbage collection, wear leveling and other functions through firmware in the buffer.

[0070] The SSD main control 201 also includes a host interface 204 and a plurality of channel controllers. The host interface 204 is used for communication with the host. The host here can refer to a server, a personal computer or an array controller, or any device. Through a plurality of channel controllers, the main control 201 can operate a plurality of flash chips 205 in parallel, thereby improving the underlying bandwidth. For example, assuming that there are 8 channels between the main control 201 and the FLASH particles, the main control 201 reads and writes data to the 8 flash chips 205 in parallel through the 8 channels.

[0071] As shown in Figure 3 A die is a package of one or more flash chips. A die can contain multiple panels, and a multi-plane NAND is a design that can effectively improve performance. As shown in Figure 3As shown, one die is divided into two planes, and the block numbers in the two planes are single and double interleaved, so that a single and double interleaved operation can be performed to improve performance during operation. One panel includes multiple blocks. One block is composed of a plurality of pages. Taking a 16 GB capacity flash chip as an example, 4314*8 = 34512 cells logically form a page, 4 KB of content and 218 B of ECC check data can be stored in each page, and the page is the minimum unit of IO operation. 128 pages form a block, 2048 blocks form a panel, and a whole flash chip is composed of two panels. One panel stores odd-numbered blocks, and the other panel stores even-numbered blocks. The two planes can be operated in parallel. This is only an example, and the size of the page, the capacity of the block, and the capacity of the flash chip can be different specifications, which are not limited in the embodiment.

[0072] When the host writes data into the block, the SSD host 201 selects the next block to continue writing when a block is full. The page is the minimum unit of data writing. In other words, the host 201 writes data into the block in page granularity. The block is the minimum unit of data erasing. The host can only erase the entire block at a time when erasing data.

[0073] The host accesses the SSD through a logical block address (LBA), and each LBA represents a sector (for example, 512 B). Inside the SSD, the host accesses the SSD in page units (for example, 4 KB). Therefore, when the application server writes one data at a time, the SSD host will find a page to write the data, and the address of the page is called a physical block address (PBA). The SSD records an LBA to PBA mapping. With such a mapping, when the host needs to read data of a certain LBA next time, the SSD knows which position of the flash chip to read the data from. Figure 4 is a schematic diagram of a flash translation layer (FTL), which is located in the firmware of the processor 202. As shown, Figure 4 When the host writes one new data, a new mapping relationship is generated, which is added (first write) or changed (overwritten) to the FTL. When reading certain data, the SSD first finds the PBA corresponding to the LBA of the data in the FTL, and then reads the corresponding data according to the PBA.

[0074] The flash chip cannot support the overwrite, which means that when the host modifies the data on a certain LBA, the PBA corresponding to the LBA cannot be directly changed, and a new PBA must be written, and a mapping is added in the FTL. For example, the FTL originally has a mapping relationship between LBA D and PBA D. When the host sends an IO request to modify the data of LBA D, the SSD finds a new location (PBA E) to write the data, and adds a mapping relationship between LBA D and PBA E in the FTL. The data on PBA D becomes invalid data. Invalid data (also known as garbage data) refers to data that has no mapping relationship. For this part of the data, the user will not access the FLASH space because it is replaced by new mapping relationships. With the continuous writing of the application server, the FLASH storage space gradually becomes smaller until it is exhausted. If the garbage data is not cleared in time, the host cannot write. The SSD has a garbage collection mechanism inside, and the basic principle is to move the valid data in several blocks to a new block, and then erase the several blocks, thus generating new available blocks.

[0075] In addition, the memory 124 also stores software programs, and the processor 123 runs the software programs in the memory 124 to realize the management of the hard disk. For example, the hard disk is abstracted into a storage resource pool, and then divided into LUNs to provide for the server. The LUN here is actually the hard disk seen on the server. Of course, some centralized storage systems themselves are file servers, which can provide shared file services for servers.

[0076] The data stored in the memory 124 can be presented through a file system. The file system is a structured data file storage and organization form. We know that all the data in the computer is 0 and 1, a series of 01 combinations stored on a hardware medium, which is completely indistinguishable and manageable to us. Therefore, we use the concept of "file" to organize these data for the same purpose, and different types of files are formed according to the structure required by different application programs. Different types are usually indicated by different suffixes, and then we give each file a name that is easy to understand and remember. When there are many files, we group them according to certain division methods, and each group of files is placed in the same directory (or folder). Moreover, under the directory, there can be a next-level directory (called subdirectory or subfolder), and all the files and directories form a tree structure. This tree structure has a special name: file system (File System). There are many types of file systems, such as FAT / FAT32 / NTFS of Windows, EXT2 / EXT3 / EXT4 / XFS / BtrFS of Linux, etc. In order to facilitate searching, from the root node, the directories are searched level by level down to the file itself, and the names of these directories, subdirectories, and files are spliced together with special characters (such as "\" in Windows / DOS and " / " in Unix-like systems). Such a string of characters is called a file path, such as " / etc / systemd / system.conf" in Linux or "C:\Windows\System32\taskmgr.exe" in Windows. The path is the unique identifier of a specific file. For example, D:\data\file.exe in Windows is a file path, which indicates the file.exe file in the data directory under the D partition.

[0077] The file system is built on the block device, and the file system not only records the file path, but also records which blocks constitute a file and which blocks record directory / subdirectory information. Different file systems have different organization structures. In order to facilitate management, a block device such as a hard disk can usually be divided into multiple logical block devices, that is, hard disk partitions (Partition). Conversely, the capacity and performance of a single medium are limited, and multiple physical block devices can be combined into a logical block device through certain technical means, such as various levels of RAID, JBOD, etc. The file system can also be built on these logical block devices. In any case, the application server application does not need to care about the specific location of the block device where the file to be accessed is located, and only needs to send the file name / ID of the file to the file system, and the file system can query the file path according to the file name / ID.

[0078] Common file access protocols are NFS, CIFS or SMB, etc. The present embodiment does not make any limitation on this.

[0079] The file system in the present application is a read-write file system. The read-write file system is a file system that can both write files to a storage device and read files from the storage device, for example, FAT, HPFS, NTFS, EXT4, F2FS, etc.

[0080] The file system generally includes a metadata area and a data area. The metadata area includes a superblock and an inode area. The superblock of the metadata area can include control information, data structures, etc. of the file system. The inode area of the metadata area can include description information of a file, such as file length, file type, etc. The file type is, for example, a regular inode, a directory inode, a symbol link inode, a special inode, etc. The data stored in the data area can be data obtained after file-level compression processing based on lossless compression technology. The data in the data area is stored on the physical storage space of the storage medium (for example, a disk, a flash memory, etc.) in the form of a collection of disk blocks. The data of the same file can be stored in continuous disk blocks, or can be stored in non-continuous disk blocks.

[0081] It should be understood that the introduction of the concept of disk blocks in the present application does not mean that the storage medium is limited to a disk. The disk block can be used to represent a small physical storage space obtained by dividing the physical storage space of the storage medium.

[0082] Of course, the storage system in the present application can also include a distributed storage system. The distributed storage system refers to a system that stores data on multiple independent storage nodes. The traditional network storage system uses a centralized storage array to store all data. The performance of the storage array is both a bottleneck of system performance and a focus of reliability and security, and cannot meet the needs of large-scale storage applications.

[0083] The above is a brief introduction to one application scenario of the present application.

[0084] In the above storage system, the rate of the device is sorted according to the data read-write capability, and the order from strong to weak is as follows: central processing unit (CPU) >> double data rate synchronous dynamic random access memory (DDR SDRAM) > flash chip falsh. It can be seen that the data access bottleneck in the storage system is the IO (input / output) time overhead of data between the memory and the flash.

[0085] In order to improve the overall IO read-write performance of the storage system, it is necessary to compress the files in the memory. Since the metadata area in the file system accounts for a small proportion of the entire file system, the data area often occupies a high device storage capacity. Therefore, when writing data to the falsh, compressing the data and writing the compressed data into the falsh can reduce the storage capacity occupation of the falsh and prolong the service life of the falsh.

[0086] At present, the read-write file systems of Linux, such as F2FS, journalling Flash file system version 2 (JFFS2), B-tree file system (BTRFS), etc., and the read-write file systems of Windows, such as NTFS, etc., can all use the following data compression method:

[0087] The original file data (or source data) to be compressed is compressed according to a fixed size of the smallest compressible unit (cluster), and the compressed file data (or compressed data) includes header data and compressed data. The header data is used to represent the attribute information of the file data, and the compressed data is used to represent the content of the file data. Then the compressed file data is saved to the falsh, and is aligned in size of 4kb.

[0088] For example, as shown in the fixed input compression mode diagram Figure 5 , four address continuous data blocks (blocks) are compressed as a cluster0 to obtain compressed file data composed of header data + compressed data. If the compressed file data is less than 4kb, the compressed file data is stored in the falsh in size of 4kb.

[0089] Suppose, Figure 5The original file data (or source data) size is 4 blocks, each block size is 4kb, a block is a logical page, and the logical pages of the original file data are numbered: 0, 1, 2, 3. The original file data is compressed into compressed file data at a compression rate of 75%, and the compressed file data size is 12kb. Therefore, the compressed file data size is 3 blocks, so the actual pages of the compressed file data are 3 pages, and the size of the flash actual pages that need to be read for reading a single logical page is shown in Table 1.

[0090] Table 1

[0091] Logical page number 0 1 2 3 Actual number of flash pages read (pages) 3 3 3 3

[0092] After saving the compressed file data to the flash, if the target logical page of the original file data needs to be read on the flash, 3 pages of compressed file data need to be read, and the target logical page of the original file data can be read after decompressing the compressed file data. For example, in the random read scenario, if the original file data of the 0th logical page needs to be read on the flash, the original file data of the 0th logical page can be successfully read after reading 3 pages of compressed file data and decompressing the compressed file data. Therefore, the data read efficiency is:

[0093]

[0094] It can be seen that the compressed file data obtained by the data compression mode shown in the figure has low read efficiency in the random read scenario. Figure 5 The data compression mode shown in the figure has low read efficiency in the random read scenario.

[0095] To solve the above technical problems, an embodiment of the present application provides a data compression method, which comprises the following steps: acquiring m data blocks in a data area of a read-write file system; and compressing the m data blocks by using a preset compression algorithm to obtain n compressed data blocks in sequence, wherein the first capacity of each compressed data block is the same, and the first capacity represents the number of bytes of the compressed data that the compressed data block can contain, and m and n are positive integers greater than or equal to 1.

[0096] The preset compression algorithm can be a compression algorithm corresponding to a fixed output compression mode, such as an (lempel-ziv 4, LZ4) compression algorithm. Of course, the preset compression algorithm can also be other compression algorithms, which are not limited in the embodiment of the present application.

[0097] For example, in an application scenario, as shown in Figure 6 For example, in an application scenario, as shown in

[0098] Assume that the 16kb source data of the logical page is divided into 3 parts, i.e. 6kb, 7kb and 5kb. The 3 parts of data are compressed by using a preset compression algorithm (e.g. LZ4) until the size of each part of the compressed data in the compressed data block is 4kb.

[0099] The data pages of the compressed data block are 3 pages, and are numbered as compressed page 4, compressed page 5 and compressed page 6 as shown in Table 2. Figure 6

[0100] Table 2

[0101] Logical page number 0 1 2 3 Actual number of flash pages read (pages) 1 2 2 1

[0102] It can be seen that the source data of the logical page 0 is compressed in the compressed page 4, and thus is compressed into 1 page. Part of the source data of the logical page 1 is compressed in the compressed page 4, and part of the source data of the logical page 1 is compressed in the compressed page 5, and thus is compressed into 2 pages. Part of the source data of the logical page 2 is compressed in the compressed page 5, and part of the source data of the logical page 2 is compressed in the compressed page 6, and thus is compressed into 2 pages. The source data of the logical page 3 is compressed in the compressed page 6, and thus is compressed into 1 page.

[0103] Therefore, in the random read scenario, any one or more logical pages described above can be read. For example, when the logical page 0 is read, only one compressed page needs to be read, as shown in the second row and the second column of Table 2. After decompression, all the data of the logical page 0 can be obtained.

[0104] At this time, the read efficiency can be calculated according to the following formula 2:

[0105]

[0106] The read efficiency of reading the logical page 3 is the same as that of reading the logical page 0.

[0107] When the logical page 1 is read, 2 compressed pages need to be read, as shown in the second row and the third column of Table 2. After decompression of the data of the compressed page 4 and the compressed page 5, all the data of the logical page 1 can be obtained.

[0108] At this time, the read efficiency can be calculated according to the following formula 3:

[0109]

[0110] The read efficiency of reading the logical page 2 is the same as that of reading the logical page 1.

[0111] In addition, the read efficiency of 4 logical pages on average can be calculated according to the following formula 4:

[0112] ​

[0113] The read efficiency calculated from formula 2, formula 3 and formula 4 can be seen that, in the random read scenario, Figure 6 The read efficiency of the data compression method shown is much greater than Figure 5 The read efficiency of the data compression method shown is much greater than

[0114] It can be seen that, by using the compression algorithm corresponding to the fixed output compression mode to compress the m data blocks in the data area of the readable and writable file system, n compressed data blocks of the same byte number are obtained in turn, so that each compressed data block output is of fixed size. When reading the data block, the read efficiency can be effectively improved, and the random read scenario can be completed with a very small read amplification factor.

[0115] In addition, Figure 8 The data block index method in the existing extendable read-only file system (erofs) is shown. In the data block address array data_addr, the accessed is the block address, which points to the address of the actual data block. When erofs is making an image, the method shown is used to compress the data. Since the structure of the storage device (for example: disk) and the file content are fixed, the modification of the file cannot be supported. However, in the actual operation scenario of the user, many compressed files on the storage device may need to be frequently modified, and erofs cannot support such demands. Figure 5 The data block index method in the existing extendable read-only file system (erofs) is shown. In the data block address array data_addr, the accessed is the block address, which points to the address of the actual data block. When erofs is making an image, the method shown is used to compress the data. Since the structure of the storage device (for example: disk) and the file content are fixed, the modification of the file cannot be supported. However, in the actual operation scenario of the user, many compressed files on the storage device may need to be frequently modified, and erofs cannot support such demands.

[0116] The data block index can find the corresponding data block, which can be understood as inode, that is, metadata. The inode is the area for storing metadata, that is, the area for storing file-related attribute information, such as the creator of the file, the creation date, the size, the location of the data block, and the like. Each inode has a number, and the operating system uses different inode numbers to identify different files. For example, on the surface, the user opens a file through the file name. In fact, the system first finds the corresponding inode number according to the file name, then obtains the inode information through the inode number, and then finds the address of the data block according to the inode information, and reads the data.

[0117] That is, inode records the file's attributes and the file's actual storage location, i.e. block number, each block (common size 4KB), through inode can realize the file's search positioning. Inode is in Linux, in Unix is vnode. Basically, inode contains information at least as follows: (1) the type of file (2) file access permissions; (3) file owner and group; (4) file size; (5) the number of links, i.e. the total number of file names pointing to the inode; (6) file status change time (ctime), the last access time (atime) and the last modification time (mtime); (7) file special attributes, SUID, SGID and SBIT; (8) the real pointer of file content.

[0118] As shown in Figure 8 , it is the existing data block index format, which is not scalable, for example: not support append write, block reservation, truncate and so on. Among them, append write refers to adding new content on the basis of the original file without deleting the content in the original file. Block reservation refers to the file system considering in advance which block space can allocate disk blocks if the file grows, and reserving these disk blocks. Truncate refers to modifying the file, for example: deleting, adding and so on.

[0119] As shown in Figure 8 , the data block index is referred to as blk entry, for convenience, it is abbreviated as blk. Among them, blk 1 is the index of compressed data block 1, which stores the address of compressed data block 1 on the storage device in blk1. Blk2 is the index of compressed data block 2, which stores the address of compressed data block 2 on the storage device in blk2. Blk3 is the index of compressed data block 3, which stores the address of compressed data block 3 on the storage device in blk3. Blk4 is the index of compressed data block 4, which stores the address of compressed data block 4 on the storage device in blk4. Therefore, according to the address stored in blk, the position of the compressed data block on the storage device can be determined.

[0120] In order to make the read-write file system support writing, overwriting, pre-allocating, truncating and the like, the data compression method provided by the embodiment of the present application further comprises: establishing a first index of each data block in the j data blocks corresponding to the i th compressed data block in the n compressed data blocks, and recording a mapping relationship between the first index and the j data blocks. Wherein, i is a positive integer greater than or equal to 1 and less than or equal to n. j is a positive integer greater than or equal to 1 and less than or equal to m. Wherein, the first index is used to identify the storage location of each data block in the j data blocks in the storage medium, and the attribute information contained in each data block in the j data blocks. Of course, the first index is also used to identify the storage location of the i th compressed data block in the storage medium, and the attribute information contained in each data block in the j data blocks.

[0121] Wherein, the attribute information is at least one of:

[0122] The first attribute is used to represent whether the storage location of the compressed data block in which the data block is located after being compressed is pre-allocated.

[0123] The second attribute is used to represent whether the data page of the data block is valid; that is, whether it is a normal data page or a hole data page, wherein the hole data page can be understood as a blank data page.

[0124] The third attribute is used to represent whether the data page of the data block is the first compressed page of the compressed data block of the data block.

[0125] The fourth attribute is used to represent whether the data page of the data block is contained in the compressed data pages of two compressed blocks.

[0126] The fifth attribute is used to represent whether the data page of the data block is the compressed page of the compressed data block after the data block is compressed.

[0127] The sixth attribute is used to represent the index address of the compressed data block in which the data page of the data block is located.

[0128] The seventh attribute is used to represent that when the data page of the data block belongs to the first compressed page of the compressed data block of the data block, the attribute value of the seventh attribute is the offset of the data block in the set corresponding to the compressed data block; when the data page of the data block does not belong to the first compressed page of the compressed data block of the data block, the attribute value of the seventh attribute is the distance of the data page of the data block from the first compressed page of the compressed data block.

[0129] Exemplarily, as Figure 7As shown, the first index of the data block includes: a blk entry corresponding to the address of the data block or the compressed data block; and an extent entry corresponding to the attribute information of the extension. Each extent entry corresponds to a blk entry, and each data page has a corresponding extent entry and blk entry.

[0130] The set of Extent entry members is shown as follows:

[0131] For example, as shown in set A, the members included in the data block index can be as shown in set A. It should be noted that each data page has a corresponding set A.

[0132] A{

[0133] is_reserved,

[0134] is_valid,

[0135] is_compress,

[0136] first_page,

[0137] cross_block,

[0138] blkidx,

[0139] offset,

[0140] …

[0141] other extensible attributes,

[0142] };

[0143] The meanings of the set members are explained as follows:

[0144] is_reserved is the first attribute described above.

[0145] is_valid is the second attribute described above.

[0146] first_page is the third attribute described above.

[0147] cross_block is the fourth attribute described above.

[0148] is_compress is the fifth attribute described above.

[0149] blkidx is the sixth attribute described above.

[0150] ofs is the seventh attribute described above.

[0151] It is evident that this is applied in read-write file systems. Figure 6 When compressing data using the method shown, it is possible to... Figure 7 The attributes contained in the index of the data block shown can be modified, thereby allowing modification of the compressed file on the storage device.

[0152] The data compression method provided in the embodiments of this application will be described below with specific examples.

[0153] Figure 9 This is a flowchart illustrating a data compression method provided in an embodiment of this application. Figure 9 As shown, the method includes:

[0154] S901. Obtain m data blocks from the data area of ​​the read-write file system, where m is a positive integer greater than or equal to 1.

[0155] These m data blocks can be understood as data blocks that need to be written back. Writing back means that during a write operation, the data is first written to memory for caching, but not immediately written to the storage device (e.g., disk). The cached data in memory is only written to the storage device under certain conditions or operations (e.g., flush mechanisms, synchronization operations, etc.).

[0156] S902. Compress m data blocks using a preset compression algorithm to obtain n compressed data blocks in sequence. Each compressed data block has the same first capacity, which represents the number of bytes of compressed data that the compressed data block can contain. n is a positive integer greater than or equal to 1.

[0157] The preset compression algorithm can be LZ4 compression algorithm, or other compression algorithms with fixed output. This application does not make specific limitations on the embodiments.

[0158] Where m can be any positive integer. For example, m is 4, m is 10, or m is 20.

[0159] Specifically, S902 can be implemented as follows:

[0160] S9021. Distribute each data block in the m data blocks to the first set in a preset order.

[0161] This preset order can be the order of consecutive storage addresses. That is, the consecutive order of m data blocks.

[0162] This first set can be called the smallest compressible unit (cluster). In other words, the first set is the smallest compressible set of data blocks. For example, Figure 6 The data block sets shown are 6kb, 7kb, and 5kb.

[0163] For example, m data blocks are mapped in a continuous address in the storage medium. Assuming that a data block is taken as a starting point, a fixed size data set is sequentially divided according to the order of the address of the data block mapped in the storage medium, such as Figure 6 As shown, 1 / 2 data of data block 0 and data block 1 are composed into a 6kb data set; 1 / 2 data of data block 1, 3 / 4 data of data block 2, and blank data pages are composed into a 7kb data set; 1 / 4 data of data block 2 and data block 3 are composed into a 5kb data set.

[0164] S9022, determine whether the data capacity of the j data blocks in the first set is equal to the rated capacity of the first set, j is a positive integer greater than or equal to 1 and less than or equal to m. If the data capacity of the j data blocks is not equal to the rated capacity of the first set, S9021 is executed; if the data capacity of the j data blocks is equal to the rated capacity of the first set, S9023 is executed.

[0165] S9023, performing fixed compression operation on the j data blocks in the first set according to the set compression threshold, and obtaining the i-th compressed data block.

[0166] The set compression threshold is used to represent the compression rate. For example, the expression formula of the set compression threshold can be:

[0167] Set compression threshold = total data length - total data length * compression rate

[0168] S9024, determining whether the total data length of the j data blocks is greater than the sum of the total data length of the header data and the compressed data of the i-th compressed data block and the set compression threshold. If yes, S903 is executed. Otherwise, the source data page is submitted to the flash memory.

[0169] S903, establishing a first index of each data block in the j data blocks corresponding to the i-th compressed data block in the n compressed data, and recording the mapping relationship between the first index and the j data blocks; wherein i is a positive integer greater than or equal to 1 and less than or equal to n; j is a positive integer greater than or equal to 1 and less than or equal to m.

[0170] It should be understood that when a compressed data block is compressed, the index of each data block corresponding to the compressed data block is established.

[0171] The first index is used to identify the storage location of each data block in the j data blocks in the storage medium, and the attribute information contained in each data block in the j data blocks.

[0172] For example, taking the linux f2fs read-write file system as an example, the first index format of the data block on the F2fs can be:

[0173] The first index includes attribute information of a data structure, which can be:

[0174] For example, an Entry data structure:

[0175]

[0176] The attribute information can include at least one of the following:

[0177] The first attribute (is_reserved) is used to represent whether the storage location of the compressed data block after the data block is compressed is pre-allocated;

[0178] The second attribute (is_valid) is used to represent whether the data page of the data block is valid;

[0179] The third attribute (first_page) is used to represent whether the data page of the data block is the first compressed page of the compressed data block of the data block;

[0180] The fourth attribute (cross_block) is used to represent whether the data page of the data block is included in the compressed data pages of two compressed blocks;

[0181] The fifth attribute (is_compress) is used to represent whether the data page of the data block is the compressed page of the compressed data block after the data block is compressed;

[0182] The sixth attribute (blkidx) is used to represent the index address of the compressed data block where the data page of the data block is located.

[0183] The seventh attribute (ofs) is used to represent that when the data page of the data block belongs to the first compressed page of the compressed data block of the data block, the attribute value of the seventh attribute is the offset of the data block in the corresponding set of the compressed data block; when the data page of the data block does not belong to the first compressed page of the compressed data block of the data block, the attribute value of the seventh attribute is the distance of the data page of the data block from the first compressed page of the compressed data block.

[0184] In a specific implementation manner, as shown in Figure 10 , a flowchart of a data block index updating provided by an embodiment of the present application is shown. As shown in Figure 10 , the attribute information can include the third attribute (first_page) and the seventh attribute (ofs), and S903 can be specifically implemented as:

[0185] S1031, determine whether the data page of each data block in the j data blocks is the first compressed page of the i-th compressed data block. If yes, the attribute value of the third attribute is assigned as 1; if not, the attribute value of the third attribute is assigned as 0.

[0186] S1032, when the attribute value of the third attribute is 1, updating the attribute value of the seventh attribute as the offset within the corresponding first set within the data block.

[0187] S1033, when the attribute value of the third attribute is 0, updating the attribute value of the seventh attribute as the distance between the data page of the data block and the first compressed page of the i-th compressed data block.

[0188] Of course, the attribute information can further include a fourth attribute (cross_block), and S103 can be specifically implemented as follows:

[0189] S1034, determining whether the data page of each data block in the j data blocks is contained in the compressed data pages of two compressed blocks. If yes, the attribute value of the fourth attribute is assigned as 1; if not, the attribute value of the fourth attribute is assigned as 0.

[0190] Of course, the attribute information can further include a second attribute (is_valid), and S103 can be specifically implemented as follows:

[0191] S1035, determining whether the data page of each data block in the j data blocks is valid. If yes, the attribute value of the second attribute is assigned as 1; if not, the attribute value of the second attribute is assigned as 0.

[0192] Of course, the attribute information can further include a sixth attribute (blkidx), and S103 can be specifically implemented as follows:

[0193] S1036, determining the index address of the compressed data block in which the data page of each data block in the m data blocks is located.

[0194] In which, according to the order of the storage positions of the m data blocks in the memory, the data is compressed in the size of the minimum fixed compression unit (such as the first set). When the first compression is completed (i.e., the first compressed data block is compressed), the data pages of the complete data blocks corresponding to the first compressed data block are all in the index positions of the first compressed block. For example, the data blocks corresponding to the first compressed data block are: data block 0, data block 1, data block 2, and part of the data of data block 3. The complete data blocks corresponding to the first compressed data block are data block 0, data block 1, and data block 2. Therefore, the data pages of data block 0, data block 1, and data block 2 are in the index positions of the first compressed data block.

[0195] It should be noted that, in addition to the seventh attribute which needs to be attached to the third attribute, the data block index updating processes corresponding to the other attributes are independent of each other. The order of the data block index updating processes corresponding to the first attribute, the second attribute, the fourth attribute, the fifth attribute, and the sixth attribute is not limited in the embodiments of the present application.

[0196] For example, assuming, such as Figure 11 As shown, the m data blocks include data block 0 (block0), data block 1 (block1), data block 2 (block2), and data block 3 (block3). Blocks 0, 1, 2, and 3 are mapped to a contiguous block of memory. When following the order in which blocks 0, 1, 2, and 3 are mapped to a contiguous block of memory (e.g., ...), the data is distributed as follows: Figure 11 Compression proceeds in a left-to-right direction, using the smallest fixed compression unit (such as the first set). During compression:

[0197] When block0 and a portion of block1 reach the minimum fixed compression unit (e.g., 4kb), the first compression is performed, resulting in the first compressed data block (compress blk0). At this point, the data block index for block0 is established, as shown in Table 3:

[0198] Table 3

[0199] Data block number (BlkNum) first_page cross_block blkidx ofs is_valid Block0 1 0 0 0 1

[0200] Combination Figure 10 Therefore, the data page of block0 falls on the first compressed page of the first compressed data block, so `first_page` is assigned a value of 1. The data page of block0 only falls on the first compressed page of the first compressed data block, so `cross_block` is assigned a value of 0. The index address of the data page of block0 in the first compressed data block is the sequence number of the first compressed data block (compress blk0), so `blkidx` is assigned a value of 0. The data page of block0 falls on the first compressed page of the first compressed data block, and the offset of block0 within its corresponding first set is 0, so `ofs` is assigned a value of 0. The data page of block0 is a valid data page, so `is_valid` is assigned a value of 1.

[0201] When the remaining portion of block1, a portion of block2, and a portion of block3 reach the minimum fixed compression unit (e.g., 4kb), a second compression is performed to obtain the second compressed data block (compress blk1). At this point, the data block indices for block1 and block2 are established, as shown in Table 4:

[0202] Table 4

[0203] Data block number (BlkNum) first_page cross_block blkidx ofs is_valid Block1 1 1 1 Ofs1 1 Block2 0 0 1 1 1

[0204] Combination Figure 10The data page of block 1 falls on the first compression page of the second compression data block, so first_page is assigned a value of 1. The data page of block 1 falls on the compression pages of the first and second compression data blocks, so cross_block is assigned a value of 1. The data page of block 1 falls on the index address of the second compression data block, which is the serial number of the second compression data block (compress blk 1), so blkidx is assigned a value of 1. The data page of block 1 falls on the first compression page of the second compression data block, and the offset of block 1 in the set of data blocks is Ofs 1, so ofs is assigned a value of Ofs 1. The data page of block 1 is a valid data page, so is_valid is assigned a value of 1.

[0205] Similarly, the data page of block 2 does not fall on the first compression page of the second compression data block, so first_page is assigned a value of 0. The data page of block 2 only falls on the compression page of the second compression data block, so cross_block is assigned a value of 0. The data page of block 2 falls on the index address of the second compression data block, which is the serial number of the second compression data block (compress blk 1), so blkidx is assigned a value of 1. The data page of block 2 does not fall on the first compression page of the second compression data block, and the distance of the data page of block 2 from the first compression page of the first compression data block is 1, so ofs is assigned a value of 1. The data page of block 2 is a valid data page, so is_valid is assigned a value of 1.

[0206] When the remaining part of block 3 reaches a minimum fixed compression unit (such as 4 kb), a third compression is performed to obtain a third compression data block (compress blk 2). At this time, the data block index of block 3 is established, as shown in Table 5:

[0207] Table 5

[0208] Data block number (BlkNum) first_page cross_block blkidx ofs is_valid Block3 1 1 2 Ofs2 1

[0209] In combination Figure 10As a result, the data page of block3 falls on the first compressed page of the third compressed data block, so first_page is assigned a value of 1. The data page of block1 falls on the compressed pages of the second and third compressed data blocks, so cross_block is assigned a value of 1. The data page of block1 falls on the index address of the third compressed data block, which is the serial number of the second compressed data block (compress blk2), so blkidx is assigned a value of 2. The data page of block3 falls on the first compressed page of the third compressed data block, and the offset of block3 in the set of data blocks is Ofs2, so ofs is assigned a value of Ofs2. The data page of block3 is a valid data page, so is_valid is assigned a value of 1.

[0210] S904, it is determined whether the m data are compressed. If yes, the compressed pages of the compressed data block are submitted to the device. If no, S902 is executed.

[0211] In some embodiments, as shown in FIG. 1, a flowchart of a data compression method provided by an embodiment of the present application is provided. As shown in FIG. 2, before S902 is executed, the data compression method provided by the embodiment of the present application further includes: Figure 12 Figure 12

[0212] S905, a second set of to-be- overwritten data is obtained.

[0213] S906, it is determined whether the second set of to-be- overwritten data contains a compressed data block. If yes, S907 is executed. If no, an existing data overwriting process is executed.

[0214] The second set can include p compressed data blocks, and p is a positive integer greater than or equal to 1.

[0215] S907, a compressed page of a first target compressed data in the p compressed data blocks in the second set is obtained, and q data blocks corresponding to the compressed page of the first target compressed data block are obtained, and q is a positive integer greater than or equal to 1.

[0216] S908, a position offset of a first target data block in the q data blocks is determined.

[0217] Specifically, the index addresses of the compressed data blocks in the second set are read, and the compressed data blocks are decompressed to obtain the data blocks corresponding to the compressed data blocks. Then, the position offsets of the data blocks in the q data blocks are determined.

[0218] S909, a data page of the first data block is obtained according to the first compressed page and the offset position of the first data block in the compressed page.

[0219] S909, a data page of the first data block is obtained according to the first compressed page and the offset position of the first data block in the compressed page.​​

[0220] S910, determine the data page of the first target data block as a data page to be overwritten with data.

[0221] S911, overwriting the second data block into the data page of the first data block.

[0222] S912, assign the second data block to the first set.

[0223] In summary, in the read-write file system f2fs, the specified so file, vdex file and odex file are compressed by using the fixed output compression mode provided in the present application, which can achieve the following benefits: for example, during the installation of 40 applications on an electronic device, each application can obtain an average time benefit of 12%. The so file exists additional writing, and the vdex file and the odex file exist the process of overwriting. The average startup benefit of 40 applications can obtain an additional 8% startup benefit compared with the fixed input compression mode compression data.

[0224] The above is the data compression method provided by the embodiment of the present application. After the data is compressed by using the above data compression method, the data needs to be read. For example Figure 14 As shown in the flowchart of the data reading process provided by the embodiment of the present application. As shown in Figure 14 The data reading process is as follows:

[0225] S141, read the first index of the first data block to obtain the index address of the first compressed data block corresponding to the first data block, wherein the first index includes the attribute information of the first data block.

[0226] The attribute information of the first data block can include at least one of the first attribute to the seventh attribute in the above embodiment. For example, the attribute information of the first data block includes the third attribute (first_page), the fourth attribute (cross_block), the sixth attribute (blkidx) and the seventh attribute (ofs):

[0227] In the overwriting and read-only scenarios, it is assumed that the first data block is Figure 13The block2 is read to obtain the index address of the first compressed data block corresponding to the block2. Specifically, referring to Table 4, the assignment of the ofs of the block2 is 1, which can determine that the data page of the block2 does not fall on the first compressed page of the second compressed data block, and can also determine that the distance between the data page of the block2 and the first compressed page of the first compressed data block is 1. The assignment of the cross_block of the block2 is 0, which can determine that the data page of the block2 only falls on the compressed page of the first compressed data block and does not fall on the compressed page of other compressed data blocks. The assignment of the blkidx of the block2 is 1, which can determine that the index address of the block2 in the first compressed data block can be the serial number of the first compressed data block, that is, the index address of the first compressed data block is 1.

[0228] S142, reading the index of the first compressed data block corresponding to the first data block.

[0229] S143, decompressing the first compressed data block according to the index of the first compressed data block to obtain a plurality of data blocks corresponding to the first compressed data block, wherein the plurality of data blocks include the first data block.

[0230] Specifically, the first compressed data block is found on the device according to the index of the first compressed data block. After the first compressed data block is found, the first compressed data block is parsed, and a plurality of data blocks after parsing are obtained. For example, as shown in Figure 13 , the first compressed data block is copress blk1, and copress blk1 is parsed to obtain part of data of a block1, a block2, and part of data of a block3.

[0231] S144, determining the offset of the first data block in the plurality of data blocks after decompression.

[0232] Specifically, according to the attribute information of the first data block, it can be determined that the first data block is Figure 13 a block2 as shown in Figure 13 , the expression of the offset (dstofs) of the block2 in the plurality of data blocks after parsing of the copress blk1 is:

[0233] dstofs = block_size - ofs1%block_size

[0234] Wherein, the dstofs represents the offset of the block2 in the plurality of data blocks after the copress blk1 is parsed; the block_size represents the length of the data block; the ofs1 represents the attribute value of the seventh attribute; and the ofs1%block_size represents the remainder.

[0235] S145, according to the offset of the first data block in the plurality of data blocks after decompression, the data of the first data block can be obtained.

[0236] According to the above example, the first data block is block2, as Figure 13 and Table 4, the data of block2 can be obtained.

[0237] Specifically, the communication system in the possible design is used to execute Figure 9 the functions of each device in the data compression method shown, so the same effect as the above data compression method can be achieved.

[0238] Figure 15 A data compression device provided by the embodiment of the application, the data compression device 1500 can include: a first acquisition unit 1501, configured to acquire m data blocks in a data area of a read-write file system, m is a positive integer greater than or equal to 1. A compression unit 1502 is configured to compress the m data blocks using a preset compression algorithm to obtain n compressed data blocks in turn, wherein the first capacity of each compressed data block is the same, the first capacity represents the number of bytes of the compressed data that the compressed data block can contain, and n is a positive integer greater than or equal to 1. An updating unit 1503 is configured to establish a first index of each data block in j data blocks corresponding to an i-th compressed data block in the n compressed data, and record a mapping relationship between the first index and the j data blocks. Wherein, i is a positive integer greater than or equal to 1 and less than or equal to n; j is a positive integer greater than or equal to 1 and less than or equal to m. Wherein, the first index is used to identify the storage location of each data block in the j data blocks in the storage medium, and the attribute information contained in each data block in the j data blocks.

[0239] In a specific implementable manner, the compression unit 1502 is configured to: allocate each data block in the m data blocks to the first set in a preset order in turn. When the data capacity of the j data blocks in the first set is equal to the rated capacity of the first set, perform a compression operation on the j data blocks according to the set compression threshold, and obtain an i-th compressed data block.

[0240] In a specific implementable manner, the updating unit 1503 is configured to: when the sum of the head data of the i-th compressed data block and the total data length of the compressed data is less than or equal to the total data length of the j data blocks, establish the first index of each data block in the j data blocks.

[0241] In a specific implementation, the attribute information includes at least one of the following: a first attribute, used to represent whether a storage location of the compressed data block after the data block is compressed is pre-allocated; a second attribute, used to represent whether the data page of the data block is valid; a third attribute, used to represent whether the data page of the data block is the first compressed page of the compressed data block of the data block; a fourth attribute, used to represent whether the data page of the data block is contained in the compressed data pages of two compressed blocks; a fifth attribute, used to represent whether the data page of the data block is the compressed page of the compressed data block after the data block is compressed; a sixth attribute, used to represent an index address of the compressed data block where the data page of the data block is located; and a seventh attribute, used to represent, when the data page of the data block belongs to the first compressed page of the compressed data block of the data block, an attribute value of the seventh attribute as an offset of the data block in a set corresponding to the compressed data block, and when the data page of the data block does not belong to the first compressed page of the compressed data block of the data block, the attribute value of the seventh attribute as a distance of the data page of the data block from the first compressed page of the compressed data block.

[0242] In a specific implementation, the attribute information includes the third attribute, and the updating unit 1503 is further configured to assign, when the data page of each of the j data blocks is the first compressed page of the i-th compressed data block, an attribute value of the third attribute as 1, and assign, when the data page of each of the j data blocks is not the first compressed page of the i-th compressed data block, the attribute value of the third attribute as 0.

[0243] In a specific implementation, the attribute information includes the seventh attribute, and the updating unit 1503 is further configured to, when the attribute value of the third attribute is 1, update an attribute value of the seventh attribute as an offset of the data block in a set corresponding to the compressed data block, and when the attribute value of the third attribute is 0, update the attribute value of the seventh attribute as a distance of the data page of the data block from the first compressed page of the compressed data block.

[0244] In a specific implementation, the attribute information includes the fourth attribute, and the updating unit 1503 is further configured to assign, when the data page of each of the j data blocks is contained in the compressed data pages of two compressed blocks, an attribute value of the fourth attribute as 1, and assign, when the data page of each of the j data blocks is not contained in the compressed data pages of two compressed blocks, the attribute value of the fourth attribute as 0.

[0245] In a specific implementation, the attribute information includes the second attribute, and the updating unit is further configured to assign, when the data page of each of the j data blocks is valid, an attribute value of the second attribute as 1, and assign, when the data page of each of the j data blocks is invalid, the attribute value of the second attribute as 0.

[0246] In some embodiments, the method further comprises: obtaining a second set of data to be overwritten, the second set of data comprising p compressed data blocks, p being a positive integer greater than or equal to 1; obtaining a compressed page of a first target compressed data in the p compressed data blocks, and q data blocks corresponding to the compressed page of the first target compressed data block, q being a positive integer greater than or equal to 1; determining a position offset of a first target data block in the q data blocks; and determining a data page of the first target data block as a data page of the data to be overwritten.

[0247] In some embodiments, the method further comprises: reading a first index of the first data block to obtain an index address of a first compressed data block corresponding to the first data block, wherein the first index comprises attribute information of the first data block; reading an index of the first compressed data block corresponding to the first data block; decompressing the first compressed data block according to the index of the first compressed data block to obtain a plurality of data blocks corresponding to the first compressed data block, the plurality of data blocks comprising the first data block; determining an offset of the first data block in the decompressed plurality of data blocks; and obtaining data of the first data block according to the offset of the first data block in the decompressed plurality of data blocks.

[0248] In a specific embodiment, the first index is used to identify a storage location of the ith compressed data block in the storage medium, and attribute information contained in each of the j data blocks.

[0249] Therefore, the data compression method provided by the embodiments of the present application can effectively improve the reading efficiency when reading data blocks, and can ensure that the random reading scenario completes the reading of data with a small read amplification factor. In addition, the attributes contained in the index of the data block can be modified, so that the compressed file on the storage device can be modified. It can be seen that the embodiments of the present application solve the problem of random read amplification of the existing read-write file system compression scheme, and solve the problem that the existing fixed output compression mode file system cannot support data and metadata updates.

[0250] The embodiments of the present application also provide a device, which comprises units for performing the steps of any of the above methods, or units for performing the steps of any of the above methods.

[0251] The embodiments of the present application also provide a computer readable storage medium comprising instructions which, when executed on a computer, cause the computer to perform any of the above methods.

[0252] The embodiment of the present application further provides a computer program product containing instructions, which, when running on a computer, causes the computer to execute any of the above methods.

[0253] The embodiment of the present application further provides a chip, which comprises a processor and an interface circuit, the interface circuit is coupled with the processor, the processor is used to run a computer program or instructions to implement the above method, and the interface circuit is used to communicate with other modules outside the chip.

[0254] In the description of the present application, unless otherwise specified, " / " means "or", for example, A / B can mean A or B. "And / or" in the present application is only a description of the association relationship between the associated objects, which means that there can be three relationships, for example, A and / or B can mean that A exists alone, A and B exist together, and B exists alone. In addition, "at least one" means one or more, and "multiple" means two or more. "First", "second", and the like do not limit the quantity and execution order, and "first", "second", and the like do not necessarily mean different.

[0255] In the description of the present application, "exemplary" or "for example" is used to mean an example, illustration, or description. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as more preferred or more advantageous than other embodiments or design schemes. Rather, the use of "exemplary" or "for example" is intended to present the relevant concept in a specific manner.

[0256] Through the description of the above embodiments, those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above division of functional modules is taken as an example, and in actual application, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.

[0257] In the several embodiments provided in the present application, it should be understood that the disclosed device and method can be implemented by other manners. For example, the device embodiment described above is only schematic, for example, the division of the modules or units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another device, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the shown or discussed mutual ones can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0258] The units described as separate components may or may not be physically separate, and the components displayed as units may be a physical unit or multiple physical units, that is, may be located in one place, or also can be distributed to multiple different places. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment of the present application.

[0259] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present alone, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0260] Although the present application is described in conjunction with specific features and embodiments thereof, it is obvious that various modifications and combinations can be made thereto without departing from the spirit and scope of the application. Accordingly, the specification and drawings are to be regarded simply as illustrative of the present application and are to be construed in accordance with the appended claims, and it is intended to cover any and all modifications, variations, combinations or equivalents that fall within the scope of the present application. Obviously, those skilled in the art can make various modifications and variations to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalent technologies, the present application is intended to include these modifications and variations.

[0261] It should be noted that: the above is merely a specific implementation of the present application, but the protection scope of the present application does not limit to this, any change or replacement within the technical scope disclosed by the present application should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A data compression method characterized by, The method comprises: Obtaining m data blocks in a data area of a read-write file system, m being a positive integer greater than or equal to 1; Compressing the m data blocks by using a preset compression algorithm to sequentially obtain n compressed data blocks, wherein a first capacity of each compressed data block is the same, the first capacity representing a number of bytes of compressed data that the compressed data block can contain, and n is a positive integer greater than or equal to 1; Establishing a first index of each data block in the j data blocks corresponding to the i-th compressed data block in the n compressed data, and recording a mapping relationship between the first index and the j data blocks; wherein i is a positive integer greater than or equal to 1 and less than or equal to n; j is a positive integer greater than or equal to 1 and less than or equal to m; The first index is used to identify a storage location of each data block in the j data blocks in a storage medium and attribute information contained in each data block in the j data blocks. The attribute information comprises at least one of the following: A first attribute, used to represent whether a storage location of a data block in a compressed data block after compression is pre-allocated; A second attribute, used to represent whether a data page of a data block is valid; A third attribute, used to represent whether a data page of a data block is a first compressed page of a compressed data block of the data block; A fourth attribute, used to represent whether a data page of a data block is contained in compressed data pages of two compressed blocks; A fifth attribute, used to represent whether a data page of a data block is a compressed page of a compressed data block after compression of the data block; A sixth attribute, used to represent an index address of a compressed data block in which a data page of a data block is located; A seventh attribute, used to represent, when a data page of a data block belongs to a first compressed page of a compressed data block of the data block, an attribute value of the seventh attribute being an offset of the data block in a set corresponding to the compressed data block; and when the data page of the data block does not belong to the first compressed page of the compressed data block of the data block, the attribute value of the seventh attribute being a distance of the data page of the data block from the first compressed page of the compressed data block.

2. The method of claim 1, wherein, The method comprises: Allocating each data block in the m data blocks to a first set in a preset order in sequence; When a data capacity of the j data blocks in the first set is equal to a rated capacity of the first set, performing a compression operation on the j data blocks according to a set compression threshold, and obtaining the i-th compressed data block.

3. The method of claim 2, wherein, The method comprises: When a sum of a total data length of header data and compressed data of the i-th compressed data block and the set compression threshold is less than or equal to a total data length of the j data blocks, establishing the first index of each data block in the j data blocks.

4. The method according to any one of claims 1 to 3, characterized in that, The attribute information comprises the third attribute, and the method comprises: when the data page of each data block in the j data blocks is the first compressed page of the i-th compressed data block, the attribute value of the third attribute is assigned as 1; when the data page of each data block in the j data blocks is not the first compressed page of the i-th compressed data block, the attribute value of the third attribute is assigned as 0.

5. The method according to any one of claims 1 to 3, characterized in that, The attribute information comprises a seventh attribute, and further comprises: when the attribute value of the third attribute is 1, the attribute value of the seventh attribute is updated as the offset of the data block in the set corresponding to the compressed data block; when the attribute value of the third attribute is 0, the attribute value of the seventh attribute is updated as the distance between the data page of the data block and the first compressed page of the compressed data block.

6. The method according to any one of claims 1-3, characterized in that, The attribute information comprises a fourth attribute, and the establishing of the first index of each data block in the j data blocks corresponding to the i-th compressed data block in the n compressed data comprises: when the data page of each data block in the j data blocks is contained in the compressed data pages of two compressed blocks, the attribute value of the fourth attribute is assigned as 1; when the data page of each data block in the j data blocks is not contained in the compressed data pages of two compressed blocks, the attribute value of the fourth attribute is assigned as 0.

7. The method according to any one of claims 1-3, characterized in that, The attribute information comprises a second attribute, and the establishing of the first index of each data block in the j data blocks corresponding to the i-th compressed data block in the n compressed data comprises: when the data page of each data block in the j data blocks is valid, the attribute value of the second attribute is assigned as 1; when the data page of each data block in the j data blocks is invalid, the attribute value of the second attribute is assigned as 0.

8. The method of any one of claims 1-3, wherein, Before the m data blocks are compressed by using the preset compression algorithm to sequentially obtain n compressed data blocks, the method further comprises: obtaining a second set of to-be-ovewritten data, the second set comprising p compressed data blocks, p being a positive integer greater than or equal to 1; obtaining a compressed page of a first target compressed data in the p compressed data blocks, and q data blocks corresponding to the compressed page of the first target compressed data block, q being a positive integer greater than or equal to 1; determining the positional offset of a first target data block in the q data blocks in the q data blocks; determining the data page of the first target data block as a data page of to-be-ovewritten data.

9. The method of any one of claims 1-3, wherein, The first index is used to identify the storage position of the i-th compressed data block in the storage medium, and the attribute information contained by each data block in the j data blocks.

10. A data compression device, characterized by, The apparatus comprises: a first obtaining unit configured to obtain m data blocks in a data area of a read-write file system, m being a positive integer greater than or equal to 1; a compression unit configured to compress the m data blocks by using a preset compression algorithm to sequentially obtain n compressed data blocks, wherein the first capacity of each compressed data block is the same, the first capacity representing the number of bytes of the data processed by compression that can be contained by a compressed data block, and n is a positive integer greater than or equal to 1; and a first obtaining unit configured to obtain m data blocks in a data area of a read-write file system, m being a positive integer greater than or equal to 1; a compression unit configured to compress the m data blocks by using a preset compression algorithm to sequentially obtain n compressed data blocks, wherein the first capacity of each compressed data block is the same, the first capacity representing the number of bytes of the data processed by compression that can be contained by a compressed data block, and n is a positive integer greater than or equal to 1; and an updating unit, configured to establish a first index of each data block in j data blocks corresponding to an i-th compressed data block in the n compressed data blocks, and record a mapping relationship between the first index and the j data blocks, wherein i is a positive integer greater than or equal to 1 and less than or equal to n, and j is a positive integer greater than or equal to 1 and less than or equal to m; wherein the first index is used to identify a storage location of each data block in the j data blocks in a storage medium, and attribute information contained in each data block in the j data blocks; the attribute information comprises at least one of the following: a first attribute, used to represent whether a storage location of a data block after compression is a pre-allocated compressed data block; a second attribute, used to represent whether a data page of a data block is valid; a third attribute, used to represent whether a data page of a data block is a first compressed page of a compressed data block of the data block; a fourth attribute, used to represent whether a data page of a data block is contained in compressed data pages of two compressed blocks; a fifth attribute, used to represent whether a data page of a data block is a compressed page of a compressed data block of the data block after compression; a sixth attribute, used to represent an index address of a compressed data block in which a data page of a data block is located; a seventh attribute, used to represent, when a data page of a data block belongs to a first compressed page of a compressed data block of the data block, an attribute value of the seventh attribute as an offset of the data block in a set corresponding to the compressed data block, and when the data page of the data block does not belong to the first compressed page of the compressed data block of the data block, the attribute value of the seventh attribute as a distance of the data page of the data block from the first compressed page of the compressed data block.

11. The apparatus of claim 10, wherein, the compression unit is configured to: allocate each data block in the m data blocks to a first set in a preset order in sequence; when a data capacity of the j data blocks in the first set is equal to a rated capacity of the first set, perform a compression operation on the j data blocks according to a set compression threshold, and obtain the i-th compressed data block.

12. The apparatus of claim 11, wherein, the updating unit is configured to: establish the first index of each data block in the j data blocks, when a sum of a total data length of header data and compressed data of the i-th compressed data block and the set compression threshold is less than or equal to a total data length of the j data blocks.

13. The apparatus of any of claims 10-12, wherein, the attribute information comprises the third attribute, and the updating unit is further configured to: assign the attribute value of the third attribute as 1, when a data page of each data block in the j data blocks is a first compressed page of the i-th compressed data block; assign the attribute value of the third attribute as 0, when the data page of each data block in the j data blocks is not the first compressed page of the i-th compressed data block.

14. The apparatus of any one of claims 10-12, wherein, the attribute information comprises the seventh attribute, and the updating unit is further configured to: update the attribute value of the seventh attribute as an offset of the data block in the set corresponding to the compressed data block, when the attribute value of the third attribute is 1; update the attribute value of the seventh attribute as a distance of the data page of the data block from the first compressed page of the compressed data block, when the attribute value of the third attribute is 0.

15. The apparatus of any one of claims 10-12, wherein, The attribute information includes a fourth attribute, and the updating unit is further configured to: assign a value of 1 to the fourth attribute when the data pages of each of the j data blocks are contained in the compressed data pages of two compressed blocks; assign a value of 0 to the fourth attribute when the data pages of each of the j data blocks are not contained in the compressed data pages of two compressed blocks.

16. The apparatus of any one of claims 10-12, wherein, The attribute information includes a second attribute, and the updating unit is further configured to: assign a value of 1 to the second attribute when the data pages of each of the j data blocks are valid; assign a value of 0 to the second attribute when the data pages of each of the j data blocks are invalid.

17. The apparatus of any one of claims 10-12, wherein, Further comprising: a second obtaining unit configured to obtain a second set of to-be- overwritten data, the second set including p compressed data blocks, p being a positive integer greater than or equal to 1; a third obtaining unit configured to obtain a compressed page of first target compressed data in the p compressed data blocks, and q data blocks corresponding to the compressed page of the first target compressed data block, q being a positive integer greater than or equal to 1; a first determining unit configured to determine a position offset of a first target data block in the q data blocks in the q data blocks; a second determining unit configured to determine that a data page of the first target data block is a data page of to-be-overwritten data.

18. The apparatus of any one of claims 10-12, wherein, The first index is used to identify a storage location of the ith compressed data block in a storage medium, and attribute information contained in each of the j data blocks.

19. An apparatus, comprising: Comprise: a unit for performing the data compression method of any one of claims 1 to 9.

20. A computer-readable storage medium, characterized in that, The computer readable storage medium includes computer instructions, when the computer instructions run on an electronic device, make the electronic device execute the data compression method as claimed in any one of claims 1 to 9.

21. A computer program product, characterised in that, When the program is called by the processor, the data compression method of any one of claims 1 to 9 is executed.

22. A chip system, characterized by Comprise one or more processors, when the one or more processors execute instructions, the one or more processors execute the data compression method as claimed in any one of claims 1 to 9.

Images