Data compression method, compressed monitor, electronic device, and storage medium

Abstract

The application provides a data compression method, a compression monitor, an electronic device and a storage medium, and belongs to the technical field of computers, and comprises the following steps: determining a configuration file and a request identifier of a write request according to a description file and a system log, wherein the description file and the system log are generated by a target device receiving the write request and responding; determining a compression mode and a compression algorithm of the write request according to the configuration file; and in the case that it is determined that the write request needs to be compressed based on the compression mode and the request identifier, calling the compression algorithm to compress the write request. The data compression method, the compression monitor, the electronic device and the storage medium provided by the application can read the configuration file of the write request, thereby determining the compression mode and the compression algorithm of the write request, and thereby providing more fine-grained implementation and control for the data compression function of the FileStore backend storage mode.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a data compression method, a compression monitor, an electronic device, and a storage medium. Background Technology

[0002] During Ceph usage, data often exhibits significant redundancy, consuming excessive space and resulting in wasted costs. Compression, which uses smaller amounts of data to represent more data, is an effective way to address this redundancy.

[0003] Currently, lossless compression supports two implementation methods: bluestore and FileStore. In specific scenarios, the FileStore backend storage implementation is more likely to be optimized to adapt to the user's data reading and writing characteristics.

[0004] However, when using FileStore with BTRFS compression, all files in the entire file system will be compressed, resulting in insufficient granularity of compression. Summary of the Invention

[0005] The data compression method, compression monitor, electronic device, and storage medium provided by this invention address the shortcomings of existing technologies where all files in the entire file system are compressed after using FileStore with BTRFS, resulting in insufficient granular compression. This invention enables the determination of the compression mode and algorithm for write requests by reading the configuration file of write requests, thereby providing more granular implementation and control of the data compression function of the FileStore backend storage method.

[0006] This invention provides a data compression method applied to a compression monitor, comprising: Based on the description file and system logs, determine the configuration file and request identifier for the write request, wherein the description file and system logs are generated by the target device receiving and responding to the write request; Based on the configuration file, the compression mode and compression algorithm of the write request are determined; Based on the compression mode and the request identifier, if it is determined that the write request needs to be compressed, the compression algorithm is invoked to compress the write request.

[0007] According to a data compression method provided by the present invention, the step of invoking the compression algorithm to compress the write request based on the compression mode and the request identifier when it is determined that the write request needs to be compressed includes: Based on the compression mode and the request identifier, the determination result of the write request is determined; If the determination result is write request compression, configure the compression ratio and compression target of the write request; the compression target is determined based on the hard disk capacity of the target device. Based on the compression ratio and the compression object, if it is determined that the write request needs to be compressed, the write request is compressed using the compression algorithm.

[0008] According to a data compression method provided by the present invention, determining the write request determination result based on the compression mode and the request identifier includes: When the compression mode is the first mode, the determination result is that the write request is not compressed; If the compression mode is the second mode and the request identifier in the write request is compression, the determination result is that the write request is compressed. If the compression mode is the third mode and the write request does not contain an uncompressed request identifier, the determination result is that the write request is compressed. When the compression mode is the fourth mode, the determination result is determined to be write request compression.

[0009] According to a data compression method provided by the present invention, configuring the compression object of the write request includes the following steps: Determine the maximum object capacity based on the hard drive capacity of the target device; If the write request exceeds the maximum object capacity, the write request is broken down into multiple data blocks to identify the multiple data blocks as compression objects.

[0010] According to a data compression method provided by the present invention, the step of compressing the write request by invoking the compression algorithm based on the compression ratio and the compression object includes: The compression algorithm is invoked to compress each data block separately according to the compression ratio.

[0011] According to a data compression method provided by the present invention, before determining the compression mode and compression algorithm of the write request based on the configuration file, the method further includes: Before determining the compression mode and compression algorithm of the write request based on the configuration file, the method further includes: The following steps are executed iteratively until the above compression mode is determined to be enabled based on the new compression switch threshold: Based on the file system description file and the storage platform system log, the compression switch threshold for the write request is determined; the compression switch threshold is used to indicate whether the compression mode of the write request is enabled or disabled. If the compression mode is determined to be disabled based on the compression switch threshold, an enable command is sent to modify the compression switch threshold in the write request. The new compression switch threshold is determined again based on the file system description file and the storage platform's system logs.

[0012] According to a data compression method provided by the present invention, before determining the configuration file and request identifier of the write request based on the description file and system log, the method further includes: The target device is divided into multiple partitions based on the weight of the local storage modules; the description files in the file system are pointed to the target part of each partition. Align the data storage area of ​​the description file with the target portion.

[0013] The present invention also provides a compression monitor, comprising: The first determining unit is used to determine the configuration file and request identifier of the write request based on the description file and system log, wherein the description file and system log are generated by the target device receiving and responding to the write request; The second determining unit is used to determine the compression mode and compression algorithm of the write request based on the configuration file. The invocation unit is used to invoke the compression algorithm to compress the write request when it is determined that the write request needs to be compressed, based on the compression mode and the request identifier.

[0014] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the data compression method as described above.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the data compression method as described above.

[0016] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements any of the data compression methods described above.

[0017] The data compression method, compression monitor, electronic device, and storage medium provided by this invention determine the compression mode and compression algorithm of the write request by reading the configuration file of the write request, thereby providing a finer-grained implementation and control of the data compression function of the FileStore backend storage method. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 This is one of the flowcharts illustrating the data compression method provided by the present invention; Figure 2 This is one of the application diagrams of the Ceph architecture provided by the present invention; Figure 3 This is the second application diagram of the Ceph architecture provided by the present invention; Figure 4 This is the second flowchart illustrating the data compression method provided by the present invention; Figure 5 This is a schematic diagram of the structure of the compression monitor provided by the present invention; Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0021] In the description of this invention, it should be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this invention, unless otherwise stated, "a plurality of" means two or more.

[0022] The following is combined with Figures 1-6 The present invention describes a data compression method, compression monitor, electronic device, and storage medium provided by embodiments thereof.

[0023] The data compression method provided in this invention can be executed by an electronic device or software, functional module, or functional entity within the electronic device capable of implementing the data compression method. In this invention, the electronic device includes, but is not limited to, a compression monitor. It should be noted that the aforementioned execution entity does not constitute a limitation on this invention.

[0024] Figure 1 This is one of the flowcharts illustrating the data compression method provided by the present invention, such as... Figure 1 As shown, including but not limited to the following steps: First, in step S1, the configuration file and request identifier of the write request are determined based on the description file and system log, which are generated by the target device receiving and responding to the write request.

[0025] The target device can be an NVME device, which is equipped with the Ceph system, and the Ceph system includes multiple OSD modules.

[0026] Figure 2 This is one of the application diagrams of the Ceph architecture provided by the present invention, such as... Figure 2 As shown, in the RADOS architecture, the File that the user needs to store or access is sliced ​​into multiple Objects according to the size of 2M or 4M. Each Object first calculates the corresponding PG through a hash algorithm. Finally, the PG is responsible for mapping the Object to the corresponding OSD module through the Controlled Replication Under Scalable Hashing (CRUSH) algorithm.

[0027] An OSD module can be abstracted into two parts: the system and the daemon. Each OSD module has its own OSD daemon. This daemon is responsible for completing all the logical functions of the OSD module. Since one OSD module typically corresponds to one hard drive, the system part of the OSD occupies some computing power, a certain amount of memory, and one hard drive and network resources, while also having its own independent file system for storing data.

[0028] In the OSD system, each object is treated as a file by the FileStore. The object's attributes (xattr) are accessed using the file's attributes. To compress redundant data within the OSD module, BTRFS can be used as the FileStore's file system to store objects. BTRFS's built-in compression functionality is used to compress data within the OSD module. Transparent file system compression occurs at the file system level; in a transparent file system, compressed files are stored on disk after compression, and to the user, they still appear as ordinary, uncompressed files.

[0029] The description file can be f2fs metadata for a flash-friendly file system (F2FS); the system log can be a Ceph file journal.

[0030] Specifically, the Compression monitor determines the configuration file and request identifier for write requests through the f2fs metadata / Ceph file journal. The configuration file for write requests includes: the size of the blocks to be written, the file format, the data location, the message, and the request description file, etc.

[0031] Optionally, before determining the configuration file and request identifier for the write request based on the description file and system logs, the method further includes: The target device is divided into multiple partitions based on the weight of the local storage modules; the description files in the file system are pointed to the target part of each partition. Align the data storage area of ​​the description file with the target portion.

[0032] The file system can be a flash-friendly file system (F2FS), and the storage module can be an object storage daemon (OSD) module.

[0033] First, parted is used to partition the NVMe device according to the weight of the local OSD module. The weight of the OSD module is based on the number and capacity of the hard drives in the NVMe device. For example, if the NVMe device has 10 hard drives, it will be divided into 10 partitions.

[0034] Divide each partition into SSD_PART1 and SSD_PART2, and designate SSD_PART1 as the target part.

[0035] The SSD_PART1 portion of each partition is used to store osd_journal metadata (cache, log metadata), mounted in the directory / Ceph / Data / Osd / osd-nvme-(device SN)-partx / journa; The SSD_PART2 section is used for OSD device caching (writing data), mounted in the directory / Ceph / Data / Osd / osd-nvme-(device sn)-partx; Point and align the F2FS metadata storage area (CP SIT NAT SSA) to SSD_PART1.

[0036] Specifically, to avoid misalignment between the file system and flash storage, which could affect the performance of NVMe devices, F2FS aligns the starting block address of the CP (Content Processor) with the segment size. Simultaneously, F2FS aligns the starting block address of the Main Area with the Zone size.

[0037] Figure 3 This is the second application diagram of the Ceph architecture provided by this invention, as shown below. Figure 3 As shown, the idea of ​​flexibly changing compression configurations is achieved by using tools to monitor input / output (IO) before data is written to disk. The backend storage, FileStore, uses the F2FS log appending file system on flash devices. The original FileJournal acceleration device, an SSD / NVME, is split into two using parted. Sufficient space is reserved for FileJournal as SSD_PART1 based on the Ceph_osd size, and the remaining SSD / NVME device space is divided into SSD_PART2. SSD_PART2 and the HDD are integrated into a single logical disk mounted on the OSD using the F2FS file system for accelerating actual data writing to disk. SSD_PART1 will host Ceph FileJournal, F2FS metadata, logs, and the Compression Monitor monitoring tool. The Compression Monitor retrieves data from the inode metadata information recorded in SSD_PART1 and, based on the configured algorithm and relevant threshold limits, determines whether a rewrite request needs to be compressed.

[0038] According to the data compression method provided by the present invention, the performance of NVME devices can be improved by pointing the data storage area to SSD_PART1 and aligning it.

[0039] Optionally, before determining the compression mode and compression algorithm of the write request according to the configuration file, the method further includes: The following steps are executed iteratively until the above compression mode is determined to be enabled based on the new compression switch threshold: Based on the file system description file and the storage platform system log, the compression switch threshold for the write request is determined; the compression switch threshold is used to indicate whether the compression mode of the write request is enabled or disabled. If the compression mode is determined to be disabled based on the compression switch threshold, an enable command is sent to modify the compression switch threshold in the write request. The new compression switch threshold is determined again based on the file system description file and the storage platform's system logs.

[0040] The storage platform can be Ceph, and the Compression Monitor can also obtain the compression switch threshold for write requests through the f2fs metadata / Cephfile journal.

[0041] Based on the compression switch threshold, it can be determined whether the compression function is enabled in the F2FS configuration, down to the data block level.

[0042] The data compression method provided by the present invention can support real-time enabling and disabling of compression functions and configuration of compression algorithms, compressing only ongoing write requests, and preventing the reconstruction of data already stored in the OSD module.

[0043] Further, in step S2, the compression mode and compression algorithm of the write request are determined according to the configuration file.

[0044] The compression algorithm and compression mode for write requests can be specified in the configuration file.

[0045] The compression algorithm can be modified in the configuration file for writing requests. This configuration only affects newly written data and does not affect existing data.

[0046] Furthermore, in step S3, based on the compression mode and the request identifier, if it is determined that the write request needs to be compressed, the compression algorithm is invoked to compress the write request.

[0047] The compression monitor can determine whether a write request needs to be compressed based on the compression mode and the request identifier carried by the write request.

[0048] When write requests require compression, the Compression monitor invokes a compression algorithm to compress the write requests.

[0049] Optionally, the step of invoking the compression algorithm to compress the write request based on the compression mode and the request identifier, when it is determined that the write request needs to be compressed, includes: Based on the compression mode and the request identifier, the determination result of the write request is determined; If the determination result is write request compression, configure the compression ratio and compression target of the write request; the compression target is determined based on the hard disk capacity of the target device. Based on the compression ratio and the compression object, if it is determined that the write request needs to be compressed, the write request is compressed using the compression algorithm.

[0050] Based on the write request compression mode and request identifier, a preliminary determination is needed to determine whether the write request needs to be compressed, and the result of the write request determination is obtained.

[0051] If the judgment result determines that write request compression is required, configure the compression ratio and compression object for the write request.

[0052] The data compression method provided by this invention determines the compression mode and compression algorithm of the write request by reading the configuration file of the write request, thereby providing a more granular implementation and control of the data compression function of the FileStore backend storage method.

[0053] Optionally, determining the write request result based on the compression mode and the request identifier includes: When the compression mode is the first mode, the determination result is that the write request is not compressed; If the compression mode is the second mode and the request identifier in the write request is compression, the determination result is that the write request is compressed. If the compression mode is the third mode and the write request does not contain an uncompressed request identifier, the determination result is that the write request is compressed. When the compression mode is the fourth mode, the determination result is determined to be write request compression.

[0054] The compression monitor determines whether to compress a write request based on the compression mode and the request hint included in each write request. Specifically, this includes: The compression mode is set to the first mode (None), and write requests are not compressed. The compression mode is the second mode (Passive). Unless the write request contains a compression hint, the write request will not be compressed. The compression mode is the third (Aggressive) mode. Unless there is a hint in the write request that it should not be compressed, all write requests will be compressed. The compression mode is the fourth mode (Force), which compresses all write requests regardless of request identifier, compression ratio, or minimum object size.

[0055] According to the data compression method provided by the present invention, by reading the configuration file of write requests that support three compression modes: None, Passive, Aggressive and Force, it is possible to determine whether the write request needs to be compressed, thereby achieving finer-grained control.

[0056] Optionally, configuring the compression object for the write request includes the following steps: Determine the maximum object capacity based on the hard drive capacity of the target device; If the write request exceeds the maximum object capacity, the write request is broken down into multiple data blocks to identify the multiple data blocks as compression objects.

[0057] The Compression monitor configures the maximum object size (Compression Max blob Size): If the write request block to be compressed is larger than the maximum object size, the write request will be divided into several data blocks and compressed separately.

[0058] The maximum object capacity can be set to different sizes for hard disk drives and solid-state drives.

[0059] In addition, a minimum object size (Compression Min blob Size) can be set. If the write request is smaller than the minimum object size, the write request will not be compressed, but the original data of the write request will be stored directly, which can effectively avoid the redundancy of compression operations.

[0060] For example, if the compression ratio of a write request is equal to or lower than that compression ratio, and the write request is larger than the minimum object capacity, a determination is made as to whether the write request needs to be compressed based on the compression mode and the request identifier.

[0061] According to the data compression method provided by the present invention, by dividing excessively large data into segments before compression, the data can be effectively compressed while ensuring that it does not exceed the capacity of a single hard drive.

[0062] Optionally, the step of invoking the compression algorithm to compress the write request based on the compression ratio and the compression object includes: The compression algorithm is invoked to compress each data block separately according to the compression ratio.

[0063] The compression ratio can be flexibly configured according to actual needs. The compression ratio is the ratio of the size of the compressed data block to the size of the original data. Subsequent judgments are only made if the compression ratio of the write request is equal to or lower than this compression ratio; otherwise, if the compression ratio of the write request does not reach the set compression ratio, the write request will not be compressed, and the original write request will be stored directly to avoid invalid compression.

[0064] For example, if the ratio is set to 0.7, then during write operations, only data that is 70% or less of the original data after compression will be compressed and stored.

[0065] According to the data compression method provided by the present invention, each data block is compressed by calling an algorithm to ensure that the capacity of a single hard disk is not exceeded.

[0066] In related technologies, BTRFS, as the backend storage file system for FileStore, does not support online modification of compression algorithms and lacks tools to monitor and statistically analyze Ceph data persistence, making it impossible to determine which data needs compression and how to control the compression ratio. This invention solves the configuration and monitoring problems of online data compression in the FileStore backend storage method.

[0067] Figure 4 This is the second flowchart illustrating the data compression method provided by this invention. For users who may require finer-grained control, such as using different compression algorithms for different directories, or disabling compression, embodiments of this invention also provide a data compression method, such as... Figure 4 As shown, it includes: First, the FileStore backend storage is dominated by the F2FS file system; within the OSD partition, parted is used to equally divide the capacity of the NVME device according to the local OSD weight; within the NVME partition, each node reserves FileStore space PART1 according to the OSD capacity, and the remaining space is used for OSD cache PART2; F2FS specifies the meta inode: pointing the F2FS metadata storage area (CPSIT NAT SSA) to PART1; Secondly, Compression Monitor monitors F2FS metadata / ceph FileJournal, enabling real-time monitoring of write requests. It captures and records filejournal metadata and write request information in the SSD in real time. Through the Compression Monitor tool, it interacts with the F2FS file system configuration to achieve more comprehensive compression functionality for the FileStore backend storage method. It also allows for flexible configuration of the F2FS file system compression function for the FileStore backend storage. Next, if the ceph FileJourmal does not have a compression flag, it is determined that the write request will not be compressed; if the ceph FileJourmal has a compression flag, the compression algorithm is read: compress="cat" / "gzip" / "bzip2" / "lzam" / "xz" / "lzo" / "lz4" / "zstd". After specifying the compression algorithm, F2FS begins to decompress the write request. Then, the compression mode is determined. If the compression mode is None, the write request will not be compressed; if the compression mode is Passive, no compression will be performed unless the write request contains a compression hint; if the compression mode is Aggressive, all write requests will be compressed unless there is a non-compression hint in the write request; if the compression mode is Force, all other write requests will be compressed regardless of the other modes. The Compression Monitor obtains filejournal, metadata, and write request identifiers from the Ceph layer to confirm data write information. It supports three compression modes: Passive, Aggressive, and Force, and clearly identifies which data needs to be compressed. Furthermore, for write requests that need to be compressed, configure the compression ratio. If the compression ratio of a write request does not reach the set compression ratio after compression, the write request will not be compressed and the original write request will be stored directly. The compression ratio threshold can be flexibly changed to compress only write requests with a high compression ratio. If the compressed write request size is no larger than the compression ratio, configure the compression biob size. If the write request size is smaller than the minimum biob size, the write request will not be compressed; instead, the original data of the write request will be stored directly. If the write request size is larger than the maximum biob size, the write request will not be compressed; instead, the original data of the write request will be stored directly. If neither of these conditions applies, the compression algorithm will be called to compress the write request. By setting a threshold, data blocks exceeding the threshold will not be compressed.

[0068] According to the data compression method provided by the present invention, the Compression Monitor interacts with the compression function of the F2FS file system through parameters such as on / off status, compression algorithm, compression ratio, and data block size to achieve multi-dimensional control of the compression function of the FileStore backend storage. This can solve the problem that previous file systems such as BTRFS, as FileStore backends, could not control the compression function in real time and with fine granularity.

[0069] The compression monitor provided by the present invention is described below. The compression monitor described below can be referred to in correspondence with the data compression method described above.

[0070] Figure 5 This is a schematic diagram of the structure of the compression monitor provided by the present invention, as shown below. Figure 5 As shown, it includes: The first determining unit 501 is used to determine the configuration file and request identifier of the write request based on the description file and system log, wherein the description file and system log are generated by the target device receiving and responding to the write request; The second determining unit 502 is used to determine the compression mode and compression algorithm of the write request according to the configuration file; The calling unit 503 is used to call the compression algorithm to compress the write request when it is determined that the write request needs to be compressed, based on the compression mode and the request identifier.

[0071] During the operation of the compression monitor, the first determining unit 501 determines the configuration file and request identifier of the write request based on the description file and system log, wherein the description file and system log are generated by the target device receiving and responding to the write request; the second determining unit 502 determines the compression mode and compression algorithm of the write request based on the configuration file; and the calling unit 503, based on the compression mode and the request identifier, calls the compression algorithm to compress the write request if it is determined that the write request needs to be compressed.

[0072] The compression monitor provided by this invention determines the compression mode and compression algorithm of the write request by reading the configuration file of the write request, thereby providing a more granular implementation and control of the data compression function of the FileStore backend storage method.

[0073] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 6As shown, the electronic device may include: a processor 610, a communications interface 620, a memory 630, and a communications bus 640. The processor 610, the communications interface 620, and the memory 630 communicate with each other through the communications bus 640. The processor 610 is used to execute executable modules, such as computer programs, stored in the memory 630.

[0074] The memory 630 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 620 (which can be wired or wireless), such as the Internet, wide area network, local area network, metropolitan area network, etc.

[0075] The communication bus 640 can be an ISA bus, PCI bus, or EISA bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0076] The memory 630 is used to store programs. After receiving an execution instruction, the processor 40 executes the program. The method executed by the compression monitor of the stream process definition disclosed in any of the foregoing embodiments of the present invention can be applied to the processor 610, or implemented by the processor 610.

[0077] The processor 610 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 610 or by instructions in software form. The processor 610 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this invention can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 630. Processor 610 reads information from memory 630 and, in conjunction with its hardware, executes a data compression method. The method includes: determining a configuration file and a request identifier for a write request based on a description file and a system log, wherein the description file and system log are generated by the target device receiving and responding to a write request; determining a compression mode and a compression algorithm for the write request based on the configuration file; and, based on the compression mode and the request identifier, invoking the compression algorithm to compress the write request if it is determined that the write request needs to be compressed.

[0078] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the compression monitor and electronic device described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0079] The compression monitor and electronic device provided in this embodiment of the invention have the same technical features as the data compression method provided in the above embodiments, so they can also solve the same technical problems and achieve the same technical effects.

[0080] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of methods and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0081] The computer program product for the data compression method provided in this embodiment of the invention includes a computer-readable storage medium storing non-volatile program code executable by a processor. The instructions included in the program code can be used to execute the methods described in the preceding method embodiments. For specific implementation details, please refer to the method embodiments, which will not be repeated here.

[0082] In the several embodiments provided by this invention, it should be understood that the disclosed compression monitor and method can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of the above units is only a logical functional division, and there may be other division methods in actual implementation. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed may be through some communication interface; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0083] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0084] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0085] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0086] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the data compression method provided by the above methods. The method includes: determining a configuration file and a request identifier for a write request based on a description file and a system log, wherein the description file and the system log are generated by a target device receiving and responding to a write request; determining a compression mode and a compression algorithm for the write request based on the configuration file; and, based on the compression mode and the request identifier, invoking the compression algorithm to compress the write request when it is determined that the write request needs to be compressed.

[0087] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When executed by a processor, the computer program is implemented to perform the data compression methods provided by the methods described above. The method includes: determining a configuration file and a request identifier for a write request based on a description file and a system log, wherein the description file and the system log are generated by a target device receiving and responding to a write request; determining a compression mode and a compression algorithm for the write request based on the configuration file; and, based on the compression mode and the request identifier, invoking the compression algorithm to compress the write request if it is determined that the write request needs to be compressed.

[0088] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0089] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0090] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features within the scope of the technology disclosed in the present invention. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data compression method, characterized in that, Applications in compression monitors include: The target device is divided into multiple partitions based on the weight of the local storage modules; each of the multiple partitions is divided into a first part and a second part; the first part is designated to store log metadata, and the second part is designated to provide device caching; the metadata storage area of ​​the file system is pointed to the first part; the first part is also used to host the compression monitor. Point the description file in the file system to the target part of each partition; Align the data storage area of ​​the description file with the target portion; The following steps are executed iteratively until the compression mode is determined to be enabled based on a new compression switch threshold: The compression switch threshold for write requests is determined based on the file system description file and the storage platform's system logs; the compression switch threshold is used to indicate whether the compression mode of the write request is enabled or disabled; if the compression mode is determined to be disabled based on the compression switch threshold, an enable command is sent to modify the compression switch threshold in the write request; a new compression switch threshold is determined again based on the file system description file and the storage platform's system logs. By obtaining data from the inode metadata information recorded in the first part, and determining the configuration file and request identifier of the write request based on the description file and system log, wherein the description file and system log are generated by the target device receiving and responding to the write request; Based on the configuration file, the compression mode and compression algorithm of the write request are determined; Based on the compression mode and the request identifier, if it is determined that the write request needs to be compressed, the compression algorithm is invoked to compress the write request, including: Based on the compression mode and the request identifier, the determination result of the write request is determined; If the determination result is write request compression, configure the compression ratio and compression target of the write request; the compression target is determined based on the hard disk capacity of the target device. Based on the hard disk capacity of the target device, determine the maximum object capacity; if the write request exceeds the maximum object capacity, decompose the write request into multiple data blocks to determine the multiple data blocks as compression objects; Based on the compression ratio and the compression object, if it is determined that the write request needs to be compressed, the compression algorithm is invoked to compress each data block according to the compression ratio. The determination of the write request result based on the compression mode and the request identifier includes: When the compression mode is the first mode, the determination result is that the write request is not compressed; If the compression mode is the second mode and the request identifier in the write request is compression, the determination result is that the write request is compressed. If the compression mode is the third mode and the write request does not contain an uncompressed request identifier, the determination result is that the write request is compressed. When the compression mode is the fourth mode, the determination result is determined to be write request compression.

2. A compression monitor for performing the data compression method as described in claim 1, characterized in that, include: The first determining unit is used to divide the target device into multiple partitions based on the weight of the local storage modules. Each of the plurality of partitions is divided into a first part and a second part; the first part is designated to store log metadata, and the second part is designated to provide device caching; Point the file system's metadata storage area to the first part; Point the description file in the file system to the target part of each partition; Align the data storage area of ​​the description file with the target portion; The first determining unit is used to determine the configuration file and request identifier of the write request based on the description file and system log, wherein the description file and system log are generated by the target device receiving and responding to the write request; The second determining unit is used to determine the compression mode and compression algorithm of the write request based on the configuration file. The calling unit is configured to, based on the compression mode and the request identifier, call the compression algorithm to compress the write request when it is determined that the write request needs to be compressed. This includes: determining a judgment result for the write request based on the compression mode and the request identifier; configuring the compression ratio and compression object of the write request when the judgment result indicates that the write request needs compression; the compression object is determined based on the hard disk capacity of the target device; and compressing the write request using the compression algorithm when it is determined that the write request needs to be compressed based on the compression ratio and the compression object. Specifically, determining the judgment result for the write request based on the compression mode and the request identifier includes: determining that the write request is not compressed when the compression mode is a first mode; determining that the write request is compressed when the compression mode is a second mode and the request identifier in the write request indicates compression; determining that the write request is compressed when the compression mode is a third mode and the write request does not contain an uncompressed request identifier; and determining that the write request is compressed when the compression mode is a fourth mode.

3. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the data compression method as described in claim 1.

4. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the data compression method as described in claim 1.

Images