Data compression method, device, equipment and system and storage medium

Abstract

The invention provides a data compression method, device, equipment and system and a storage medium, which are used for compressing data and then storing the data to block equipment, and the block equipment comprises at least one block. The method comprises the following steps: acquiring a corresponding relationship between a block group and a compression algorithm; wherein the block group comprises one block or a plurality of continuous blocks, and the historical access frequency of the block group is in positive correlation with the compression speed of the compression algorithm; in response to a data write-in operation, executing the following compression process: acquiring an access address of the data write-in operation, and determining a target block where the access address is located; and compressing to-be-written data by using a target compression algorithm corresponding to the target block group to which the target block belongs according to the corresponding relationship between the block group and the compression algorithm.

Description

Technical Field

[0001] The present disclosure relates to the field of data compression technology, and in particular to a data compression method, device, equipment, system and storage medium. Background Art

[0002] Data compression processing is applied in many scenarios. Taking cloud computing as an example, a block storage cluster is a unit of cloud computing block storage system deployment, which consists of many physical machines. These physical machines are connected to the network to provide block device services to the outside. A certain number of block devices are mounted on the physical machine, which can provide storage services to the outside. At present, each physical machine in the cluster uses a unified compression algorithm. This processing method is easy to implement and manage, but it also has obvious defects. Different compression algorithms have different compression rates. The higher the compression rate, the greater the performance consumption. If there are a few hot blocks in the cluster, using an algorithm with a higher compression rate will cause the cluster CPU (Central Processing Unit) to consume too much, which may lead to resource squeeze and slow I / O (Input / Output) response. If a compression algorithm with a faster compression speed is conservatively used, since the compression algorithm with a faster compression speed usually has a lower compression rate, it will lead to poor compression effect and waste of space. Summary of the invention

[0003] To overcome the problems existing in the related art, the present disclosure provides a data compression method, device, equipment, system and storage medium.

[0004] According to a first aspect of an embodiment of this specification, a data compression method is provided, for compressing data and storing it in a block device, wherein the block device includes at least one block; the method includes:

[0005] Acquire a correspondence between a block group and a compression algorithm; wherein the block group includes one block or multiple consecutive blocks, and a historical access frequency of the block group is positively correlated with a compression speed of the compression algorithm;

[0006] In response to a data write operation, the following compression process is executed: an access address of the data write operation is obtained, and a target block where the access address is located is determined; based on the correspondence between the block grouping and the compression algorithm, the data to be written is compressed using a target compression algorithm corresponding to the target block group to which the target block belongs.

[0007] According to a second aspect of an embodiment of this specification, there is provided a data compression apparatus for compressing data and storing it in a block device, wherein the block device includes at least one block; the apparatus includes:

[0008] The acquisition module is used to: acquire the corresponding relationship between the block grouping and the compression algorithm; wherein the block grouping includes one block or multiple consecutive blocks, and the historical access frequency of the block grouping is positively correlated with the compression speed of the compression algorithm;

[0009] A compression module is used to: in response to a data writing operation, execute the following compression process: obtain the access address of the data writing operation, determine the target block where the access address is located; based on the correspondence between the block grouping and the compression algorithm, compress the data to be written using the target compression algorithm corresponding to the target block group to which the target block belongs.

[0010] According to a third aspect of an embodiment of this specification, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the method embodiment described in the first aspect are implemented.

[0011] According to a third aspect of an embodiment of this specification, a distributed system is provided, the distributed system comprising:

[0012] A computer device for determining a correspondence between block groupings and compression algorithms; and

[0013] One or more computer devices for executing the data compression method described in the first aspect.

[0014] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method embodiment described in the first aspect are implemented.

[0015] The technical solutions provided by the embodiments of this specification may have the following beneficial effects:

[0016] In the embodiments of this specification, the same fixed compression algorithm is not used for each block in the block device. Instead, the access popularity of the block is measured based on the historical access frequency of the block, and the compression algorithm used for the block is determined based on this. Therefore, the compression algorithm of the block can dynamically change with the access frequency, and can maximize the overall data compression rate while ensuring the I / O quality, thereby achieving a higher space utilization efficiency.

[0017] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the specification and, together with the description, serve to explain the principles of the disclosure.

[0019] Figure 1 It is a flow chart of a data compression method shown in this specification according to an exemplary embodiment.

[0020] Figure 2 This is a schematic diagram of a data compression scenario shown in this specification according to an exemplary embodiment.

[0021] Figure 3 It is a hardware structure diagram of a computer device where a data compression device is located according to an exemplary embodiment of this specification.

[0022] Figure 4 It is a block diagram of a data compression device according to an exemplary embodiment of the present specification. DETAILED DESCRIPTION

[0023] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with this specification. Instead, they are merely examples of devices and methods consistent with some aspects of this specification as detailed in the appended claims.

[0024] The terms used in this specification are for the purpose of describing specific embodiments only and are not intended to limit this specification. The singular forms "a", "the" and "the" used in this specification and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0025] It should be understood that although the terms first, second, third, etc. may be used in this specification to describe various information, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of this specification, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0026] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0027] Take the cloud computing scenario as an example. In the cloud computing environment, the block storage cluster is the unit of cloud computing block storage system deployment, which is composed of many physical machines. These physical machines are connected to the network to provide block device services to the outside. A certain number of block devices are mounted on the physical machine, and these block devices can provide storage services to the outside. When the number of block devices is fixed, that is, when the number of machines in the entire cluster remains unchanged, the space that can be used for external services is fixed. Therefore, in order to make the most of the cluster space, it is necessary to compress the data stored in the block devices.

[0028] The usual compression method is: a fixed compression algorithm is set on the physical machine that manages the block device, such as Lz4 (Lempel-Ziv 4th generation compression algorithm) and Zstd (Zstandard, a compression algorithm) and other commonly used compression algorithms, to compress the data written to the block device to save cluster space. For example, in a block device cluster using the LSBD (Log Structured Block Device, a block device based on log structure) architecture, the physical machine compresses the data when dumping data to reduce the impact on I / O latency. Specifically, LSBD uses a data organization method similar to that of a log file system, and writes data to the block device in the form of a log in sequence. The working principle of LSBD is to write write requests to the log in an appended manner instead of directly writing the original data block. Over time, the data will be collected and organized to form a series of larger data blocks, and then written to the storage medium in a more efficient way. Under the LSBD architecture, GC (Garbage Collection) is a mechanism for managing and releasing storage space that is no longer in use. When performing a GC operation, the physical machine triggers the data dump process and compresses the data and stores it in the block device.

[0029] At present, each physical machine in the cluster uses a unified compression algorithm. This processing method is easy to implement and manage, but it also has obvious defects. Different compression algorithms have different compression speeds and different compression rates; generally, the higher the compression rate, the greater the performance consumption and the slower the compression speed. If there are a few hot blocks in the cluster, using an algorithm with a higher compression rate will result in a slower compression speed, which will in turn lead to excessive CPU consumption in the cluster, which may lead to resource squeeze and slow I / O response. If a compression algorithm with a lower compression rate is conservatively used, it will result in poor compression effect and waste of space.

[0030] Based on this, the embodiment of this specification provides a data compression solution, which does not use the same fixed compression algorithm for each block in the block device, but measures the access popularity of the block based on the historical access frequency of the block, and determines the compression algorithm used by the block based on this. Therefore, the compression algorithm of the block can change dynamically with the access frequency, and can improve the overall data compression rate as much as possible while ensuring the I / O quality, so as to achieve a higher space utilization efficiency. Next, the embodiment of this specification is described in detail.

[0031] Figure 1 The present specification is a flowchart of a data compression method according to an exemplary embodiment. The method can be used to compress data and store it in a block device. The block device includes at least one block. The method may include the following steps:

[0032] Step 102: Obtain the correspondence between block grouping and compression algorithm.

[0033] The block group includes one block or multiple continuous blocks, and the historical access frequency of the block group is positively correlated with the compression speed of the compression algorithm.

[0034] Step 104, in response to the data write operation, execute the following compression process: obtain the access address of the data write operation, and determine the target block where the access address is located; according to the correspondence between the block grouping and the compression algorithm, compress the data to be written using the target compression algorithm corresponding to the target block group to which the target block belongs.

[0035] The data compression scheme of this embodiment can be applied to various scenarios. For example, in a single-machine scenario, the data compression method can be applied to a physical machine that can manage one or more block devices and process data of some or all blocks in the block devices.

[0036] It can also be applied to cluster scenarios, where the cluster includes multiple physical machines. In some examples, each physical machine executes the solution of this embodiment, and each physical machine processes the data of the blocks in each block device it manages. In other examples, it can also be applied to a management node in a cluster, where the cluster includes one or more physical machines, and each physical machine can manage one or more block devices. With the changes of the physical machines in the cluster, the block devices managed by each physical machine may change. Therefore, the management node can execute the solution of this embodiment to determine the compression algorithm corresponding to each block group under each block device.

[0037] In this embodiment, a block device is a computer storage device that uses a block storage method, and data is read and written in blocks of fixed size. Block devices generally include hard disk drives, flash drives (such as solid-state drives), or optical disk drives, etc., which are used to store data, software, operating systems, etc., and are managed and accessed through a file system. Block devices can also be used as storage media in a virtualized environment for sharing and access by one or more virtual machines.

[0038] In this embodiment, the "block" in the block device refers to a group of continuous physical storage units, which can be physical blocks or logical blocks. Logical blocks refer to abstract blocks managed by the file system, which are divided and managed in units of the block size of the file system and are used to represent data or metadata in the file system. The size of a logical block is usually 4KB (Kilobyte) or larger, which can be set when formatting or mounting the file system, and this embodiment does not limit this. Applications and file systems perform read and write operations through logical blocks without considering the specific details of the underlying hardware storage. Physical blocks refer to actual blocks on the storage medium, also known as storage units such as sectors and pages, which are the smallest physical storage units. The size of physical blocks is usually 512 bytes (Byte), 1KB (KiloByte), 2KB, 4KB, etc., and is managed and controlled by the hardware of the storage medium and its driver. The file system will align and organize according to the size of the physical blocks to improve I / O performance and reduce waste. In a block device, there is a mapping relationship between logical blocks and physical blocks, that is, logical blocks can only be read and written after being mapped to physical blocks.

[0039] In this embodiment, the historical access record of the block device can be obtained through the access operation of the block device. The access operation of the block device is an I / O (Input or Output) operation, which generally includes two operations: read and write. The read operation refers to reading data from the block device into the memory. It is usually implemented through a file system interface or a system call, and the application can use the read function to initiate a read request. When performing a read operation, the operating system will first determine the location and size of the block to be read, and then read the block from the device into the system cache. When the read is completed, the operating system will copy the data from the system cache to the user cache for use by the application. The write operation refers to writing data from the memory to the block device. It is also implemented through a file system interface or a system call, and the application can use the write function to initiate a write request. When performing a write operation, the operating system will first copy the data from the user cache to the system cache, and then write the block from the system cache to the device. When the write is completed, the operating system will return a success or failure message to the application.

[0040] The correspondence between the block grouping and the compression algorithm in step 102 can be obtained in a variety of ways, for example, it can be pre-configured or obtained in an automated manner, for example, it can be configured based on the historical access frequency of the block grouping.

[0041] Among them, the block grouping can be a block or multiple continuous blocks. Among them, considering that block devices are usually managed in blocks of 4KB size, it is called a logical block address (Logical Block Address, LBA). The block grouping can be a block size. In this way, when the block device is large, for example, the block device is generally at the GB level, the amount of calculation of the access frequency to be calculated will be large. It is more difficult to calculate the access frequency for each LBA. Therefore, it is necessary to group the LBAs, and divide each adjacent 256 LBAs into a logical block address group, called an LBA Group. Then, the size of each LBA Group is 1MB, which greatly reduces the amount of calculation. Of course, the 256 here are just examples, and they can actually be configured as needed. This embodiment does not limit this.

[0042] In this embodiment, after the data is compressed using a compression algorithm, a flag bit recording the compression algorithm identifier will be included in the compressed data; when the data is read, the flag bit can be used to decompress the data using the corresponding compression algorithm.

[0043] There may be one or more compression algorithms; multiple compression algorithms with different compression speeds may be pre-configured. In practical applications, the compression speed of a compression algorithm is usually related to the compression ratio of the compression algorithm. The greater the compression ratio, the slower the compression speed; of course, it is not excluded that the compression speed is positively correlated with the compression ratio. In this embodiment, the more frequently a block group is accessed, the faster the compression speed of the block group should be to meet the access requirements.

[0044] In some examples, the corresponding relationship may be determined in the following manner:

[0045] Acquire historical access records of at least one of the block devices, and determine historical access frequencies of a plurality of block groups based on the acquired historical access records;

[0046] Dividing each block group into multiple categories using the historical access frequency of each block group;

[0047] Determine the compression algorithm corresponding to each block group under the category.

[0048] The information included in the historical access record may include one or more of the following information: access time, type of access operation (read or write), access address or access length, etc. The historical access frequency of the block group can be determined through the historical access record, and the historical access frequency here can be the number of accesses in a time period. As needed, the access operation may include a read operation or a write operation, so it can also be the number of read operations, the number of write operations, or the total number of read operations and write operations.

[0049] After obtaining the historical access frequency of each block group, each block group can be divided into multiple categories; the division method can be implemented in multiple ways, for example, each block group can be sorted according to the historical access frequency according to the number of categories, and divided into multiple categories according to the sorting results, and then each category includes one or more block groups, and a corresponding compression algorithm is configured under each category. Based on this, an accurate correspondence between block groups and compression algorithms can be obtained.

[0050] There are also multiple ways to determine the historical access frequency of a block group. For example, the historical access record may include: an access address and an access length; determining the historical access frequency of multiple block groups may include:

[0051] Determine the access scope of each historical access record according to the access address and access length in each historical access record;

[0052] According to the address range of each block group and the access range of each historical access record, the historical access frequency of each block group is counted.

[0053] In this embodiment, an access operation does not necessarily involve only one block. The starting block of the access operation can be determined according to the access address, and the number of blocks targeted by the access operation can be determined according to the access length, based on which the access range of the access operation can be determined. Based on this, the access frequency of each block group can be counted according to the address range of the divided block group and the access range of each historical access record. Based on this, the access frequency of the block group can be quickly counted and obtained accurately.

[0054] In practical applications, the timing of determining the access frequency of the block group can be configured as needed, for example, it can be determined after receiving a command initiated by a user, or it can be automatically triggered according to a set time, or it can be based on a set statistical time event. For example, obtaining the historical access record of at least one of the block devices may include:

[0055] In response to a statistical event, obtaining a historical access record generated by at least one of the block devices from the last statistical time to the current time; wherein each historical access record is generated and stored after each access operation to the block device occurs;

[0056] According to the acquired historical access records, historical access frequencies of multiple block groups are counted.

[0057] The statistical events of this embodiment may include multiple implementation methods. The statistical events may include statistical events initiated by users through instructions or events reaching a predetermined statistical time, etc., which can be configured as needed in actual applications. In this embodiment, each historical access record can be generated and stored after each access operation to a block device occurs; after a statistical event occurs, the historical access records generated from the last statistical time to the current time can be obtained, so that the historical access frequency of each block group can be counted based on the obtained historical access records. Based on this, the required statistical events can be set as needed to meet customized statistical needs.

[0058] In some examples, dividing each block group into multiple categories using the historical access frequency of each block group includes: obtaining the number of compression algorithms, and clustering the each block group into the number of categories using the historical access frequency of each block group by a clustering algorithm.

[0059] In this embodiment, in order to obtain the correspondence between the block groups and the compression algorithms, the clustering algorithm can be used to classify the block groups. The number of the divided categories can be the number of compression algorithms, and the clustering algorithm uses the historical access frequency of each block group to cluster the block groups into the number of categories.

[0060] Among them, the selection of clustering algorithm can be configured as needed. Clustering algorithm is a commonly used unsupervised learning algorithm for dividing a set of data objects into several similar, closely related clusters. In this embodiment, the clustering algorithm can select K-means clustering, which can divide the data objects into K clusters, each cluster has similar characteristics, where K is the number of clusters specified by the user. It can also include a density clustering algorithm, which can find clusters of arbitrary shapes based on the density of data points.

[0061] It can also include the Natural Breaks Algorithm, also known as the Jenks Natural Breaks Algorithm. It can divide the data into multiple non-intersecting groups (also called categories, intervals, or breaks), with the data within each group having similar characteristics and the data between groups having significant differences. By using the Natural Breaks Algorithm, a large amount of continuous data can be converted into several roughly equal groups, thereby better understanding and representing the data.

[0062] The principle of the natural breakpoint algorithm is to find the best grouping method for a given data set through trial and error. First, based on the distribution of the data itself, an initial value is selected as the breakpoint of the first group, and then the data is grouped and the average value of the data within the group is calculated. Next, new breakpoints are inserted between the groups and the average values ​​of the groups are recalculated until the sum of squared deviations between the average values ​​of all groups and the breakpoints of their groups is minimized. This process is equivalent to finding a suitable set of breakpoints so that the variability within each group is small, the variability between groups is large, and as few breakpoints as possible are used. As an example, the process can be:

[0063] Sort the data, and let the sorted data be v1, v2, ..., vn. Here, each of the n pieces of data is the access frequency of each block group.

[0064] If it is determined that k categories need to be divided, then there are k-1 discontinuity points. Traverse all possible discontinuity points p1, p2, ..., pk-1, so that p1 <= p2 <= ... <= pk-1, and calculate the sum of square deviations sdemnow corresponding to each discontinuity point; specifically: divide the data into k parts, the i-th part contains elements between pi and pi+1-1 (i.e., the access frequency of the block group), calculate the average value of each part, and then calculate the sum of square deviations between the internal elements of each part and the average value, and then add these sums of square deviations to get sdemnow.

[0065] Select the set of discontinuity points that minimizes sdemnow and obtain the classification result.

[0066] In this way, multiple block groups can be quickly divided into categories through the above algorithm. Each category includes one or more block groups, and a corresponding compression algorithm can be configured for each category; optionally, after being divided into multiple categories, the overall access frequency under each category is different, and the compression algorithm can be determined based on the overall access frequency of each category. The higher the overall access frequency, the faster the compression speed of the compression algorithm. The overall access frequency here can be determined based on a variety of methods, such as the mean of the historical access frequencies of all block groups under the category, or the central value of each category in the clustering algorithm. The calculation method of the central value may be different under different clustering algorithms, such as the sum of squared deviations used in the above-mentioned natural break point algorithm.

[0067] In this embodiment, in a single machine scenario or a cluster scenario, the physical machine can determine the correspondence between multiple block groups and compression algorithms under the block device it manages. In a cluster scenario, one physical machine can also determine the correspondence between multiple block groups and compression algorithms under the block devices managed by some or all physical machines in the cluster; in addition, this embodiment can be performed continuously, that is, the correspondence can be determined multiple times, and the physical machine that determines the correspondence each time can be different; this embodiment does not limit this.

[0068] In actual applications, there may be multiple architectures of physical machines in cluster scenarios, such as a storage-computing integrated architecture or a storage-computing separated architecture. In the storage-computing integrated architecture, computing nodes and storage nodes are highly integrated; the data that needs to be accessed for computing is on the local storage node, without the need for remote access, which is equivalent to the above-mentioned single-machine scenario in the storage-computing integrated architecture. In the storage-computing separated architecture, computing nodes and storage nodes are isolated and connected through a network. Computing nodes need to access remote storage nodes to read data. The advantage of this architecture is high flexibility, computing or storage can be upgraded independently, and the configuration and scale can be dynamically adjusted as needed. In the storage-computing separated architecture, the physical machine cluster of this embodiment is a storage cluster, and the physical machine is a storage node. The cluster includes multiple block devices, and each physical machine (storage node) can manage one or more block devices, and the managed block devices can be flexibly changed as needed.

[0069] In some examples, the method is applied to a physical machine in a physical machine cluster, which manages at least one block device; obtaining the correspondence between block grouping and compression algorithm includes: obtaining the correspondence between block grouping and compression algorithm determined by a control physical machine that serves as a control node in the physical machine cluster; the method also includes: in response to an access operation to the block device, generating an access record to provide to the control physical machine to determine the correspondence.

[0070] Among them, the physical machine cluster may include one or more physical machines, one of which may serve as a control node. The physical machine serving as the control node may be fixed or may be flexibly changed as required. The method for determining the control node in the cluster may be configured as required, and this embodiment does not limit this. Through the above method, the control node may comprehensively consider the access conditions of each block group under the block devices managed by multiple physical machines in the physical machine cluster to determine the compression algorithm corresponding to each block group.

[0071] In some examples, obtaining the correspondence between the block grouping and the compression algorithm determined by the control physical machine as the control node in the physical machine cluster may include:

[0072] Obtaining a comparison table sent by a control physical machine as a control node in the physical machine cluster, wherein the comparison table records a correspondence between each block group under each block device in the physical machine cluster and a compression algorithm;

[0073] The corresponding relationship between each block group and the compression algorithm under each block device managed by itself is determined from the comparison table.

[0074] In this embodiment, the records in the comparison table determined by the control node may include, as an example, the identifier of the block group under the block device and the identifier of the compression algorithm corresponding thereto; or, the category corresponds to the compression algorithm, or may be the identifier of the category. Based on this, each time the comparison table is determined, the control node may send the comparison table to each physical machine in the cluster, so that each physical machine determines the correspondence between each block group and the compression algorithm under each block device it manages based on the comparison table; this method may be applicable to the situation where the block devices managed by the physical machines in the cluster may change, so that the physical machines can obtain the accurate correspondence through the comparison table.

[0075] In some examples, generating the access record to provide to the managed physical machine to determine the corresponding relationship includes:

[0076] An access record is generated and stored in a database, so that the management and control physical machine can read the historical access record of the block device provided by each physical machine in the physical machine cluster from the database.

[0077] In this embodiment, in order to enable the management and control node to determine the corresponding relationship, each physical machine in the cluster can access one or more databases. After an access operation to a block device occurs in the physical machine, an access record can be generated and stored in the database. For example, each time the physical machine receives an I / O operation to a block device, since the I / O operation carries parameters such as the access address or access length of the block device to be accessed, the accessed block can be determined based on these parameters; the access record generated based on this may include: access time, access operation (read or write), access address or access length operation record, etc., and then the access record can be stored in the database. In this way, when the management and control physical machine needs to determine the corresponding relationship, it can obtain the historical access records of the block devices provided by each physical machine in the cluster from the database.

[0078] like Figure 2 FIG. 1 is a schematic diagram of a cluster according to an exemplary embodiment of the present specification. Figure 2 In the example of the cluster, three physical machines are used, and in the cluster, three virtual machines (VMs) are used. The physical machines to which the virtual machines belong are not limited. Components for accessing block devices can be installed in virtual machines. As an example, TDC (Tap Device Connector) is used in the figure. TDC is a component for accessing the virtual machine side of the block device, which enables the virtual machine to access and use the underlying physical block device. By using TDC, the virtual machine can connect to the block device on the host and read and write it like a local device. TDC allows virtual machines to directly access physical block devices without indirect access through virtual disk image files. The working principle of TDC is to establish a communication channel between the host and the virtual machine, through which the read and write requests of the block device are passed to the physical device driver on the host. In this way, the virtual machine can access it like a local block device.

[0079] For example, based on the virtualization platform, a physical machine can virtualize many VMs, and the applications deployed by users in these VMs will read and store data on the disks in the VM. Among them, the VM will have at least one system disk for storing the operating system; there may also be multiple data disks, which store their own business data. The I / O request of each disk can reach the back-end block device through the block device access component TDC in the VM. When performing an I / O operation, the physical machine will print an operation record containing the access time, access operation (read or write), access address, and access length, that is, the access record mentioned above; optionally, the I / O operation record can be extracted and saved in the operation record database through a scheduled synchronization task.

[0080] By analyzing the I / O operation records, the access popularity of different logical block addresses of the block device can be calculated. For example, if the size of each block group LBA Group is 1MB, the amount of calculation can be greatly reduced. Count the number of accesses to the block devices in the cluster within a period (such as 1 day or 1 week). For example, there are m block devices in the cluster, and the i-th block device has n i block group LBA Group, the nth i The number of visits to an LBA Group is recorded as Vn i , use clustering algorithms, such as the natural breakpoint algorithm, to divide the heat level of each LBA Group.

[0081] For example, the input of the algorithm may be: the number of accesses to each LBA Group under each block device in the cluster; the output of the algorithm may be the heat level of each LBA Group of each block device.

[0082] Thus, the number of levels to be graded is determined based on the number of available compression algorithms, and the access heat level of each block group is calculated based on this to form a heat level comparison table of LBA Groups. The heat level here refers to the category to which the block group belongs.

[0083] When the VM performs IO reading and writing, the block group LBA group to which the I / O (the I / O operation carries the access address to be operated) belongs is calculated, the heat level of the LBA group is further determined, and the corresponding compression algorithm of the heat level is selected for compression. The higher the heat level, the faster the compression speed of the corresponding compression algorithm and the lower the compression rate.

[0084] Therefore, through the above process, this embodiment can classify the heat of the logical block address of the block device, and determine the compression algorithm used by the block by access heat, so as to maximize the overall data compression rate while ensuring IO quality and achieve higher space utilization efficiency.

[0085] Corresponding to the embodiments of the aforementioned data compression method, this specification also provides embodiments of a data compression device and a computer device to which it is applied.

[0086] The embodiments of the data compression device of this specification can be applied to computer devices, such as servers or terminal devices. The device embodiments can be implemented by software, hardware, or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor in which it is located reading the corresponding computer program instructions in the non-volatile memory into the memory and running them. From the hardware level, if Figure 3 The figure is a hardware structure diagram of the computer device where the data compression device of this manual is located, except Figure 3 In addition to the processor 310, memory 330, network interface 320, and non-volatile memory 340 shown, the computer device where the data compression device 331 is located in the embodiment may also include other hardware according to the actual function of the computer device, which will not be described in detail.

[0087] like Figure 4 As shown, Figure 4 The block diagram of a data compression device according to an exemplary embodiment of the present specification is shown, wherein the compressed data is stored in a block device, wherein the block device includes at least one block; the device includes:

[0088] The acquisition module 41 is used to: acquire the corresponding relationship between the block grouping and the compression algorithm; wherein the block grouping includes one block or multiple consecutive blocks, and the historical access frequency of the block grouping is positively correlated with the compression speed of the compression algorithm;

[0089] The compression module 42 is used to: in response to a data writing operation, execute the following compression process: obtain the access address of the data writing operation, and determine the target block where the access address is located; according to the correspondence between the block grouping and the compression algorithm, compress the data to be written using the target compression algorithm corresponding to the target block group to which the target block belongs.

[0090] In some examples, the corresponding relationship is determined by:

[0091] Acquire historical access records of at least one of the block devices, and determine historical access frequencies of a plurality of block groups based on the acquired historical access records;

[0092] Dividing each block group into multiple categories using the historical access frequency of each block group;

[0093] Determine the compression algorithm corresponding to each block group under the category.

[0094] In some examples, obtaining a historical access record of at least one of the block devices includes:

[0095] In response to a statistical event, a historical access record generated by at least one of the block devices from the last statistical time to the current time is obtained; wherein each historical access record is generated and stored after each access operation to the block device occurs.

[0096] In some examples, the historical access record includes: an access address and an access length;

[0097] The determining of the historical access frequencies of the plurality of block groups comprises:

[0098] Determine the access scope of each historical access record according to the access address and access length in each historical access record;

[0099] According to the address range of each block group and the access range of each historical access record, the historical access frequency of each block group is counted.

[0100] In some examples, the block groups are divided into multiple categories using the historical access frequencies of the block groups, including:

[0101] The number of compression algorithms is obtained, and the clustering algorithm clusters the block groups into the number of categories by using the historical access frequencies of the block groups.

[0102] In some examples, the method is applied to a physical machine in a cluster of physical machines, the physical machine managing at least one block device;

[0103] The obtaining of the correspondence between the block grouping and the compression algorithm includes:

[0104] Obtaining a correspondence between a block grouping and a compression algorithm determined by a control physical machine as a control node in the physical machine cluster;

[0105] The method further comprises:

[0106] In response to an access operation to the block device, an access record is generated to provide to the managed physical machine to determine the corresponding relationship.

[0107] In some examples, obtaining the correspondence between the block grouping and the compression algorithm determined by the control physical machine as the control node in the physical machine cluster includes:

[0108] Obtaining a comparison table sent by a control physical machine as a control node in the physical machine cluster, wherein the comparison table records a correspondence between each block group under each block device in the physical machine cluster and a compression algorithm;

[0109] The corresponding relationship between each block group and the compression algorithm under each block device managed by itself is determined from the comparison table.

[0110] In some examples, generating the access record to provide to the managed physical machine to determine the corresponding relationship includes:

[0111] An access record is generated and stored in a database, so that the management and control physical machine can read the historical access record of the block device provided by each physical machine in the physical machine cluster from the database.

[0112] The implementation process of the functions and effects of each module in the above-mentioned data compression device is specifically described in the implementation process of the corresponding steps in the above-mentioned data compression method, which will not be repeated here.

[0113] Accordingly, an embodiment of the present specification further provides a computer program product, including a computer program, which implements the steps of the aforementioned data compression method embodiment when executed by a processor.

[0114] Accordingly, an embodiment of the present specification also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the data compression method embodiment when executing the program.

[0115] Accordingly, the embodiment of this specification also provides a distributed system, the distributed system comprising:

[0116] A computer device for determining a correspondence between block groupings and compression algorithms; and

[0117] One or more computer devices for executing the aforementioned data compression method embodiments.

[0118] Accordingly, an embodiment of the present specification further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the data compression method embodiment are implemented.

[0119] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this specification. A person of ordinary skill in the art can understand and implement it without paying creative labor.

[0120] The above embodiments can be applied to one or more computer devices, where the computer device is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions. The hardware of the computer device includes but is not limited to a microprocessor, an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital signal processor (DSP), an embedded device, etc.

[0121] The computer device may be any electronic product that can perform human-computer interaction with a user, such as a personal computer, a tablet computer, a smart phone, a personal digital assistant (PDA), a game console, an interactive network television (IPTV), a smart wearable device, etc.

[0122] The computer device may also include a network device and / or a user device, wherein the network device includes, but is not limited to, a single network server, a server group consisting of multiple network servers, or a cloud consisting of a large number of hosts or network servers based on cloud computing.

[0123] The network where the computer device is located includes but is not limited to the Internet, a wide area network, a metropolitan area network, a local area network, a virtual private network (VPN), etc.

[0124] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0125] The step division of the above methods is only for clear description. When implemented, they can be combined into one step or some steps can be split and decomposed into multiple steps. As long as they include the same logical relationship, they are all within the protection scope of this patent; adding insignificant modifications to the algorithm or process or introducing insignificant designs without changing the core design of the algorithm and process are all within the protection scope of this application.

[0126] The description of "specific examples" or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of this specification. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0127] Those skilled in the art will readily appreciate other embodiments of the specification after considering the specification and practicing the invention claimed herein. The specification is intended to cover any variations, uses or adaptations of the specification that follow the general principles of the specification and include common knowledge or customary techniques in the art that are not claimed in the specification. The specification and examples are to be considered exemplary only, and the true scope and spirit of the specification are indicated by the following claims.

[0128] It should be understood that the present description is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present description is limited only by the appended claims.

[0129] The above description is only a preferred embodiment of this specification and is not intended to limit this specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of this specification should be included in the scope of protection of this specification.

Claims

1. A data compression method for compressing data and storing it in a block device, wherein the block device includes at least one block; the method comprises: Acquire a correspondence between a block group and a compression algorithm; wherein the block group includes one block or multiple consecutive blocks, and a historical access frequency of the block group is positively correlated with a compression speed of the compression algorithm; In response to a data write operation, the following compression process is executed: an access address of the data write operation is obtained, and a target block where the access address is located is determined; based on the correspondence between the block grouping and the compression algorithm, the data to be written is compressed using a target compression algorithm corresponding to the target block group to which the target block belongs.

2. According to the method of claim 1, the corresponding relationship is determined by: Acquire historical access records of at least one of the block devices, and determine historical access frequencies of a plurality of block groups based on the acquired historical access records; Dividing each block group into multiple categories using the historical access frequency of each block group; Determine a compression algorithm corresponding to each block grouping under the category.

3. According to the method of claim 2, the obtaining of the historical access record of at least one of the block devices comprises: In response to a statistical event, a historical access record generated by at least one of the block devices from the last statistical time to the current time is obtained; wherein each historical access record is generated and stored after each access operation to the block device occurs.

4. The method according to claim 2, wherein the historical access record comprises: Access address and access length; The determining of the historical access frequencies of the plurality of block groups comprises: Determine the access scope of each historical access record according to the access address and access length in each historical access record; According to the address range of each block group and the access range of each historical access record, the historical access frequency of each block group is counted.

5. The method according to claim 2, wherein the step of dividing each block group into multiple categories by using the historical access frequency of each block group comprises: The number of compression algorithms is obtained, and the clustering algorithm clusters the block groups into the number of categories by using the historical access frequencies of the block groups.

6. The method according to any one of claims 1 to 5, wherein the method is applied to a physical machine in a physical machine cluster, and the physical machine manages at least one block device; The obtaining of the correspondence between the block grouping and the compression algorithm includes: Obtaining a correspondence between a block grouping and a compression algorithm determined by a control physical machine as a control node in the physical machine cluster; The method further comprises: In response to an access operation to the block device, an access record is generated to provide to the managed physical machine to determine the corresponding relationship.

7. According to the method of claim 6, the step of obtaining the correspondence between the block grouping and the compression algorithm determined by the control physical machine as the control node in the physical machine cluster comprises: Obtaining a comparison table sent by a control physical machine as a control node in the physical machine cluster, wherein the comparison table records a correspondence between each block group under each block device in the physical machine cluster and a compression algorithm; The corresponding relationship between each block group and the compression algorithm under each block device managed by itself is determined from the comparison table.

8. According to the method of claim 6, the generating of the access record to provide to the managed physical machine to determine the corresponding relationship comprises: An access record is generated and stored in a database, so that the management and control physical machine can read the historical access record of the block device provided by each physical machine in the physical machine cluster from the database.

9. A data compression device, for compressing data and storing it in a block device, wherein the block device includes at least one block; the device comprises: The acquisition module is used to: acquire the corresponding relationship between the block grouping and the compression algorithm; wherein the block grouping includes one block or multiple consecutive blocks, and the historical access frequency of the block grouping is positively correlated with the compression speed of the compression algorithm; A compression module is used to: in response to a data writing operation, execute the following compression process: obtain the access address of the data writing operation, determine the target block where the access address is located; according to the correspondence between the block grouping and the compression algorithm, compress the data to be written using the target compression algorithm corresponding to the target block group to which the target block belongs.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

11. A distributed system, comprising: Computer device for determining the correspondence between block groupings and compression algorithms; as well as One or more computer devices for executing the data compression method according to any one of claims 1 to 8.

12. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images