Data reduction method and related system

Abstract

Provided in the present application are a data reduction method and a related system. The method comprises: acquiring a predicted data block; analyzing the predicted data block to obtain data reduction features of the predicted data block, wherein the data reduction features of the predicted data block comprise data reduction rates of the predicted data block under different data reduction parameters; and adjusting data reduction parameters of a storage system on the basis of the data reduction features of the predicted data block. In the method, a content-based data reduction feature perception algorithm is used to actively analyze data reduction features of a predicted data block, so as to obtain real-time data reduction features, and a data reduction scheduling decision is made on the basis of the real-time data reduction features.

Description

A data reduction method and related system

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on September 14, 2023, with application number 202311192193.0 and invention name “A data reduction method and related system”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of storage technology, and in particular to a data reduction method, device, storage system, computer-readable storage medium, and computer program product. Background Art

[0003] As data volumes continue to grow, lossless data reduction technologies are widely used in storage systems to reduce the amount of data actually stored and lower storage costs. Data reduction technology is a collection of techniques that can achieve this goal, including but not limited to algorithms such as data compression, data deduplication, and data compaction. These data reduction algorithms can typically be applied simultaneously within a storage system, achieving cumulative reduction effects.

[0004] As the infrastructure of the data age, storage systems typically need to support diverse workloads to meet the storage needs of different applications. For example, a storage system can support text or spreadsheet workloads to meet the storage needs of office applications. Another example is a storage system that supports executable file workloads to meet the storage needs of software development applications. Alternatively, a storage system can support video workloads to meet the storage needs of video playback applications.

[0005] To improve data reduction rates, related technologies typically configure a data reduction algorithm based on user-specified payload types or by identifying the payload type through file name suffixes, based on expert experience. For example, for text or table-based payloads, the data reduction algorithm can be configured as variable-length deduplication and online compression; for executable file-based payloads, the data reduction algorithm can be configured as fixed-length deduplication and background deep compression; and for video-based payloads, the data reduction algorithm can be configured as file deduplication without compression.

[0006] However, the loads connected to storage systems usually change dynamically. Even if the load is a single load, the data reduction characteristics within the load also change dynamically. The static matching mechanism makes it impossible to efficiently apply the matched data reduction strategy after the load changes, and the data reduction rate is difficult to meet business needs.

[0007] Summary of the Invention

[0008] This application provides a data reduction method that proactively senses and predicts data reduction characteristics of data blocks and dynamically adjusts data reduction parameters of a storage system based on these characteristics. This method dynamically adjusts the data reduction parameters as the load changes, thereby improving the overall data reduction rate. This application also provides a data reduction device, storage system, computer-readable storage medium, and computer program product corresponding to the above method.

[0009] In a first aspect, the present application provides a data reduction method. This method can be applied to a storage system. The storage system can be a centralized storage system, such as a centralized storage system with integrated disk and controller, or a centralized storage system with separate disk and controller. The storage system can also be a distributed storage system, such as a distributed storage system with integrated storage and computing, or a distributed storage system with separate storage and computing.

[0010] Specifically, the storage system can obtain the predicted data block, and then analyze the predicted data block to obtain the data reduction characteristics of the predicted data block. The data reduction characteristics of the predicted data block include the data reduction rate of the predicted data block under different data reduction parameters, and then adjust the data reduction parameters of the storage system according to the data reduction characteristics of the predicted data block.

[0011] This method actively analyzes the data reduction characteristics of the predicted data block to obtain real-time data reduction characteristics, and makes data reduction scheduling decisions (such as adjusting data reduction parameters) based on the real-time data reduction characteristics. This can achieve dynamic adjustment of data reduction strategies (such as data reduction parameters) as load changes, thereby improving the overall data reduction rate.

[0012] In some possible implementations, considering that the storage system can apply different types of data reduction algorithms to achieve data reduction, the data reduction parameters may include at least one of a compression parameter or a deduplication parameter. Accordingly, the data reduction rate may be determined based on at least one of the compression rate or the deduplication rate. The compression parameters may include one or more of whether compression is enabled, compression timing (online compression, background deep compression), and compression form (dictionary / entropy coding, public dictionary compression), and the deduplication parameters may include one or more of whether deduplication is enabled, deduplication scope (global deduplication, local deduplication), deduplication time (online deduplication, post-deduplication), and deduplication form (fixed-length deduplication, variable-length deduplication).

[0013] This method supports selecting compression parameters and / or deduplication parameters according to needs to perceive the data reduction characteristics of predicted data blocks, and adjusts the data reduction parameters of the storage system based on the perception results of the predicted data blocks. On the basis of ensuring the data reduction rate, it also improves the flexibility of data reduction.

[0014] In some possible implementations, the storage system may determine the entropy and correlation of the predicted data block. When the entropy of the predicted data block is greater than a first threshold and the correlation of the predicted data block is greater than a second threshold, or when the entropy of the predicted data block is less than the first threshold, the storage system may compress the predicted data block using different compression parameters to obtain compression ratios of the predicted data block under the different compression parameters.

[0015] Among them, the storage system can make a compressibility judgment by predicting the entropy and / or correlation of the data block. When the predicted data block is compressible, it is compressed (pre-compressed) to determine the compression rate of the predicted data block under different compression parameters, thus avoiding unnecessary resource overhead.

[0016] In some possible implementations, the predicted data block includes multiple data blocks. The storage system may determine fingerprints of the multiple data blocks, determine a duplication distribution of the predicted data block based on the fingerprints of the multiple data blocks, and then determine a deduplication rate of the predicted data block under different deduplication parameters based on the duplication distribution.

[0017] The storage system determines deduplication suitability by predicting the duplication distribution of data blocks. When a data block is predicted to be deduplicatable, the storage system determines the deduplication rate of the predicted data block under different deduplication parameters, thus avoiding additional resource overhead.

[0018] In some possible implementations, the storage system may also obtain system resource status. The system resource status includes the utilization rate of at least one system resource. Accordingly, the storage system may dynamically adjust the system resources used for data reduction based on the system resource status to obtain a system resource allocation result. The storage system may then adjust the storage system's data reduction parameters based on the system resource allocation result and the predicted data reduction characteristics of the data block.

[0019] This can break through the simple adaptive mechanism of a single reduction algorithm and a single system resource, and realize unified adaptive scheduling of multiple data reduction algorithms. At the same time, it can ensure the system's foreground read and write performance and reduce the impact of the data reduction algorithm's after-effect performance (for example, read amplification caused by reading deduplicated data across nodes).

[0020] In some possible implementations, the storage system can determine candidate compression parameters based on the computing power inequality and the allocation of system resources. The storage system can then determine an upper bound on the compression ratio based on the candidate compression parameters and the compression ratios corresponding to the candidate compression parameters, and determine an upper bound on the deduplication ratio based on the network amplification inequality or the storage amplification inequality. The storage system can determine candidate deduplication parameters based on the upper bound on the deduplication ratio and the predicted deduplication ratios of data blocks under different deduplication parameters, and update the candidate compression parameters based on the candidate deduplication parameters and the computing power inequality until a target compression parameter and target deduplication parameter are determined that satisfy the data reduction ratio requirements, and then adjust the data reduction parameters of the storage system to the target compression parameter and target deduplication parameter.

[0021] This method constructs inequalities based on domain knowledge, and determines the target compression parameters and target deduplication parameters based on the inequalities so that the data reduction rate meets the requirements. The data reduction parameters of the storage system are adjusted accordingly. On the one hand, the data reduction rate is guaranteed, and on the other hand, the foreground read and write performance of the system is guaranteed, the after-effect performance impact of the data reduction algorithm is reduced, and the balance between data reduction rate and performance is achieved.

[0022] In some possible implementations, the storage system may construct a planning model using the allocation of system resources as a constraint and meeting data reduction requirements as a goal. The storage system may solve the planning model to obtain a target compression ratio and a target deduplication ratio, determine target compression parameters and target deduplication parameters based on the target compression ratio, target deduplication ratio, and the data reduction characteristics, and adjust the storage system's data reduction parameters to the target compression parameters and target deduplication parameters.

[0023] This method solves the target compression rate and target deduplication rate based on the planning model, and then determines the target compression parameters and target deduplication parameters based on the data reduction characteristics. The data reduction parameters of the storage system are adjusted to achieve a balance between data reduction rate and performance.

[0024] In some possible implementations, the storage system can dynamically adjust the system resources used for data reduction based on the system resource status through reinforcement learning or decision trees. This can achieve smooth adjustments to system resource allocation and ensure the stability of the storage system.

[0025] In a second aspect, the present application provides a data reduction device. The device includes:

[0026] An acquisition module, used for acquiring prediction data blocks;

[0027] an analyzing module, configured to analyze the predicted data block to obtain a data reduction feature of the predicted data block, wherein the data reduction feature of the predicted data block includes a data reduction rate of the predicted data block under different data reduction parameters;

[0028] An adjustment module is configured to adjust data reduction parameters of a storage system according to the data reduction characteristics of the predicted data block.

[0029] In some possible implementations, the data reduction parameter includes at least one of a compression parameter and a deduplication parameter, and the data reduction rate is determined according to at least one of the compression rate and the deduplication rate.

[0030] In some possible implementations, the analysis module is specifically configured to:

[0031] Determining the entropy and relevance of the prediction data block;

[0032] When the entropy of the prediction data block is greater than a first threshold and the correlation degree of the prediction data block is greater than a second threshold, or the entropy of the prediction data block is less than the first threshold, the prediction data block is compressed using different compression parameters to obtain the compression rate of the prediction data block under different compression parameters.

[0033] In some possible implementations, the predicted data block includes multiple data blocks, and the analysis module is specifically configured to:

[0034] determining fingerprints of the plurality of data blocks, and determining a repetition distribution of the predicted data block based on the fingerprints of the plurality of data blocks;

[0035] Deduplication rates of the predicted data block under different deduplication parameters are determined according to the duplication distribution.

[0036] In some possible implementations, the acquisition module is further configured to:

[0037] Acquiring a system resource status, wherein the system resource status includes a utilization rate of at least one system resource;

[0038] The adjustment module is specifically used for:

[0039] Dynamically adjusting system resources used for data reduction according to the system resource status to obtain an allocation result of the system resources;

[0040] According to the allocation result of the system resources and the data reduction characteristics of the predicted data block, data reduction parameters of the storage system are adjusted.

[0041] In some possible implementations, the adjustment module is specifically configured to:

[0042] Determining candidate compression parameters based on the computing power inequality and the allocation result of the system resources;

[0043] Determining an upper bound on the compression ratio based on the candidate compression parameters and the compression ratios corresponding to the candidate compression parameters, and determining an upper bound on the deduplication ratio based on a network amplification inequality or a storage amplification inequality;

[0044] determining a candidate deduplication parameter according to the deduplication rate upper bound and the deduplication rates of the predicted data block under different deduplication parameters;

[0045] Updating the candidate compression parameters according to the candidate deduplication parameters and the computing power inequality until a target compression parameter and a target deduplication parameter are determined so that the data reduction rate meets the requirement;

[0046] The data reduction parameters of the storage system are adjusted to the target compression parameter and the target deduplication parameter.

[0047] In some possible implementations, the adjustment module is specifically configured to:

[0048] A planning model is constructed with the allocation result of the system resources as a constraint condition and the data reduction rate meeting the requirement as a goal;

[0049] Solving the planning model to obtain a target compression ratio and a target deduplication ratio, and determining a target compression parameter and a target deduplication parameter according to the target compression ratio, the target deduplication ratio, and the data reduction characteristic;

[0050] The data reduction parameters of the storage system are adjusted to the target compression parameter and the target deduplication parameter.

[0051] In some possible implementations, the adjustment module is specifically configured to:

[0052] According to the system resource status, the system resources used for data reduction are dynamically adjusted through reinforcement learning or decision tree.

[0053] In a third aspect, the present application provides a storage system. The storage system includes a controller and a memory, wherein the memory stores computer-readable instructions; the controller executes the computer-readable instructions to cause the storage system to perform the data reduction method described in the first aspect or any implementation of the first aspect.

[0054] In a fourth aspect, the present application provides a computer-readable storage medium, in which instructions are stored, and the instructions instruct a computing device or a computing device cluster to execute the data reduction method described in the first aspect or any implementation of the first aspect.

[0055] In a fifth aspect, the present application provides a computer program product comprising instructions, which, when executed on a computing device or a computing device cluster, enables the computing device or computing device cluster to execute the data reduction method described in the first aspect or any one of the implementations of the first aspect.

[0056] Based on the implementation methods provided in the above aspects, this application can also be further combined to provide more implementation methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the technical methods of the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments.

[0058] FIG1A is a schematic diagram of the architecture of a centralized storage system with disk-controller separation provided in an embodiment of the present application;

[0059] FIG1B is a schematic diagram of the architecture of a centralized storage system with integrated disk and controller provided in an embodiment of the present application;

[0060] FIG1C is a schematic diagram of the architecture of a distributed storage system integrating storage and computing provided in an embodiment of the present application;

[0061] FIG1D is a schematic diagram of the architecture of a distributed storage system with storage and computing separation provided in an embodiment of the present application;

[0062] FIG2 is a diagram showing a scheduling principle of a data reduction algorithm provided in an embodiment of the present application;

[0063] FIG3 is a flow chart of a data reduction method provided in an embodiment of the present application;

[0064] FIG4 is a schematic diagram of a perception layer performing data reduction feature or system resource status perception according to an embodiment of the present application;

[0065] FIG5 is a hierarchical logic diagram of a storage system provided in an embodiment of the present application;

[0066] FIG6 is a schematic diagram of a resource allocator and policy scheduler provided in an embodiment of the present application;

[0067] FIG7 is a schematic structural diagram of a data reduction device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0068] The terms "first" and "second" in the embodiments of this application are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Therefore, features specified as "first" or "second" may explicitly or implicitly include one or more of the features.

[0069] First, some technical terms involved in the embodiments of this application are introduced.

[0070] Data reduction refers to a collection of technologies used to reduce data volume, including but not limited to data compression (DC), deduplication, and compaction. These data reduction algorithms can typically be applied simultaneously within a storage system. For example, data compression and deduplication can be applied simultaneously within a storage system, achieving a combined data reduction effect.

[0071] Data compression (DC) is the process of representing information using fewer bits (or other information-related units) than in the original encoding scheme. Data compression compresses data blocks by exploiting local repetitive and frequency characteristics, achieving data reduction. The compression and decompression processes typically consume significant computing resources, such as central processing unit (CPU) resources.

[0072] Data deduplication usually involves determining the fingerprint (FP) of a data block after data segmentation, and discovering duplicate data blocks by comparing the fingerprints of the data blocks. Duplicate data blocks are only saved once. Data deduplication can be divided into local deduplication and global deduplication. Local deduplication refers to deduplication of a partial range of data, such as data on the same node, and global deduplication refers to deduplication of data in a global range, such as deduplication of data on all nodes. During the data deduplication process, content-based data segmentation and fingerprint calculation can consume CPU resources. In addition, global deduplication can consume network resources when forwarding data blocks or fingerprints across nodes, and lead to read and write amplification of the back-end storage. In addition, the deduplication process performs fine-grained data splitting, and the deduplicated data generally needs to be read across nodes and disks, which will cause read input and output (IO) amplification when reading data.

[0073] Data compression, specifically in a storage system that stores data at a fixed granularity, achieves the purpose of data reduction by rearranging the data to avoid the padding overhead caused by pre-reduction algorithms such as data compression that cause data to be misaligned with the storage granularity.

[0074] The data reduction method provided in this application can be applied to storage systems. The above-mentioned storage systems can be divided into the following categories according to logical functions: primary storage and backup storage. Primary storage serves as a storage pool or hard disk for upper-layer applications and can be accessed by upper-layer applications such as applications deployed in the form of virtual machines (VMs). Backup storage serves as a warehouse for storing images, backup disks or snapshots, which are usually not directly accessible to VMs. Primary storage and backup storage can be physically separated storage systems, or the same storage system can serve as both primary storage and backup storage. Storage systems can also be divided into file storage, database storage (including row-based databases or column-based databases), and big data storage according to application scenarios.

[0075] The above storage system (including primary storage and backup storage) can adopt local storage mode or cloud storage mode. Local storage refers to the mode of storing data on own physical storage devices, while cloud storage refers to the mode of storing data in a cloud environment to support network access. The cloud environment refers to the cluster built by cloud service providers (also known as cloud vendors) to provide computing resources, storage resources, and network resources. The storage system supports block storage, object storage, or file storage.

[0076] The storage system may adopt a centralized storage architecture or a distributed storage architecture. In other words, the storage system may be a centralized storage system or a distributed storage system.

[0077] A centralized storage system consists of one or more master devices forming a central node. Data is centrally stored in this central node, and all data processing operations for the entire system are centrally deployed on this central node. In other words, in a centralized storage system, terminals or clients are only responsible for data input and output, while data storage and control processing are completely handled by the central node. The most significant feature of a centralized system is its simple deployment structure. There is no need to consider how to deploy services on multiple nodes, nor does it require the consideration of distributed collaboration between multiple nodes.

[0078] For ease of understanding, Figure 1A shows an architectural diagram of a centralized storage system. In the application scenario shown in Figure 1A, users access data through applications. The computers that run these applications are called "application servers." The application server 100 can be a physical machine or a virtual machine VM. Physical machines (physical application servers) include but are not limited to desktop computers, servers, laptops, and mobile devices. The application server accesses the storage system through the fiber optic switch 110 to access data. However, the switch 110 is only an optional device, and the application server 100 can also communicate directly with the storage system 120 through the network. Alternatively, the fiber optic switch 110 can also be replaced with an Ethernet switch, an InfiniBand (IB) switch, an RDMA (RDMA over Converged Ethernet, RoCE) switch based on converged Ethernet, etc.

[0079] The storage system 120 shown in Figure 1A is a centralized storage system. A centralized storage system is characterized by a unified entry point through which all data from external devices must pass. This entry point is the centralized storage system's engine 121. Engine 121 is the core component of the centralized storage system, implementing many of the system's advanced functions.

[0080] As shown in Figure 1A, there are one or more controllers in the engine 121. Figure 1A uses the example of an engine containing two controllers for explanation. There is a mirror channel between controller 0 and controller 1. Then, after controller 0 writes a copy of data into its memory 124, it can send a copy of the data to controller 1 through the mirror channel, and controller 1 stores the copy in its own local memory 124. In this way, controller 0 and controller 1 back up each other. When controller 0 fails, controller 1 can take over the business of controller 0. When controller 1 fails, controller 0 can take over the business of controller 1, thereby avoiding the unavailability of the entire storage system 120 due to hardware failure. When there are 4 controllers deployed in the engine 121, there is a mirror channel between any two controllers, so any two controllers back up each other.

[0081] The engine 121 also includes a front-end interface 125 and a back-end interface 126. The front-end interface 125 is used to communicate with the application server 100, thereby providing storage services for the application server 100. The back-end interface 126 is used to communicate with the hard disk 134 to expand the storage system capacity. Through the back-end interface 126, the engine 121 can connect to more hard disks 134, thereby forming a very large storage resource pool.

[0082] In terms of hardware, as shown in FIG1A , the controller 0 includes at least a processor 123 and a memory 124. The processor 123 is a central processing unit (CPU) that processes data access requests from outside the storage system (servers or other storage systems), and also processes requests generated within the storage system. For example, when the processor 123 receives write data requests from the application server 100 through the front-end port 125, it temporarily stores the data in these write data requests in the memory 124. When the total amount of data in the memory 124 reaches a certain threshold, the processor 123 sends the data stored in the memory 124 to the hard disk 134 through the back-end port for persistent storage.

[0083] Memory 124 refers to internal memory that directly exchanges data with the processor. It can read and write data at any time and at a high speed, and serves as temporary data storage for the operating system or other running programs. Memory includes at least two types of memory. For example, memory can be either random access memory (RAM) or read-only memory (ROM). For example, RAM is dynamic random access memory (DRAM) or storage class memory (SCM), and ROM can be programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), etc.

[0084] Memory 124 stores software programs, and processor 123 executes the software programs in memory 124 to manage the hard disks. For example, hard disks can be abstracted into a storage resource pool, which is then divided into LUNs for server use. The LUNs here are essentially the hard disks seen on the server. Of course, some centralized storage systems are also file servers, providing shared file services to servers.

[0085] The hardware components and software structure of controller 1 (and other controllers not shown in FIG. 1A ) are similar to those of controller 0 and are not described in detail here.

[0086] Figure 1A shows a centralized storage system with separate disk and controller. In this system, the engine 121 may not have a hard drive slot. The hard drive 134 must be placed in the hard drive frame 130, and the back-end interface 126 communicates with the hard drive frame 130. The back-end interface 126 exists in the form of an adapter card in the engine 121. Two or more back-end interfaces 126 can be used simultaneously on one engine 121 to connect to multiple hard drive frames. Alternatively, the adapter card can be integrated into the motherboard, in which case the adapter card can communicate with the processor 123 via the PCIE bus.

[0087] It should be noted that FIG1A shows only one engine 121 . However, in actual applications, the storage system may include two or more engines 121 , and redundancy or load balancing may be performed between the multiple engines 121 .

[0088] The disk enclosure 130 includes a control unit 131 and several hard disks 134. The control unit 131 can have various forms. Typically, the number of control units 131 can be one, two, or more. When the disk enclosure 130 includes at least two control units 131, the hard disks 134 and the control units 131 may have an ownership relationship. The functions of the control unit 131 can be offloaded to the network card 104. In this embodiment, the disk enclosure 130 does not have a control unit 131. Instead, the network card 104 performs data reading and writing, address translation, and other computing functions. In this case, the network card 104 is an intelligent network card. It can include a CPU and memory. In some application scenarios, the network card 104 may also include persistent memory media, such as persistent memory (PM), non-volatile random access memory (NVRAM), or phase change memory (PCM). The CPU is used to perform operations such as address translation and reading and writing data. The memory is used to temporarily store data to be written to the hard disk 134 or data read from the hard disk 134 to be sent to the controller.

[0089] A centralized storage system can also adopt a disk-controller integrated architecture. In this architecture, engine 121 has hard drive slots, and hard drives 134 can be directly deployed in engine 121. Backend interface 126 is optional. When the system's storage space is insufficient, additional hard drives or hard drive enclosures can be connected through backend interface 126. As shown in Figure 1B, engine 121 can connect to more hard drives 134 through backend interface 126.

[0090] A distributed storage system stores data across multiple independent storage nodes. This system utilizes a scalable architecture, sharing the storage load across multiple storage nodes. This not only improves system reliability, availability, and access efficiency, but also facilitates scalability.

[0091] Distributed storage systems can adopt either a storage-computing integrated architecture or a storage-computing separated architecture. Each is described below.

[0092] As shown in FIG1C , the distributed storage system with integrated storage and computing includes a storage cluster. The storage cluster includes one or more servers 110 (three servers 110 are shown in FIG1C , but are not limited to three servers 110), and each server 110 can communicate with each other. Server 110 is a device that has both computing and storage capabilities, such as a server or desktop computer. For example, an ARM server or an X86 server can be used as server 110 here. In terms of hardware, server 110 includes at least a processor 112, memory 113, a network card 114, and a hard disk 105. The processor 112, memory 113, network card 114, and hard disk 105 are connected via a bus. Processor 112 is a central processing unit (CPU) that processes data access requests from outside server 110 (application servers or other servers 110) and also processes requests generated within server 110. In addition, processor 112 is used for data calculation or processing, such as metadata management, data deduplication, data compression, data verification, virtualized storage space, and address translation.

[0093] As shown in Figure 1D, the distributed storage cluster with separated storage and computing includes a computing cluster (also called a computing node cluster) and a storage cluster (also called a storage node cluster). Each computing node 110 can communicate with each other. The computing node 110 is a computing device, such as a server, a desktop computer, or a controller of a storage array. In terms of hardware, as shown in Figure 1D, the computing node 110 includes at least a processor 112, a memory 113, and a network card 114. Among them, the processor 112 is a central processing unit (CPU) used to process data access requests from outside the computing node 110, or requests generated inside the computing node 110. When the total amount of data in the memory 113 reaches a certain threshold, the processor 112 sends the data stored in the memory 113 to the storage node 100 for persistent storage. In addition, the processor 112 is also used for computing or processing data, such as metadata management, deduplication, data compression, virtualized storage space, and address translation.

[0094] Any computing node 110 can access any storage node 100 in the storage node cluster through the network. The storage node cluster includes multiple storage nodes 100 (three storage nodes 100 are shown in Figure 1D, but are not limited to three storage nodes 100). A storage node 100 includes one or more controllers 101, a network card 104 and multiple hard disks 105. The network card 104 is used to communicate with the computing node 110. The hard disk 105 is used to store data and can be a disk or other type of storage medium, such as a solid-state drive or a shingled magnetic recording hard disk. The controller 101 is used to write data to the hard disk 105 or read data from the hard disk 105 according to the read / write data request sent by the computing node 110. In the process of reading and writing data, the controller 101 needs to convert the address carried in the read / write data request into an address that can be recognized by the hard disk. It can be seen that the controller 101 also has some simple computing functions.

[0095] It should be noted that the above is merely an example description of the storage system. In other possible implementations of the embodiments of the present application, the storage system may also be a distributed storage system or a centralized storage system of other architectures. For example, the storage system may also be a distributed storage system of a fully converged architecture.

[0096] When storing data, a storage system can configure a data reduction algorithm that corresponds to the perceived load type (e.g., user-specified load type or identified by file name suffix) based on expert experience. However, the load carried by the storage system changes over time, and the static matching mechanism makes it impossible to efficiently apply the matching data reduction strategy after the load changes. On the other hand, the data reduction characteristics within a single load are also dynamic. For example, the data reduction rate of a typical database load is between 2 and 4. Applying the same data reduction strategy cannot adapt to the dynamic changes in the data reduction characteristics within the load.

[0097] In view of this, the present application provides a data reduction method. The method adopts a content-based data reduction feature perception algorithm to actively analyze the data reduction features of data blocks (for example, sampled predictive data blocks used to decide the data reduction parameters of the storage system) in real time to obtain real-time data reduction features, which include but are not limited to the compression ratio of data blocks under different compression parameters (characterizing data compression potential) and the deduplication ratio under different deduplication parameters (characterizing data deduplication potential). Then, data reduction scheduling decisions are made based on the real-time data reduction features, for example, adjusting the data reduction parameters of the storage system according to the real-time data reduction features. In this way, the data reduction strategy (such as deduplication parameters, compression parameters and other data reduction parameters) can be dynamically adjusted as the load changes, thereby improving the overall data reduction rate.

[0098] In addition, this method is based on the content's active perception of data reduction features, and does not require upper-layer applications to provide load types, achieving upper-layer non-intrusion. It is suitable for a variety of storage scenarios, including distributed storage scenarios as infrastructure, and does not require the use of file name suffixes to perceive load types. It supports multiple storage methods such as file storage, block storage, or object storage, and has high availability.

[0099] Furthermore, the method can also collect system resource status, which includes the utilization rate of at least one system resource, such as CPU utilization, network bandwidth utilization, and storage bandwidth (also known as IO bandwidth) utilization. Based on the utilization rate of the above system resources, the system resources used for data reduction (also known as the system resources allocated to data reduction) are dynamically adjusted, and data reduction algorithm decisions and scheduling are made based on the system resource allocation results and data reduction characteristics. This can break through the simple adaptive mechanism of a single reduction algorithm and a single system resource, and achieve unified adaptive scheduling of multiple data reduction algorithms, while ensuring the system's foreground read and write performance and reducing the impact of the data reduction algorithm's after-effect performance (for example, the read IO amplification caused by reading deduplicated data across nodes mentioned above).

[0100] As shown in Figure 2, there are many parameter choices when multiple data reduction algorithms such as data compression and data deduplication are applied together. Among them, compression parameters and deduplication parameters, including algorithm selection (compression only, deduplication only, deduplication + compression), scope (global deduplication, local deduplication), application timing (online deduplication, post-deduplication), and algorithm form (fixed-length deduplication, variable-length deduplication, dictionary / entropy coding, public dictionary compression), can be decided based on the data reduction characteristics of the load. Furthermore, the above-mentioned compression parameters and deduplication parameters can be jointly decided based on the data reduction characteristics and system resource status, thereby achieving efficient data reduction (for example, the impact on read and write performance is as small as possible, and the data reduction rate is as high as possible).

[0101] Among them, the storage system can collect system resource status (such as system resource utilization) through the resource feedback layer (for example, it can be a layer that feeds back system resource status in the storage node), and perceive data reduction characteristics through the prediction and evaluation layer (for example, it is a layer in the storage node used to predict the reduction rate), such as obtaining compression rate and deduplication rate through data reduction rate prediction. It should be noted that the prediction and evaluation layer can also first perform reduction detection, such as compressibility (abbreviated as compressibility) detection and deduplication detection, and then perform compression rate prediction when compressibility is detected, and perform deduplication rate prediction when deduplication is detected. The storage system uses the above-mentioned system resource status and data reduction characteristics as reference data for decision-making, performs resource modeling based on data reduction domain knowledge, and performs data reduction algorithm decision scheduling by constructing a joint optimization problem of multiple data reduction algorithms, while considering the foreground read and write performance and the performance impact of the reduced data, to achieve efficient data reduction.

[0102] In some possible implementations, the storage system can also sense the application type or identify the load type. The application type includes but is not limited to database applications and virtualization applications (for example, virtual machine-based applications). Applications such as databases and virtualization are marked scenarios, and scenarios where the application type is not sensed are unmarked scenarios. The load type can include one or more of read / write, heat, sequence, and random. When the storage system makes a decision to schedule the data reduction algorithm, it can also make a decision based on the above application type and load type. For example, for the marked scenario of database, the data reduction strategy (such as compression parameters and deduplication parameters) determined by the storage system in the data reduction algorithm decision scheduling can be (3), (4), (6), (8), (10), and (12), where (3) represents deduplication + compression, (4) represents global deduplication, (6) represents online deduplication, (8) represents online compression, (10) represents fixed-length deduplication, and (12) represents dictionary / entropy coding. For another example, for an untagged scenario, the data reduction strategies (such as compression parameters and deduplication parameters) determined by the storage system during data reduction decision scheduling can be (3), (5), (7), (9), (11), and (13), where (5) represents local deduplication, (7) represents post-deduplication, (9) represents background deep compression, (11) represents variable-length deduplication, and (13) represents public dictionary compression. After determining the above data reduction strategies, the storage system can inject the data reduction strategies to execute the above data reduction strategies, thereby achieving efficient data reduction.

[0103] In order to make the technical solution of the present application clearer and easier to understand, the data reduction method of the present application is introduced below in conjunction with embodiments.

[0104] Referring to the flowchart of a data reduction method shown in FIG3 , the method includes the following steps:

[0105] S302: The storage system obtains a prediction data block.

[0106] A predicted data block is a data block used to predict the data reduction characteristics of the storage system's load and to determine the storage system's data reduction strategy based on these characteristics. Given that the data reduction characteristics of sample data can reflect the data reduction characteristics of global data to a certain extent, the storage system can obtain the predicted data block by sampling global data. In other words, the predicted data block can be a data block obtained through sampling. The predicted data block can include one or more data blocks.

[0107] In a specific implementation, the storage system can sample global data from a temporal perspective to obtain predicted data blocks. For example, the storage system can set a sampling period and sample global data within each sampling period according to the sampling period to obtain predicted data blocks. In some possible implementations, the storage system can also sample global data from a spatial perspective to obtain predicted data blocks. For example, the storage system can set a sampling interval, such as sampling every N data blocks to obtain sampled data blocks, where N is a positive integer.

[0108] In some possible implementations, the storage system can also sample and obtain predicted data blocks when the persistence condition is triggered. Specifically, the storage system can serve as the underlying infrastructure and connect to at least one upper-layer application. When the upper-layer application performs data IO (such as write IO), the storage system can persist the data to meet the storage needs of the upper-layer application. Among them, the storage system can cache the data first. When the cache meets the set persistence conditions, for example, the cache utilization reaches a preset value, the persistence operation can be triggered, and the storage system can sample the data in the cache to obtain the predicted data block, and then decide on the data reduction strategy based on the data reduction characteristics of the predicted data block, so as to persist the data in the cache according to the data reduction strategy (also known as offloading to disk, flushing the disk).

[0109] S304: The storage system analyzes the predicted data block to obtain data reduction features of the predicted data block.

[0110] Data reduction characteristics specifically describe the potential for data reduction. Data reduction characteristics may vary depending on the data content. For a predicted data block, the data reduction characteristics may include the data reduction ratio of the predicted data block under different data reduction parameters. The higher the data reduction ratio, the greater the data reduction potential.

[0111] Considering that the storage system can apply different types of data reduction algorithms to achieve data reduction, the data reduction parameters may include at least one of a compression parameter or a deduplication parameter. Accordingly, the data reduction rate may be determined based on at least one of the compression rate or the deduplication rate. In some examples, when the storage system simultaneously applies a data compression algorithm and a data deduplication algorithm to perform data reduction, the data reduction rate may be the product of the compression rate and the deduplication rate. In other examples, when the storage system simultaneously applies a data compression algorithm and a data deduplication algorithm to perform data reduction, the data reduction rate may also be characterized by the compression rate (characterizing the data compression potential) and the deduplication rate (characterizing the data deduplication potential), respectively.

[0112] Among them, the compression parameters may include one or more of whether compression is enabled, the compression timing (online compression, background deep compression), and the compression form (dictionary / entropy coding, public dictionary compression); the deduplication parameters may include one or more of whether deduplication is enabled, the deduplication scope (global deduplication, local deduplication), the deduplication time (online deduplication, post-deduplication), and the deduplication form (fixed-length deduplication, variable-length deduplication).

[0113] For data compression, the storage system can first perform a compressibility test on the predicted data block. If the predicted data block is compressible, it can pre-compress the predicted data block according to different compression parameters to obtain the compression ratio of the predicted data block under different compression parameters. The storage system can perform the compressibility test on the predicted data block based on entropy or correlation.

[0114] Specifically, the storage system may determine the entropy and correlation of the prediction data block. In this embodiment, the entropy of the prediction data block (e.g., sampled data) may be byte entropy, and the correlation of the prediction data block may be the correlation probability of the prediction data block. When the entropy of the prediction data block is greater than a first threshold and the correlation of the prediction data block is greater than a second threshold, or when the entropy of the prediction data block is less than the first threshold, it indicates that the prediction data block is compressible.

[0115] In some possible implementations, referring to FIG4 , the storage system may first determine the byte entropy of the predicted data block. When the byte entropy is less than the first threshold value (in other words, the byte entropy is low entropy), it indicates that the predicted data block is compressible, and the storage system may use different compression parameters to pre-compress the predicted data block to obtain the compression ratio of the predicted data block under different compression parameters. When the byte entropy is greater than or equal to the first threshold value (in other words, the byte entropy is high entropy), the storage system may further determine the correlation of the predicted data block (such as sampled data) according to the central set algorithm, specifically the correlation probability based on the central set, denoted as L2. Among them, L2 can be determined by the following formula:

[0116] p(a) represents the empirical probability that a belongs to the center set, p(b) represents the empirical probability that b belongs to the center set, and p(a,b) represents the probability that both a and b belong to the center set.

[0117] When the byte entropy is greater than the first threshold and L2 is greater than the second threshold (in other words, a high L2 indicates a high number of repeated patterns), the predicted data block is compressible, and the storage system can pre-compress the predicted data block using different compression parameters to obtain the compression ratio of the predicted data block under different compression parameters. When the byte entropy is greater than the first threshold and L2 is less than or equal to the second threshold, the predicted data block is not compressible, and the storage system can skip pre-compression.

[0118] For data deduplication, the storage system can calculate the predicted data block's duplication distribution and thereby obtain a predicted deduplication rate for the predicted data block under different deduplication parameters. Specifically, the predicted data block may include multiple data blocks. The storage system can determine the fingerprints of multiple data blocks in the predicted data block, determine the predicted data block's duplication distribution based on the fingerprints of the multiple data blocks, and then determine the predicted data block's deduplication rate under different deduplication parameters based on the duplication distribution.

[0119] Still using the example of Figure 4, the storage system can write the fingerprints of multiple data blocks in the predicted data block into a lightweight cache-like structure, wherein the cache-like structure can be a data structure similar to a cache whose cache mechanism is Least Recently Used (LRU). This type of cache structure can be used to count the duplication distribution (Duplicate Frequency Histogram, DFH) of statistical data blocks. Specifically, the storage system can count the duplication frequency (or repetition frequency) of the fingerprints based on the fingerprints of each data block in this type of cache structure. For example, the repetition frequency of FP0 and FP1 is 1, the repetition frequency of FP2 and FP4 is 2, and the repetition frequency of FP3 is 3. The storage system can make a deduplication judgment based on the repetition frequency of the fingerprints. When the repetition frequency is all 1, it indicates that it is not deduplicatable. When there is a fingerprint with a repetition frequency greater than 1, it indicates that it is deduplicatable. If deduplication is possible, the storage system can further determine the predicted data block's duplication distribution, for example, by counting the number of fingerprints with different duplication frequencies, for example, the number of fingerprints with a duplication frequency of 1 is 2, the number of fingerprints with a duplication frequency of 2 is 2, and the number of fingerprints with a duplication frequency of 3 is 1. The duplication distribution can be determined with the duplication frequency as the horizontal axis and the number of fingerprints as the vertical axis. Based on the above duplication distribution, the storage system can determine the predicted deduplication rate of the predicted data block under different deduplication parameters. The different deduplication parameters can be global deduplication, local deduplication, or percentage deduplication. For ease of understanding, Figure 4 uses percentage deduplication to illustrate examples including 50% deduplication and 30% deduplication.

[0120] S306: The storage system adjusts data reduction parameters of the storage system according to the data reduction characteristics of the predicted data block.

[0121] Specifically, the predicted data block reduction rate can be determined based on the predicted data block compression rate and deduplication rate. The storage system can determine a target compression parameter and / or target deduplication parameter based on the predicted data block compression rate under different compression parameters and the predicted data block deduplication rate under different deduplication parameters, so that the data reduction rate meets the requirements (for example, the data reduction rate reaches a maximum, or the data reduction rate is greater than a preset reduction rate). The storage system can then adjust the storage system's data reduction parameters to the target compression parameter and / or target deduplication parameter.

[0122] Among them, the target compression parameter is used to indicate whether to perform compression. When the target compression parameter indicates compression, the target compression parameter can also indicate the compression timing and compression form. Accordingly, the storage system can perform data compression on the data to be reduced (for example, data in the cache) according to the compression timing indicated by the above-mentioned target compression parameter, using the compression mechanism corresponding to the corresponding compression form. Similarly, the target deduplication parameter is used to indicate whether to perform deduplication. When the target deduplication parameter indicates deduplication, the target deduplication parameter can also indicate the deduplication scope, deduplication timing, and deduplication form. The storage system can perform data deduplication on the predicted data block according to the indicated deduplication timing, within the range corresponding to the indicated deduplication scope, and according to the deduplication mechanism corresponding to the deduplication form.

[0123] Based on the above description, it can be seen that the data reduction method of the present application adopts a content-based data reduction feature perception algorithm to actively analyze the data reduction features of the predicted data blocks to obtain real-time data reduction features, and make data reduction scheduling decisions based on the real-time data reduction features (for example, adjusting data reduction parameters). The data reduction strategy (deduplication parameters, compression parameters, etc.) can be dynamically adjusted as the load changes, thereby improving the overall data reduction rate.

[0124] Considering that data compression and data deduplication can affect system performance during the data reduction process, and thus affect other services besides data reduction, the storage system can also obtain the system resource status to realize dynamic data reduction algorithm scheduling based on system resource status perception and data reduction feature perception. As shown in Figure 5, the data reduction device for executing the data reduction method in the storage system may include a perception layer, a decision layer, and an execution layer. The perception layer, decision layer, and execution layer can be deployed on storage nodes in the storage system, including but not limited to storage nodes in a storage system with separated storage and computing, and integrated storage nodes in a storage system with integrated storage and computing.

[0125] Among them, the perception layer can correspond to the load scheduling layer, prediction and evaluation layer, and resource sharing layer of Figure 2, which is used to perceive the system resource status and content-based data reduction characteristics, thereby providing a basis for the decision-making layer. The decision-making layer adopts the idea of ​​decoupling to decouple resource allocation from policy scheduling. Specifically, the decision-making layer includes a resource allocator and a policy scheduler. The resource allocator is used to dynamically adjust the system resources used for data reduction (system resources allocated to data reduction) based on the system resource status. For example, the resource allocator determines the computing power, network and IO magnification factors available for data reduction based on the resource priority to meet the foreground read and write. The policy scheduler determines the data reduction strategy based on the allocation results of system resources (for example, the computing power, network and IO magnification factors available for data reduction) and the data reduction characteristics of the predicted data block. The data reduction strategy includes data reduction parameters that make the data reduction rate meet the requirements, such as target compression parameters and target deduplication parameters. The execution layer can execute the data reduction strategy issued by the decision-making layer. The execution layer can employ a filter mechanism to expand the binary on / off adjustable range of the data reduction algorithm to a continuous space, for example, from 0 to the achievable reduction rate. This supports a richer range of data reduction strategies and improves the reduction rate. Taking deduplication as an example, the execution layer can use the filter mechanism to perform node-level deduplication (such as local deduplication), object-level deduplication, and fingerprint-level deduplication (such as percentage deduplication), thereby expanding the adjustable range of the data reduction algorithm to a continuous space.

[0126] When allocating system resources, the storage system can dynamically adjust the system resources used for data reduction based on the system resource status through reinforcement learning or a decision tree to obtain a system resource allocation result. After obtaining the system resource allocation result, the storage system can adjust the storage system's data reduction parameters based on the system resource allocation result and the predicted data reduction characteristics of the data block to implement data reduction policy scheduling. The storage system can implement data reduction policy scheduling in a variety of ways.

[0127] In some possible implementations, the storage system can determine candidate compression parameters based on the computing power inequality and the allocation results of system resources, and then determine the upper bound of the compression rate based on the candidate compression parameters and the compression rates corresponding to the candidate compression parameters, and determine the upper bound of the deduplication rate based on the network amplification inequality or the storage amplification inequality. The candidate deduplication parameters are determined based on the upper bound of the deduplication rate and the predicted deduplication rates of data blocks under different deduplication parameters. The storage system then updates the candidate compression parameters based on the candidate deduplication parameters and the computing power inequality until a target compression parameter and a target deduplication parameter are determined that satisfy the data reduction rate requirements. The storage system can then adjust the data reduction parameters of the storage system to the target compression parameter and the target deduplication parameter. In other possible implementations, the storage system can also use the allocation results of system resources as constraints and satisfy the data reduction rate requirements as a goal to construct a planning model, and then solve the planning model to obtain the target compression rate and the target deduplication rate. The target compression parameter and the target deduplication parameter are determined based on the target compression rate, the target deduplication rate, and the data reduction characteristics. The storage system can then adjust the data reduction parameters of the storage system to the target compression parameter and the target deduplication parameter.

[0128] Next, different embodiments are used to describe how to implement system resource allocation and data reduction policy scheduling in different ways.

[0129] First, referring to FIG4 , the perception layer of the storage system not only perceives the compression rate and deduplication rate of the predicted data block (including but not limited to the predicted global deduplication rate and the predicted local deduplication rate) based on the content of the predicted data block, but can also perceive the system resource status, including but not limited to CPU utilization, network bandwidth utilization, and backend bandwidth utilization. Furthermore, the storage system can pre-process the system resource status, for example, normalizing and discretizing the utilization of the system resources to obtain a status value. The storage system can also determine the load status of the system resources based on the utilization of the system resources, for example, it can be divided into the following gears according to the size of the utilization: low, middle (abbreviated as mid), high, and overload.

[0130] Then, as shown in Figure 6, the decision-layer resource allocator receives the state values ​​of statistics such as CPU utilization, network bandwidth utilization, and backend bandwidth utilization from the perception layer after normalization and discretization. The resource allocator can use a reinforcement learning algorithm, such as a lightweight Q-learning algorithm, to progressively adjust the system resources allocated to data reduction, including computing power (CPU), network amplification (NET), and bandwidth amplification (BW). Progressive adjustment actions can include rapid increase, increase, no change, decrease, and rapid decrease.

[0131] The reward of Q-learning can be obtained based on the status of the two stages as follows: reward = R(S t ,S t-1 ) (2)

[0132] Among them, S t 、S t-1 Represent the state values ​​of step t and step t-1 respectively.

[0133] The goal of Q-learning can be to maintain high utilization of system resources while avoiding overload. Therefore, Q-learning obtains positive feedback (for example, a reward greater than 0) when the storage system is out of overload or fully utilizing resources; and obtains negative feedback (for example, a reward less than 0) when it enters an overload state or has low resource utilization.

[0134] Taking into account extreme conditions, the storage system can also be configured with a safety guarantee strategy. Specifically, based on domain knowledge, the storage system can inject safety actions through the expert system in specific dangerous scenarios to ensure the stability of the system. Among them, the specific dangerous scenario can be a scenario where the system resource status is in an extreme state, for example, a scenario where system resources such as CPU, BW, NET are all overloaded (for the sake of convenience, also referred to as the first scenario), or a scenario where system resources such as CPU, BW, NET are all underutilized (for the sake of convenience, also referred to as the second scenario). Among them, the safety action corresponding to the first scenario can be a fast decrease, and the safety action corresponding to the second scenario can be a fast increase.

[0135] When the resource allocator determines the allocation of system resources, such as available computing power, network amplification factor, and I / O amplification factor, it can input this allocation result into the policy scheduler. As shown in Figure 6, the policy scheduler can solve the following optimization problem: given computing power, network amplification factor, and I / O amplification factor, with the goal of maximizing the data reduction rate (deduplication rate × compression rate), determine the target compression parameter and target deduplication parameter.

[0136] Among them, the policy scheduler can construct the following inequality based on domain knowledge: CPU (compression algorithm) + CPU (deduplication algorithm) ≤ CPU quota (3) 1 + deduplication network amplification (deduplication algorithm, deduplication rate) ≤ NET amp (4) IO amplification (deduplication algorithm, deduplication rate) ≤ IO amp ×Compression ratio (5)

[0137] Among them, CPU quotaIndicates available computing power, NET amp Indicates the network amplification factor, IO amp Indicates the I / O amplification factor. CPU (Compression Algorithm) indicates the CPU usage of the compression algorithm, and CPU (Deduplication Algorithm) indicates the CPU usage of the deduplication algorithm. Deduplication Network Amplification (Deduplication Algorithm, Deduplication Rate) indicates the network amplification caused by deduplication. I / O Amplification (Deduplication Algorithm, Deduplication Rate) indicates the I / O amplification caused by deduplication.

[0138] In specific implementation, the policy scheduler can be solved according to the following solution pipeline:

[0139] Step (a): Determine candidate compression parameters based on the computing power inequality (such as the above inequality (3)) and the allocation result of system resources.

[0140] Specifically, when different compression parameters are used for data compression, the CPU (compression algorithm) can be different, and the policy scheduler can obtain the mapping relationship between the compression parameters and the CPU (compression algorithm). When the deduplication algorithm is not executed, the maximum value of the CPU (compression algorithm) can be CPU quota The policy scheduler can be based on the CPU quota The mapping relationship between the compression parameters and the CPU (compression algorithm) is used to determine the candidate compression parameters.

[0141] Step (b): determining an upper bound on the compression ratio based on the candidate compression parameters and the predicted compression ratio, and determining an upper bound on the deduplication ratio based on the network amplification inequality or the storage amplification inequality.

[0142] Specifically, after obtaining the predicted compression ratio of the predicted data block under different compression parameters perceived by the perception layer, the policy scheduler can determine the predicted compression ratio of the predicted data block under each candidate compression parameter, and the upper bound of the compression ratio can be determined based on the predicted compression ratio of the predicted data block under each candidate compression parameter. The policy scheduler can then bring the upper bound of the compression ratio into the backend amplification inequality (such as the above-mentioned inequality (5)) to determine the upper bound of the deduplication ratio. The policy scheduler can also obtain another upper bound of the deduplication ratio (such as the global upper bound of the deduplication ratio) based on the network amplification inequality. In some examples, the policy scheduler can determine the minimum value among the upper bounds of the deduplication ratio as the final upper bound of the deduplication ratio.

[0143] Step (c): determining candidate deduplication parameters according to the upper bound of the deduplication rate and the predicted deduplication rates of the data block under different deduplication parameters.

[0144] Specifically, after the policy scheduler obtains the predicted deduplication rate under different deduplication parameters perceived by the perception layer, it can reversely deduce candidate deduplication parameters based on the upper bound of the deduplication rate and the predicted deduplication rate of the data block under different deduplication parameters. The candidate deduplication parameters include deduplication strategy and ratio.

[0145] Step (d): updating the candidate compression parameters according to the candidate deduplication parameters and the computing power inequality until the target compression parameters and target deduplication parameters that make the data reduction rate meet the requirements are determined.

[0146] Specifically, similar to data compression, data deduplication will also occupy the CPU. When different deduplication parameters are used for data deduplication, the CPU (deduplication algorithm) may be different. The policy scheduler can obtain the mapping relationship between the deduplication parameters and the CPU (deduplication algorithm). Then, the policy scheduler can determine the CPU occupied when data deduplication is performed according to the candidate deduplication parameters, that is, the CPU (deduplication algorithm) based on the above mapping relationship. The policy scheduler brings the CPU (deduplication algorithm) back to the computing power inequality to determine whether the computing power inequality holds. If so, the next round of iteration is performed until the target compression parameters and target deduplication parameters are determined so that the data reduction rate meets the requirements (for example, the deduplication rate × compression rate is maximized).

[0147] The policy scheduler can send the target compression parameters and target deduplication parameters that meet the data reduction rate requirements as a data reduction policy to the execution layer. The execution layer applies the target deduplication parameters and target compression parameters to adjust the data reduction policy of the storage system.

[0148] In another embodiment of the data reduction method of the present application, the resource allocator may also be implemented through a feedback system based on expert experience, and the policy scheduler may also be implemented through a general numerical optimization solution method.

[0149] The expert experience-based feedback system may include a decision tree. Specifically, the resource allocator may set thresholds corresponding to different system resources, determine resource utilization status based on the utilization rate of the system resources and the thresholds corresponding to the system resources, and configure data reduction resource targets corresponding to different resource utilization statuses in the form of a decision tree to determine the allocation results of system resources.

[0150] Among them, the decision tree can be an if-else decision tree, as shown below:

[0151] If (system resource utilization < x) -> system resource = a;

[0152] If (x<system resource utilization<y)->system resource=b;

[0153] Else(system resource utilization < y)->system resource = c.

[0154] Hyperparameters x, y, a, and b can be adjusted using a negative feedback mechanism. For example, if the resource allocator in a storage system finds that there are still idle system resources after performing data reduction, it can appropriately increase a and b. This allows for smooth resource allocation and avoids the effects of noise.

[0155] For ease of understanding, the following uses the dynamic adjustment of CPU resources as an example. For example, when the CPU utilization rate < 80%, the resource allocator determines that the data reduction resource target is in a progressive growth state. At this time, the resource allocator can provide a gradually increasing CPU allocation result for data reduction; when the CPU utilization rate reaches 95%, the resource allocator determines that the data reduction resource target is in a stable state and maintains the current CPU allocation; when the CPU utilization rate reaches 100%, the resource allocator determines that the data reduction resource target is in a progressive decrease state. At this time, the resource allocator can provide a gradually decreasing CPU allocation result for data reduction.

[0156] Given the allocation result of system resources by the resource allocator, the policy scheduler (such as a general solver) can use the allocation result of system resources as a constraint condition, aiming to meet the requirement of the data reduction rate, and construct a planning model in combination with the data reduction characteristics of the predicted data block. This planning model can be a linear programming model or a non-linear programming model.

[0157] For ease of understanding, the present application also provides an example of a linear programming model as follows: max compression rate * deduplication rate, α * compression rate + β < CPU resources; γ * deduplication rate + ε < bandwidth resources * compression rate (6)

[0158] Among them, CPU resources and bandwidth resources are the corresponding types of system resources allocated by the resource allocator. α, β, γ, and ε are hyperparameters.

[0159] The general solver can solve the planning model to obtain the target compression rate and target deduplication rate that meet the requirement of the data reduction rate (such as maximizing the data reduction rate). Then, according to the target compression rate, target deduplication rate, and data reduction characteristics (such as the compression rate of the predicted data block under different compression parameters and the deduplication rate under different deduplication parameters), the target compression parameter and target deduplication parameter can be determined from different compression parameters and different deduplication parameters respectively. The target compression parameter and target deduplication parameter can be used as data reduction policies and sent by the decision layer to the execution layer for execution.

[0160] Based on the above description, the present application provides a data reduction strategy for dynamically scheduling a storage system based on the data reduction characteristics of predicted data blocks and the state of system resources, thereby implementing an adaptive adjustment data reduction method. Compared with traditional data reduction methods, the data reduction method of the present application can obtain higher performance under the same reduction rate or higher reduction rate under the same performance.

[0161] Based on the foregoing embodiments of the data reduction method, the present application also provides a data reduction device. The data reduction device will be described below in combination with the embodiments.

[0162] 7 shows a schematic diagram of a data reduction device, the device 700 includes:

[0163] An acquisition module 702 is used to acquire a prediction data block;

[0164] An analysis module 704 is configured to analyze the predicted data block to obtain a data reduction feature of the predicted data block, wherein the data reduction feature of the predicted data block includes a data reduction rate of the predicted data block under different data reduction parameters;

[0165] The adjustment module 706 is configured to adjust data reduction parameters of the storage system according to the data reduction characteristics of the predicted data block.

[0166] Among them, the specific implementation of the acquisition module 702 can refer to the relevant content description of S302 in the embodiment shown in Figure 3, the specific implementation of the analysis module 704 can refer to the relevant content description of S304 in the embodiment shown in Figure 3, and the specific implementation of the adjustment module 706 can refer to the relevant content description of S306 in the embodiment shown in Figure 3.

[0167] In some possible implementations, the data reduction parameter includes at least one of a compression parameter and a deduplication parameter, and the data reduction rate is determined according to at least one of the compression rate and the deduplication rate.

[0168] In some possible implementations, the analysis module 704 is specifically configured to:

[0169] Determining the entropy and relevance of the prediction data block;

[0170] When the entropy of the prediction data block is greater than a first threshold and the correlation degree of the prediction data block is greater than a second threshold, or the entropy of the prediction data block is less than the first threshold, the prediction data block is compressed using different compression parameters to obtain the compression rate of the prediction data block under different compression parameters.

[0171] The specific implementation of the compressibility prediction and the compression rate prediction by the analysis module 704 can be found in the description of the relevant contents of FIG. 4 , which will not be described in detail here.

[0172] In some possible implementations, the predicted data block includes multiple data blocks, and the analysis module 704 is specifically configured to:

[0173] determining fingerprints of the plurality of data blocks, and determining a repetition distribution of the predicted data block based on the fingerprints of the plurality of data blocks;

[0174] Deduplication rates of the predicted data block under different deduplication parameters are determined according to the duplication distribution.

[0175] The specific implementation of the deduplication possibility prediction and the deduplication rate prediction performed by the analysis module 704 can be found in the description of the relevant contents in FIG. 4 , which will not be described in detail here.

[0176] In some possible implementations, the obtaining module 702 is further configured to:

[0177] Acquiring a system resource status, wherein the system resource status includes a utilization rate of at least one system resource;

[0178] The adjustment module 706 is specifically configured to:

[0179] Dynamically adjusting system resources used for data reduction according to the system resource status to obtain an allocation result of the system resources;

[0180] According to the allocation result of the system resources and the data reduction characteristics of the predicted data block, data reduction parameters of the storage system are adjusted.

[0181] The specific implementation of the adjustment module 706 adjusting the data reduction parameters of the storage system in combination with the allocation result of the system resources can be found in the description of the resource allocator and the policy scheduler, for example, see the description of FIG. 5 .

[0182] In some possible implementations, the adjustment module 706 is specifically configured to:

[0183] Determining candidate compression parameters based on the computing power inequality and the allocation result of the system resources;

[0184] Determining an upper bound on the compression ratio based on the candidate compression parameters and the compression ratios corresponding to the candidate compression parameters, and determining an upper bound on the deduplication ratio based on a network amplification inequality or a storage amplification inequality;

[0185] determining a candidate deduplication parameter according to the deduplication rate upper bound and the deduplication rates of the predicted data block under different deduplication parameters;

[0186] Updating the candidate compression parameters according to the candidate deduplication parameters and the computing power inequality until a target compression parameter and a target deduplication parameter are determined so that the data reduction rate meets the requirement;

[0187] The data reduction parameters of the storage system are adjusted to the target compression parameter and the target deduplication parameter.

[0188] Among them, the adjustment module 706 constructs computing power inequality, network amplification inequality, and storage amplification inequality through domain knowledge, and based on the above inequalities, continuously iteratively adjusts the data reduction parameters of the storage system. The specific implementation can be seen in the relevant content description of Figure 6, which will not be repeated here.

[0189] In some possible implementations, the adjustment module 706 is specifically configured to:

[0190] A planning model is constructed with the allocation result of the system resources as a constraint condition and the data reduction rate meeting the requirement as a goal;

[0191] Solving the planning model to obtain a target compression ratio and a target deduplication ratio, and determining a target compression parameter and a target deduplication parameter according to the target compression ratio, the target deduplication ratio, and the data reduction characteristic;

[0192] The data reduction parameters of the storage system are adjusted to the target compression parameter and the target deduplication parameter.

[0193] The adjustment module 706 adjusts the data reduction parameters of the storage system based on the planning model. For details, please refer to the description of the above embodiments, which will not be repeated here.

[0194] In some possible implementations, the adjustment module 706 is specifically configured to:

[0195] According to the system resource status, the system resources used for data reduction are dynamically adjusted through reinforcement learning or decision tree.

[0196] The specific implementation of the adjustment module 706 adjusting the system resources for data reduction can be found in the description of the related content of the above embodiment, which will not be repeated here.

[0197] The present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device, or a data storage device such as a data center that contains one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute the data reduction method described above for use in the data reduction apparatus 700.

[0198] Embodiments of the present application also provide a computer program product comprising instructions. The computer program product may be software or a program product comprising instructions that can be executed on a computing device or stored on any available medium. When the computer program product is executed on at least one computing device, the at least one computing device executes the aforementioned data reduction method.

[0199] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the protection scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A data reduction method, characterized in that: The method comprises: Get the prediction data block; Analyzing the predicted data block to obtain a data reduction feature of the predicted data block, wherein the data reduction feature of the predicted data block includes a data reduction rate of the predicted data block under different data reduction parameters; The data reduction parameters of the storage system are adjusted according to the data reduction characteristics of the predicted data block.

2. The method according to claim 1, characterized in that The data reduction parameter includes at least one of a compression parameter or a deduplication parameter, and the data reduction rate is determined according to at least one of the compression rate or the deduplication rate.

3. The method according to claim 1 or 2, characterized in that: The analyzing the predicted data block to obtain a data reduction feature of the predicted data block includes: Determining the entropy and relevance of the prediction data block; When the entropy of the prediction data block is greater than a first threshold and the correlation degree of the prediction data block is greater than a second threshold, or the entropy of the prediction data block is less than the first threshold, the prediction data block is compressed using different compression parameters to obtain the compression rate of the prediction data block under different compression parameters.

4. The method according to claim 1 or 2, characterized in that: The predicted data block includes a plurality of data blocks, and the analyzing the predicted data block to obtain the data reduction feature of the predicted data block includes: Determining fingerprints of the plurality of data blocks, and determining a repetition distribution of the predicted data block according to the fingerprints of the plurality of data blocks; Deduplication rates of the predicted data blocks under different deduplication parameters are determined according to the repetition distribution.

5. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: Acquiring a system resource status, wherein the system resource status includes a utilization rate of at least one system resource; The adjusting the data reduction parameters of the storage system according to the data reduction characteristics of the predicted data block includes: Dynamically adjusting the system resources used for data reduction according to the system resource status to obtain the allocation result of the system resources; According to the allocation result of the system resources and the data reduction characteristics of the predicted data block, a data reduction parameter of the storage system is adjusted.

6. The method according to claim 5, characterized in that The step of adjusting the data reduction parameters of the storage system according to the allocation result of the system resources and the data reduction characteristics of the predicted data block includes: Determining candidate compression parameters according to the computing power inequality and the allocation result of the system resources; Determining an upper bound of the compression ratio according to the candidate compression parameters and the compression ratios corresponding to the candidate compression parameters, and determining an upper bound of the deduplication ratio according to a network amplification inequality or a storage amplification inequality; Determine a candidate deduplication parameter according to the deduplication rate upper bound and the deduplication rates of the predicted data block under different deduplication parameters; updating the candidate compression parameters according to the candidate deduplication parameters and the computing power inequality until a target compression parameter and a target deduplication parameter that enable a data reduction rate to meet the requirement are determined; The data reduction parameters of the storage system are adjusted to the target compression parameter and the target deduplication parameter.

7. The method according to claim 5, characterized in that The step of adjusting the data reduction parameters of the storage system according to the allocation result of the system resources and the data reduction characteristics of the predicted data block includes: Taking the allocation result of the system resources as a constraint condition and taking the data reduction rate meeting the requirement as a goal, a planning model is constructed; Solving the planning model to obtain a target compression ratio and a target deduplication ratio, and determining a target compression parameter and a target deduplication parameter according to the target compression ratio, the target deduplication ratio and the data reduction characteristic; The data reduction parameters of the storage system are adjusted to the target compression parameter and the target deduplication parameter.

8. The method according to any one of claims 5 to 7, characterized in that: The dynamically adjusting the system resources used for data reduction according to the system resource state includes: According to the system resource status, the system resources used for data reduction are dynamically adjusted through reinforcement learning or decision tree.

9. A data reduction device, characterized in that: The device comprises: An acquisition module, used for acquiring a prediction data block; An analysis module, configured to analyze the prediction data block to obtain a data reduction feature of the prediction data block, wherein the data reduction feature of the prediction data block includes a data reduction rate of the prediction data block under different data reduction parameters; The adjustment module is used to adjust the data reduction parameters of the storage system according to the data reduction characteristics of the predicted data block.

10. The device according to claim 9, characterized in that The data reduction parameter includes at least one of a compression parameter or a deduplication parameter, and the data reduction rate is determined according to at least one of the compression rate or the deduplication rate.

11. The device according to claim 9 or 10, characterized in that The analysis module is specifically used for: Determining the entropy and relevance of the prediction data block; When the entropy of the prediction data block is greater than a first threshold and the correlation degree of the prediction data block is greater than a second threshold, or the entropy of the prediction data block is less than the first threshold, the prediction data block is compressed using different compression parameters to obtain the compression rate of the prediction data block under different compression parameters.

12. The device according to claim 9 or 10, characterized in that The predicted data block includes a plurality of data blocks, and the analysis module is specifically used for: Determining fingerprints of the plurality of data blocks, and determining a repetition distribution of the predicted data block according to the fingerprints of the plurality of data blocks; Deduplication rates of the predicted data blocks under different deduplication parameters are determined according to the repetition distribution.

13. The device according to any one of claims 9 to 12, characterized in that The acquisition module is also used for: Acquiring a system resource status, wherein the system resource status includes a utilization rate of at least one system resource; The adjustment module is specifically used for: Dynamically adjusting the system resources used for data reduction according to the system resource status to obtain the allocation result of the system resources; According to the allocation result of the system resources and the data reduction characteristics of the predicted data block, a data reduction parameter of the storage system is adjusted.

14. The device according to claim 13, characterized in that The adjustment module is specifically used for: Determining candidate compression parameters according to the computing power inequality and the allocation result of the system resources; Determining an upper bound of the compression ratio according to the candidate compression parameters and the compression ratios corresponding to the candidate compression parameters, and determining an upper bound of the deduplication ratio according to a network amplification inequality or a storage amplification inequality; Determine a candidate deduplication parameter according to the deduplication rate upper bound and the deduplication rates of the predicted data block under different deduplication parameters; updating the candidate compression parameters according to the candidate deduplication parameters and the computing power inequality until a target compression parameter and a target deduplication parameter that enable a data reduction rate to meet the requirement are determined; The data reduction parameters of the storage system are adjusted to the target compression parameter and the target deduplication parameter.

15. The device according to claim 13, characterized in that The adjustment module is specifically used for: Taking the allocation result of the system resources as a constraint condition and taking the data reduction rate meeting the requirement as a goal, a planning model is constructed; Solving the planning model to obtain a target compression ratio and a target deduplication ratio, and determining a target compression parameter and a target deduplication parameter according to the target compression ratio, the target deduplication ratio and the data reduction characteristic; The data reduction parameters of the storage system are adjusted to the target compression parameter and the target deduplication parameter.

16. The device according to any one of claims 13 to 15, characterized in that The adjustment module is specifically used for: According to the system resource status, the system resources used for data reduction are dynamically adjusted through reinforcement learning or decision tree.

17. A storage system, characterized in that: The storage system includes a controller and a memory, wherein the memory stores computer-readable instructions; the controller executes the computer-readable instructions so that the storage system executes the method according to any one of claims 1 to 8.

18. A computer-readable storage medium, characterized in that: The method comprises computer-readable instructions; the computer-readable instructions are used to implement the method according to any one of claims 1 to 8.

19. A computer program product, characterized in that The method comprises computer-readable instructions; the computer-readable instructions are used to implement the method according to any one of claims 1 to 8.

Images