A resource management method, device and equipment

Abstract

The application provides a resource management method, device and equipment. The method comprises: for each resource object, the resource object comprising at least one resource identification point, acquiring load data of each resource identification point, the load data representing resource usage of the resource identification point; determining a load level of the resource object based on the load data of each resource identification point; determining an overload control object based on the load level, the overload control object comprising at least one of a background task of the resource object, a foreground service object of the resource object and a token bucket object for counter-pressing host IO; and controlling resources occupied by the overload control object. Through the technical solution of the application, the current load can be prevented from exceeding the maximum processing capacity of the distributed storage system, and the operation stability and service reliability of the distributed storage system can be effectively improved.

Description

A resource management method, device and equipment TECHNICAL FIELD

[0001] The present application relates to the technical field of communication, in particular to a resource management method, device and equipment. BACKGROUND

[0002] A distributed storage system can provide data storage and data protection services, and includes tasks such as feature value-added, cache disk flushing, deduplication compression, garbage collection, and data reconstruction. The above tasks occupy a large amount of resources of the distributed storage system, thereby increasing the load of the distributed storage system.

[0003] When the current load of the distributed storage system exceeds the maximum processing capacity of the distributed storage system, the time delay of IO (Input / Output) operations may increase, the error rate may increase, IO operations may fail, and even business interruption may occur, and problems such as distributed storage system failure may occur.

[0004] SUMMARY

[0005] The present application provides a resource management method applied to a distributed storage system, wherein the distributed storage system includes a plurality of resource objects, and the method comprises:

[0006] For each resource object, the resource object includes at least one resource identification point, and load data of each resource identification point is obtained, wherein the load data represents resource usage of the resource identification point;

[0007] A load level of the resource object is determined based on the load data of each resource identification point;

[0008] If it is determined that the resource object is overloaded based on the load level, an overload control object is determined based on the load level, wherein the overload control object includes at least one of a background task of the resource object, a foreground service object of the resource object, and a token bucket object for backpressure host IO; wherein the background task and the foreground service object share a local token bucket of the resource object;

[0009] The resources occupied by the overload control object are controlled.

[0010] The present application provides a resource management device applied to a distributed storage system, wherein the distributed storage system includes a plurality of resource objects, and the device comprises an acquisition module configured to, for each resource object, obtain load data of at least one resource identification point of the resource object, wherein the load data represents resource usage of the resource identification point;

[0011] determining a load level of the resource object based on the load data of each resource identification point; determining an overload control object based on the load level if it is determined that the resource object is overloaded, the overload control object comprising at least one of a background task of the resource object, a foreground service object of the resource object, and a token bucket object for backpressing host IO; wherein the background task and the foreground service object share a local token bucket of the resource object;

[0012] controlling resources occupied by the overload control object.

[0013] The application provides an electronic device, comprising a processor and a machine readable storage medium, the machine readable storage medium storing machine executable instructions capable of being executed by the processor; wherein the processor is configured to execute the machine executable instructions to implement the resource management method.

[0014] The application provides a computer program product, comprising a computer program, the computer program being executed by a processor to implement the resource management method.

[0015] The application provides a machine readable storage medium, the machine readable storage medium storing machine executable instructions capable of being executed by a processor; wherein the processor is configured to execute the machine executable instructions to implement the resource management method.

[0016] As can be seen from the above technical solutions, in the embodiments of the application, the load level of the resource object can be accurately identified in time, the overload control object is determined based on the load level, and the resources occupied by the overload control object are controlled, so as to reduce the resources occupied by the overload control object, avoid the current load of the distributed storage system exceeding the maximum processing capacity of the distributed storage system, reduce the time delay of the IO operation, make the IO operation have stable time delay, reduce the error rate of the IO operation, avoid the failure of the IO operation, and avoid problems such as service interruption and distributed storage system failure, thereby effectively improving the operation stability and service reliability of the distributed storage system. BRIEF DESCRIPTION OF DRAWINGS

[0017] Fig. 1 is a flow diagram of a resource management method in an embodiment of the application;

[0018] Fig. 2 is a schematic diagram of performance of an IO operation when the distributed storage system is overloaded;

[0019] Fig. 3 is a schematic diagram of a resource identification point of a resource object;

[0020] Fig. 4 is a schematic diagram of the structure of a distributed storage system;

[0021] Figure 5 is a schematic diagram of foreground services and background tasks associated with a resource object;

[0022] Figure 6 is a schematic diagram of an overload control framework;

[0023] Figure 7 is a schematic diagram of a structure of a resource management apparatus in an embodiment of the present application;

[0024] Figure 8 is a hardware structure diagram of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0025] An embodiment of the present application proposes a resource management method, which can be applied to a distributed storage system. The distributed storage system can include a plurality of resource objects. For each resource object, the resource object can be a storage node, or the resource object can be a storage pool. The type of the resource object is not limited. Referring to Figure 1, a flowchart of the method is shown. The method can include the following steps.

[0026] In step 101, for each resource object, the resource object includes at least one resource identification point. Load data of each resource identification point is obtained, which indicates resource usage of the resource identification point.

[0027] In step 102, a load level of the resource object is determined based on the load data of each resource identification point.

[0028] In step 103, if the resource object is determined to be overloaded based on the load level, an overload control object is determined based on the load level. The overload control object can include at least one of a background task of the resource object, a foreground service object of the resource object, and a token bucket object for throttling host IO. The background task and the foreground service object can share a local token bucket of the resource object.

[0029] In step 104, resources occupied by the overload control object are controlled.

[0030] In one example, if the resource object is a storage node, the resource object can include, but is not limited to, at least one of the following resource identification points: a CPU (Central Processing Unit) identification point, a memory identification point, a metadata database identification point, a write cache identification point, a read cache identification point, a token bucket identification point, a storage pool identification point, and a cache background task identification point. If the resource object is a storage pool, the resource object can include, but is not limited to, at least one of the following resource identification points: a memory identification point, a write cache identification point, a read cache identification point, a token bucket identification point, and a cache background task identification point.

[0031] The load data of the CPU identification point can include a CPU usage rate of the storage node; the load data of the memory identification point can include a remaining memory of the storage node; the load data of the metadata database identification point can include a CPU usage rate of the metadata database; the load data of the write cache identification point can include a water level and a time delay of the write cache; the load data of the read cache identification point can include a read cache time delay; the load data of the token bucket identification point can include an IO operation token waiting time delay; and the load data of the storage pool identification point can include a foreground IO statistical time delay; and the load data of the cache background task identification point can include a background task read time delay.

[0032] In one example, the load data of the token bucket identification point can include determining a target ratio between a data length of an IO operation and a configured standard IO length, and determining an IO operation token waiting time delay based on the target ratio and a standard time delay corresponding to the standard IO length; wherein the standard time delay can represent a time delay for processing data of the standard IO length. For the CPU usage rate of the storage node, if the storage node includes multiple CPU cores, a usage rate of a CPU core bound by the IO operation can be determined, and the CPU usage rate of the storage node can be determined based on the usage rate. For the CPU usage rate of the metadata database, if the storage node includes multiple CPU cores, a usage rate of a CPU core bound by the metadata database can be determined, and the CPU usage rate of the metadata database can be determined based on the usage rate.

[0033] In one example, the first level interval includes K control levels, the resource object includes K priority background tasks, and the K priorities correspond to the K control levels one by one; if the load level matches a control level of the first level interval, the overload control object can include a background task of a priority corresponding to the control level. Alternatively, the second level interval includes P control levels, the resource object includes P priority foreground service objects, and the P priorities correspond to the P control levels one by one; if the load level matches a control level of the second level interval, the overload control object can include a foreground service object of a priority corresponding to the control level and all priority background tasks; or, if the load level matches the third level interval, the overload control object can include a token bucket object for throttling host IO, all priority foreground service objects, and all priority background tasks; wherein the control levels of the first level interval are less than the control levels of the second level interval, and the control levels of the second level interval are less than the control levels of the third level interval.

[0034] In one example, the controlling the resources occupied by the overload control object can include, but is not limited to: reducing or increasing (e.g., reducing in the case of overload, increasing in the case of light load) the current resource notch of the overload control object to obtain a control resource notch; controlling the resources occupied by the overload control object based on the control resource notch; wherein the token bucket resources occupied by the overload control object can be controlled based on the token quantity corresponding to the control resource notch, and / or the processing resources occupied by the overload control object can be controlled based on the concurrency quantity corresponding to the control resource notch; wherein the smaller the control resource notch, the smaller the token quantity and concurrency quantity corresponding to the control resource notch.

[0035] In one example, determining the overload control object based on the load level can include, but is not limited to: if the load level matches the control level of the first level interval, and the background task of the priority corresponding to the control level is multiple, selecting a target resource identification point from all resource identification points based on the load data of each resource identification point, and selecting a target background task from the multiple background tasks based on the target resource identification point; wherein the background task sensitive to the target resource identification point is the target background task; the background task sensitive to the target resource identification point refers to that when the resources occupied by the background task are adjusted, the load data of the target resource identification point changes greatly, and when the resources occupied by other background tasks except the background task are adjusted, the load data of the target resource identification point changes little; and determining the target background task as the overload control object.

[0036] In one example, the controlling the resources occupied by the overload control object can include, but is not limited to: if the overload control object is a foreground service object of a resource object, and multiple different foreground service objects correspond to multiple priority queues, determining the token quantity corresponding to the overload control object, and the resource object reads data from the multiple priority queues based on the token quantity and processes the read data. Wherein, the multiple priority queues share a token bucket; different priority queues are used to control dequeuing, that is, when data in different priority queues is dequeued, tokens are taken from a token bucket.

[0037] If the overload control object is a token bucket object for anti-pressure host IO, and the token bucket object corresponds to multiple priority queues, the token quantity corresponding to the overload control object is determined, and the resource object reads data from the multiple priority queues based on the token quantity and processes the read data.

[0038] In the reading of the data from the plurality of priority queues based on the token quantity, the order value of each data in the plurality of priority queues can be determined, and the data with the smallest order value can be read. For example, for each data, the order value of the data can be determined based on the enqueue time of the data and the adjustment value of the priority queue in which the data is located; wherein the earlier the enqueue time of the data, the smaller the order value of the data, and the greater the priority of the priority queue, the smaller the adjustment value, and the smaller the order value of the data.

[0039] In one example, after the resource occupied by the overload control object is controlled, the load level of the resource object can be further counted. If it is determined that the resource object is lightly loaded based on the load level, the load of the overload control object is recovered; wherein when the load of the overload control object is recovered, the load of the token bucket object for backpressure host IO is recovered first, if it is determined that the resource object is lightly loaded based on the load level, the load of the foreground service object of the resource object is further recovered, and if it is determined that the resource object is lightly loaded based on the load level, the load of the background task of the resource object is further recovered.

[0040] As can be seen from the above technical solutions, in the embodiments of the present application, the load level of the resource object can be accurately identified in time, the overload control object is determined based on the load level, the resource occupied by the overload control object is controlled, thereby reducing the resource occupied by the overload control object, avoiding the current load of the distributed storage system exceeding the maximum processing capacity of the distributed storage system, reducing the time delay of the IO operation, making the IO operation have stable time delay, reducing the error rate of the IO operation, avoiding the failure of the IO operation, and avoiding problems such as business interruption and distributed storage system failure, effectively improving the operation stability and business reliability of the distributed storage system.

[0041] The resource management method of the embodiments of the present application will be described below in combination with specific application scenarios.

[0042] The distributed storage system can provide data storage and data protection services. Data storage and data protection are for IO operations (i.e. host IO operations) of the distributed storage system. In addition to IO operations, the distributed storage system also includes tasks such as feature value-added, cache flushing, deduplication compression, garbage collection, and data reconstruction. The above tasks occupy a large amount of resources of the distributed storage system, thereby increasing the load of the distributed storage system. When the current load of the distributed storage system exceeds the maximum processing capacity of the distributed storage system, i.e. the distributed storage system is overloaded, it can cause the time delay of the IO operation to increase, the error rate to increase, the IO operation to fail, and even cause business interruption and problems such as distributed storage system failure.

[0043] Referring to Fig. 2, a performance diagram of an IO operation when a distributed storage system is overloaded is shown. For storage requirements under large concurrent burst pressure, requests accumulate at the storage bottom layer, performance has large fluctuations, background tasks compete for resources of the distributed storage system, causing the performance of the IO operation to sharply decrease, and even to be completely 0.

[0044] For the above discovery, the embodiment proposes a resource management method, which can timely and accurately identify the overload reason when the distributed storage system is overloaded, timely take appropriate overload processing measures, ensure reliable operation of the distributed storage system, and effectively improve the operation stability and business reliability of the distributed storage system.

[0045] The resource management method proposed in the embodiment can involve the following processes:

[0046] First, load data of the distributed storage system is collected. For example, load data of the distributed storage system is periodically collected, and load identification and control of the distributed storage system are performed based on the load data.

[0047] In an example, storage resources of the distributed storage system can be referred to as resource objects, and the resource objects can be storage nodes or storage pools, that is, the resource objects are divided into two types of storage nodes and storage pools, and each storage node or storage pool corresponds to a resource object. A storage node (node) is a single server form providing storage resources, which can also be referred to as a storage server. A storage pool (pool) is a cluster form providing storage resources, and the storage resources in the storage pool can be dynamically expanded or dynamically contracted.

[0048] For each resource object, the resource object can include at least one resource identification point, which can also be referred to as a resource load type, that is, the resource object can be composed of multiple resource load types. Based on this, load data of each resource identification point can be obtained, which indicates resource usage of the resource identification point, and load data of all resource identification points constitutes load data of the resource object.

[0049] In summary, for each resource object, load data of the resource object can be periodically collected, and the load data can include load data of each resource identification point of the resource object.

[0050] In one example, referring to FIG. 3, a diagram of resource identification points of a resource object is shown. For each resource object, if the resource object is a storage node, the resource object can include, but is not limited to, at least one of the following resource identification points (resource load types): a CPU identification point, a memory identification point, a metadata database identification point, a write cache identification point, a read cache identification point, a token bucket identification point, a storage pool identification point, a cache background task identification point. If the resource object is a storage pool, the resource object can include, but is not limited to, at least one of the following resource identification points (resource load types): a memory identification point, a write cache identification point, a read cache identification point, a token bucket identification point, a cache background task identification point. The above are only a few examples of resource identification points, as long as the resource identification points can reflect the resource usage of the resource object.

[0051] The load data of the CPU identification point can include, but is not limited to, the CPU usage of the storage node. For example, if the storage node includes multiple CPU cores, the usage of the CPU core bound to the IO operation can be determined, and the CPU usage of the storage node can be determined based on the usage. The CPU core bound to the IO operation refers to the CPU core used to process the IO operation, and the IO operation is the IO operation to the storage node, such as writing data to the storage node, reading data from the storage node, etc. The usage of the CPU core refers to the usage percentage of the CPU core, such as 50%, etc., indicating that 50% of the CPU core has been used.

[0052] The load data of the memory identification point can include, but is not limited to, the remaining memory of the storage node (or the remaining memory of the storage pool). For example, for the storage node or the storage pool, the available memory of the storage node or the storage pool can be used as the remaining memory of the storage node.

[0053] The load data of the metadata database identification point can include, but is not limited to, the CPU usage of the metadata database. The metadata database refers to a database (DB) used to store metadata, and the metadata refers to metadata of the stored data, which can include the storage location of the stored data, data identification, etc. For example, if the storage node includes multiple CPU cores, the usage of the CPU core bound to the metadata database can be determined, and the CPU usage of the metadata database can be determined based on the usage, such as using the usage as the CPU usage of the metadata database. The CPU core bound to the metadata database refers to the CPU core used to manage the metadata database. The usage of the CPU core refers to the usage percentage of the CPU core, such as 50%, etc., indicating that 50% of the CPU core has been used.

[0054] The load data of the write cache identification point can include, but is not limited to, the water level and the latency of the write cache. For example, for a storage node or a storage pool, the latency of the write cache refers to how long it takes to write data, that is, the delay time of data writing. Obviously, the larger the latency of the write cache, the worse the write performance of the storage node or the storage pool, and it takes longer to complete data writing.

[0055] The load data of the read cache identification point can include, but is not limited to, the read cache latency. For example, for a storage node or a storage pool, the read cache latency refers to how long it takes to successfully read data, that is, the delay time of data reading. Obviously, the larger the latency of the read cache, the worse the read performance of the storage node or the storage pool, and it takes longer to complete data reading.

[0056] The load data of the token bucket identification point can include, but is not limited to, the token waiting latency of the IO operation, which can also be referred to as the token waiting latency of the host IO. For example, for a storage node or a storage pool, a target ratio between the data length of the IO operation and the configured standard IO length is determined, and the token waiting latency of the IO operation is determined based on the target ratio and the standard latency corresponding to the standard IO length, that is, the load data of the token bucket identification point.

[0057] The data length of the IO operation refers to the data length of the IO operation processed by the storage node or the storage pool. The data length can be 16KB, 32KB, 64KB, etc. For a data length of 16KB, it means that the storage node or the storage pool can process the IO operation of the data length of 16KB.

[0058] The standard IO length refers to the standardized data length, which can be a pre-configured data length, such as a standard IO length of 8KB, 16KB, etc. The standard latency can represent the latency of processing data of the standard IO length, that is, the latency under the standard IO length. For example, if the standard IO length is 8KB, the standard latency under the data length of 8KB can be counted, that is, the latency of the IO operation is counted when the storage node or the storage pool processes the IO operation of the data length of 8KB, and this latency is taken as the standard latency.

[0059] Based on this, the proportional relationship (i.e., the target ratio) between the data length and the standard IO length can be pre-configured, such as setting the proportional relationship between the data length of 32KB and the standard IO length of 8KB as 2:1. Given that the sampling latency (i.e., the standard latency) of 32KB (possibly only 32KB, without 8KB size service) is 4ms, then the 8KB standardized latency of the current environment is 2ms, that is, the token waiting latency of the IO operation.

[0060] The load data of the storage pool identification point can include, but is not limited to, foreground IO statistical latency, which can also be referred to as persistent request processing latency. For example, the foreground IO statistical latency of a storage node refers to a statistical latency value of persisting data to a disk when the data needs to be persisted to the disk.

[0061] The load data of the cache background task identification point can include, but is not limited to, background task read latency and background task token latency. For example, the background task read latency of a storage node or a storage pool can be a cache background fill-in read latency, or the background task read latency can be a cache background prefetch read latency. The background task token latency refers to how long a background task takes to obtain a token and then process the background task.

[0062] The above embodiments introduce the load data of each resource identification point, and the load data is not limited.

[0063] In one example, referring to FIG. 3, for each resource object, such as a storage node or a storage pool, basic information of the resource object can be recorded, i.e., basic information used to describe the resource object. For example, the basic information can include, but is not limited to, at least one of the following: a unique identifier of the resource object, a disk type of the resource object (i.e., the resource object uses a storage medium of the disk type to store data), and the like.

[0064] For each resource object, such as a storage node or a storage pool, performance information of the resource object can be recorded, i.e., performance information used to describe the resource object, and the performance entries correspond to the resource identification points.

[0065] For each resource object, such as a storage node or a storage pool, load data of the resource object can be periodically collected, and the load data can include load data of each resource identification point of the resource object. The collection process of the load data is not limited, and the load data of the resource object is recorded.

[0066] As can be seen from the above, in the load data collection process of the distributed storage system, for each resource object, basic information, performance information, and load data of the resource object can be recorded.

[0067] Second, overload recognition (OLR) of the distributed storage system. For example, for each resource object, such as a storage node or a storage pool, after obtaining load data of the resource object, the load data includes load data of each resource identification point, a load level of the resource object is determined based on the load data of each resource identification point, and whether the resource object is overloaded is determined based on the load level.

[0068] In one example, in the load data collection process of the distributed storage system, the basic information of the resource object, the performance information, and the load data of each resource identification point can be recorded, and in the load identification process of the distributed storage system, the changed load data (such as the load data of the resource identification point changes, the load data of the resource identification point is recorded) can also be recorded, and the load level of the resource object is determined based on the load data of each resource identification point.

[0069] In summary, the basic information of the resource object, the performance information, the load data of each resource identification point, the changed load data, and the load level of the resource object can be recorded.

[0070] In one example, the load data of each resource identification point can be normalized, that is, the load data is normalized to the same specified numerical interval (such as the numerical area of 0-1, or the numerical interval of 0-100, etc.). For example, the CPU usage 1 of the storage node can be normalized to obtain the normalized CPU usage 2 in the specified numerical interval, and the greater the CPU usage 1, the greater the load, and the greater the CPU usage 2. The remaining memory 1 of the storage node can be normalized to obtain the normalized remaining memory 2 of the storage node in the specified numerical interval, and the greater the remaining memory 1 of the storage node, the smaller the load, and the smaller the remaining memory 2 of the storage node.

[0071] The CPU usage 3 of the metadata database can be normalized to obtain the normalized CPU usage 4 in the specified numerical interval, and the greater the CPU usage 3, the greater the load, and the greater the CPU usage 4. The write cache latency 1 can be normalized to obtain the normalized write cache latency 2 in the specified numerical interval, and the greater the write cache latency 1, the greater the load, and the greater the write cache latency 2. The read cache latency 1 can be normalized to obtain the normalized read cache latency 2 in the specified numerical interval, and the greater the read cache latency 1, the greater the load, and the greater the read cache latency 2.

[0072] The IO operation token latency 1 can be normalized to obtain the normalized statistical latency 2 in the specified numerical interval, and the greater the statistical latency 1, the greater the load, and the greater the statistical latency 2. The foreground IO statistical latency 1 can be normalized to obtain the normalized foreground IO statistical latency 2 in the specified numerical interval, and the greater the foreground IO statistical latency 1, the greater the load, and the greater the foreground IO statistical latency 2. The background task read latency 1 can be normalized to obtain the normalized background task read latency 2 in the specified numerical interval, and the greater the background task read latency 1, the greater the load, and the greater the background task read latency 2.

[0073] After obtaining the normalized load data, the normalized load data can be weighted to obtain a load parameter value. For example, the CPU usage 2, the remaining memory of the storage node 2, the CPU usage 4, the write cache latency 2, the read cache latency 2, the statistical latency 2, the foreground IO statistical latency 2, and the background task read latency 2 can be weighted, and the weighted result can represent the load parameter value. When the normalized load data is weighted, the weighting coefficient values of different load data can be the same or different. For example, the weighting coefficient value of the CPU usage 2 can be the same or different from the weighting coefficient value of the remaining memory of the storage node 2, and the weighting coefficient value of the CPU usage 2 can be the same or different from the weighting coefficient value of the CPU usage 4. The weighting coefficient values of the load data are not limited.

[0074] Suppose that a total of W load levels need to be divided, if the specified value interval is a value interval of 0-100, and the sum of the weighting coefficient values of all load data is 1, that is, the load parameter value is also in the value interval of 0-100, then the value interval of 0-100 can be divided into W value intervals, and the lengths of different value intervals can be the same or different. For example, taking W as 10 as an example, the value interval of 0-10, the value interval of 11-20, and so on, the value interval of 91-100 can be divided.

[0075] Based on this, if the load parameter value is in the value interval of 0-10, it indicates that the load level of the resource object is 1, if the load parameter value is in the value interval of 11-10, it indicates that the load level of the resource object is 2, and so on, if the load parameter value is in the value interval of 91-100, it indicates that the load level of the resource object is 10. In summary, the load level corresponding to the value interval in which the load parameter value is located (the data interval and the load level one-to-one) can be taken as the load level of the resource object based on the load parameter value.

[0076] For example, after obtaining the load level of the resource object, when the load level of the resource object is small, such as 1, 2, 3, etc., it is determined that the resource object is not overloaded based on the load level. When the load level of the resource object is large, such as 8, 9, 10, etc., it is determined that the resource object is overloaded based on the load level.

[0077] Third, the overload control object identification process of the distributed storage system. For example, for each resource object, such as a storage node or a storage pool, after obtaining the load level of the resource object, it can be determined whether the resource object is overloaded based on the load level. For example, the load level can be divided into light load, normal, and overload levels. In the case of light load, the overload recovery process of the distributed storage system can be executed, see the subsequent process. In the normal case, the overload recovery process of the distributed storage system is not executed, nor is the overload control process of the distributed storage system, but the current state is maintained. When the resource object is determined to be overloaded based on the load level, the overload control object can be determined based on the load level, i.e. the overload control object needs to be controlled to reduce the load of the resource object. The overload control object can include at least one of the background task of the resource object, the foreground business object of the resource object, and the token bucket object for throttling host IO.

[0078] In one example, referring to FIG. 4, a structure diagram of a distributed storage system is shown. The distributed storage system can include multiple resource objects, and for each resource object, the resource object can be a storage node or a storage pool. The client can send the data to be stored to the computing node, and the computing node writes the data to be stored to the high-speed storage area of the resource object, such as SSD (solid state disk), etc. For the case of multiple copies, the computing node can store a copy of the data to multiple resource objects. The resource object can read the data in the SSD (read multiple data at a time), and write the read data to the low-speed storage area of the resource object, such as HDD (mechanical hard disk). Similarly, in the data reading process, the computing node can read data from the resource object and return the read data to the client.

[0079] For the resource object, the resource object can maintain a local token bucket and a token bucket for throttling host IO (also referred to as a global token bucket). For the token bucket for throttling host IO, by controlling the number of tokens in the token bucket, the rate at which the computing node (also referred to as the host) writes data to the resource object and the rate at which the computing node reads data from the resource object can be controlled. Therefore, the above control process is referred to as throttling host IO, i.e. the token bucket can control the host IO of the computing node. For the token bucket for throttling host IO, the initial number of tokens in the token bucket can be set according to experience, and during the operation of the resource object, the number of tokens in the token bucket can be adjusted. The process of controlling the overload control object is the process of adjusting the number of tokens, and by reducing the number of tokens, the purpose of reducing the load is achieved.

[0080] The local token bucket of the resource object is shared by the foreground service object and the background task of the resource object, and the initial number of tokens in the local token bucket can be set according to experience. During the running of the resource object, the number of tokens in the local token bucket can also be adjusted.

[0081] For example, the service related to the IO operation service is referred to as a foreground service object, and the foreground service object can also be referred to as an IO operation service. For example, data writing (such as writing data by a computing node to an SSD of the resource object) and data reading (such as reading data by a computing node from the resource object) of the resource object are referred to as foreground service objects of the resource object. Further, the remaining tasks other than the foreground service objects of the resource object are referred to as background tasks, and the background tasks are tasks that are not in the foreground host service IO flow. For example, feature value-added tasks, deduplication compression tasks, garbage collection tasks, data reconstruction tasks, and the like are all background tasks of the resource object. The above are only a few examples of background tasks, and the background tasks are not limited and can be any task supported by the resource object.

[0082] As described above, the overload control object of the resource object can be divided into foreground service objects and background tasks. The foreground service objects include IO operations on volumes (i.e., data writing and data reading performed on volumes), IO operations on files (i.e., data writing and data reading performed on files), and other types of foreground services can also exist. In addition, the background tasks can include, but are not limited to, feature value-added tasks, cache flushing tasks, deduplication compression tasks, garbage collection tasks, data reconstruction tasks, and the like.

[0083] In one example, the first level interval can be pre-configured, and the priority of the background task can be configured. For example, the first level interval includes K control levels, and the resource object includes K priority background tasks, the K priorities correspond to the K control levels one by one, and K can be a positive integer.

[0084] For example, taking K as 3 as an example, the first level interval includes control level 1, control level 2, and control level 3, and the K priorities include high priority, medium priority, and low priority. For example, taking K as 2 as an example, the first level interval includes control level 1 and control level 2, and the K priorities include high priority and low priority. The number of control levels and priorities can be more, and this is not limited.

[0085] For example, assuming that the resource object corresponds to 10 load levels in total, control level 1 can correspond to load level 4, control level 2 can correspond to load level 5, and control level 3 can correspond to load level 6, which represent that there is some overload, but the overload is not serious. When the load level is less than control level 1, such as load levels 1, 2, and 3, it represents light load or normal.

[0086] All background tasks can be divided into high-priority background tasks, medium-priority background tasks, and low-priority background tasks, the high-priority background tasks correspond to control level 3, the medium-priority background tasks correspond to control level 2, and the low-priority background tasks correspond to control level 1.

[0087] In one example, the second level interval can be pre-configured, and the priority of the foreground service object can be configured. For example, the second level interval includes P control levels, and the resource object includes P foreground service objects of P priorities, the P priorities correspond to the P control levels one by one, and P can be a positive integer.

[0088] For example, the control level of the first level interval can be less than the control level of the second level interval. For example, P is 3, the second level interval includes control level 4, control level 5, and control level 6, and the P priorities include high priority, medium priority, and low priority. For another example, P is 2, the second level interval includes control level 3 and control level 4, and the P priorities include high priority and low priority.

[0089] For example, assuming that the resource object corresponds to 10 load levels in total, control level 4 can correspond to load level 7, control level 5 can correspond to load level 8, and control level 6 can correspond to load level 9, which represent that there is overload, and the overload is serious.

[0090] All foreground service objects can be divided into high-priority foreground service objects, medium-priority foreground service objects, and low-priority foreground service objects, the high-priority foreground service objects correspond to control level 6, the medium-priority foreground service objects correspond to control level 7, and the low-priority foreground service objects correspond to control level 8. For example, the IO operation for file 1 is taken as a high-priority foreground service object, the IO operation for file 2 is taken as a medium-priority foreground service object, and so on.

[0091] In one example, for each resource object, the object information of the foreground service object associated with the resource object can be recorded, and the object information of the foreground service object can include but is not limited to: type (lun / fs), object identifier, priority (such as high priority, medium priority, and low priority).

[0092] In one example, for each resource object, the object information of the background task associated with the resource object can be recorded, which can include but is not limited to: background task type (such as characteristic value-added task, cache flushing task, re-deletion compression task, garbage collection task, data reconstruction task, etc.), object identifier, current resource gear, resource gear lower limit value, resource gear upper limit value, resource sensitive type (such as CPU identification point, memory identification point, write cache identification point, etc.), background task control type and strategy (such as using token bucket for control or using concurrent number for control), whether front and back balance is needed (if front and back balance is needed, it means that the background task is closely related to the foreground service), whether there is a reliability event, priority (such as high priority, medium priority and low priority).

[0093] In the overload control process, the foreground service and the background task of the resource object need to be balanced, therefore, for the background task that needs front and back balance, or for the background task that has been marked as "having a reliability event", the priority needs to be increased, that is, as a high-priority background task, so as to reduce the probability of being controlled, that is, to ensure that the background task can run normally.

[0094] As can be seen from the above, for each resource object, the priority of the foreground service object associated with the resource object and the priority of the background task associated with the resource object can be obtained.

[0095] Referring to FIG. 5, a schematic diagram of the foreground service object and the background task associated with the resource object is shown, the resource object can be a storage node or a storage pool, the resource object can be associated with multiple background tasks, these background tasks can correspond to K priorities, for example, background task 1 and background task 2 correspond to high priority, background task 3 and background task 4 correspond to medium priority, and background task 5 corresponds to low priority. The resource object can also be associated with multiple foreground service objects, and these foreground service objects can correspond to P priorities.

[0096] In one example, when determining the overload control object based on the load level, the control order of background task first and foreground service object second can be used, and for the background task, the control order of low priority, medium priority and high priority is used, and for the foreground service object, the control order of low priority, medium priority and high priority is used.

[0097] In one example, when determining overload control objects based on load levels, local overload control can be attempted first, and global overload control can be performed when the local overload control fails. For example, the local overload control refers to taking the background task of the resource object and the foreground service object of the resource object as overload control objects, and performing overload control on the overload control objects. In addition, the global overload control refers to taking the token bucket object for resisting host IO as an overload control object, and performing overload control on the overload control object.

[0098] In order to achieve the control sequence of first background task and then foreground service, and the control sequence of first local overload control and then global overload control, if the load level of the resource object matches the control level of the first level interval, the overload control object can include the background task of the resource object, that is, the background task of the resource object can be taken as an overload control object, and overload control can be performed on the overload control object. If the load level of the resource object matches the control level of the second level interval, the overload control object can include the background task of the resource object and the foreground service object of the resource object, that is, the background task of the resource object and the foreground service object of the resource object can be taken as overload control objects, and overload control can be performed on the overload control objects. Obviously, based on the above control process, the control sequence of first background task and then foreground service can be achieved.

[0099] In addition, if the load level matches the control level of the third level interval, the overload control object can include the background task of the resource object, the foreground service object of the resource object, and the token bucket object for resisting host IO, that is, the background task of the resource object, the foreground service object of the resource object, and the token bucket object for resisting host IO can be taken as overload control objects, and overload control can be performed on the overload control objects. Based on the above control process, the control sequence of first local overload control and then global overload control can be achieved.

[0100] Among them, the third level interval can be preconfigured, the control level of the second level interval is less than the control level of the third level interval, and assuming that the resource object corresponds to 10 load levels in total, the third level interval can include the control level 7, and the control level 7 can correspond to the load level 10, the load level 10 indicates that there is overload, and the overload is very serious, and overload control needs to be performed on the token bucket object for resisting host IO.

[0101] In one example, the control sequence of low priority, medium priority and high priority is adopted for the background task, and based on this, if the load level of the resource object matches the control level of the first level interval, the overload control object can include the background task of the priority corresponding to the control level and the background task lower than the priority. For example, if the load level of the resource object matches the control level 1 (the control level 1 can correspond to the load level 4), such as the load level being the load level 4, the background task of low priority is taken as the overload control object. If the load level of the resource object matches the control level 2, such as the load level being the load level 5, the background task of low priority and the background task of medium priority are taken as the overload control object. If the load level of the resource object matches the control level 3, such as the load level being the load level 6, the background task of low priority, the background task of medium priority and the background task of high priority are taken as the overload control object.

[0102] The control sequence of low priority, medium priority and high priority is adopted for the foreground service object, and based on this, if the load level of the resource object matches the control level of the second level interval, the overload control object can include the foreground service object of the priority corresponding to the control level, the foreground service object lower than the priority and the background task of all priorities. For example, if the load level of the resource object matches the control level 4, such as the load level being the load level 7, the foreground service object of low priority and the background task of all priorities are taken as the overload control object. If the load level of the resource object matches the control level 5, such as the load level being the load level 8, the foreground service object of medium priority, the foreground service object of low priority and the background task of all priorities are taken as the overload control object. If the load level of the resource object matches the control level 6, such as the load level being the load level 9, the foreground service object of high priority, the foreground service object of medium priority, the foreground service object of low priority and the background task of all priorities are taken as the overload control object.

[0103] In addition, if the load level of the resource object matches the control level 7 of the third level interval, such as the load level being the load level 10, the token bucket object for anti-pressure host IO, the foreground service object of all priorities and the background task of all priorities can all be taken as the overload control object.

[0104] In one example, if the load level matches the control level of the first level interval and the background task of the priority corresponding to the control level is multiple, the target resource identification point can be selected from all resource identification points based on the load data of each resource identification point, the target background task is selected from the multiple background tasks based on the target resource identification point, and the target background task is determined as the overload control object.

[0105] For example, a background task sensitive to a target resource identification point can be a target background task. The background task sensitive to the target resource identification point refers to that, when the resource occupied by the background task is adjusted, the load data of the target resource identification point changes greatly, and when the resource occupied by other background tasks is adjusted, the load data of the target resource identification point changes slightly.

[0106] For example, a control threshold can be set for each resource identification point, such as setting a control threshold 1 for a CPU identification point, a control threshold 2 for a memory identification point, a control threshold 3 for a metadata database identification point, a control threshold 4 for a write cache identification point, a control threshold 5 for a read cache identification point, a control threshold 6 for a token bucket identification point, a control threshold 7 for a storage pool identification point, and a control threshold 8 for a cache background task identification point. For a disk read latency identification point, thresholds of background tasks of low, medium and high priorities are set for HDD / SSD types. A percentage threshold is used for the CPU identification point for flow control of background tasks of medium and high priorities. A percentage threshold is used for the metadata database identification point for flow control of background tasks of low priority. Thresholds of background tasks of low, medium and high priorities are set for the token bucket identification point. Read latency thresholds of background tasks of medium and high priorities are set for flow control of background tasks of low priority.

[0107] Based on the control threshold of a resource identification point and the load data of the resource identification point, if the load data exceeds the range of the control threshold, the resource identification point is taken as a target resource identification point. If the load data does not exceed the range of the control threshold, the resource identification point is not taken as a target resource identification point.

[0108] For example, if the CPU usage of a storage node is greater than the control threshold 1, the CPU identification point is taken as a target resource identification point, otherwise, the CPU identification point is not taken as a target resource identification point. If the remaining memory of the storage node is less than the control threshold 2, the memory identification point is taken as a target resource identification point. If the CPU usage of a metadata database is greater than the control threshold 3, the metadata database identification point is taken as a target resource identification point. If the write cache latency is greater than the control threshold 4, the write cache identification point is taken as a target resource identification point. If the read cache latency is greater than the control threshold 5, the read cache identification point is taken as a target resource identification point. If the token waiting latency of an IO operation is greater than the control threshold 6, the token bucket identification point is taken as a target resource identification point. If the foreground IO statistical latency is greater than the control threshold 7, the storage pool identification point is taken as a target resource identification point. If the background task read latency is greater than the control threshold 8, the cache background task identification point is taken as a target resource identification point.

[0109] After the target resource identification point is obtained, a background task sensitive to the target resource identification point can be determined. For example, the object information of the background task associated with the resource object includes a resource sensitive type, which indicates which resource identification points the background task is sensitive to. Based on this, whether the background task is sensitive to the target resource identification point can be known based on the object information of the background task, so that the background task sensitive to the target resource identification point can be known, and the background task sensitive to the target resource identification point is taken as the target background task. Then, the target background task can be determined as the overload control object.

[0110] For example, the background task sensitive to the token bucket identification point refers to that when the resource occupied by the background task is adjusted, the token waiting time of the IO operation changes greatly, and when the resource occupied by other background tasks other than the background task is adjusted, the token waiting time of the IO operation changes little.

[0111] For example, if the load level of the resource object matches the control level 1, the target background task is selected from all low-priority background tasks, and the target background task is taken as the overload control object.

[0112] If the load level of the resource object matches the control level 2, the target background task is selected from all medium-priority background tasks, and the target background task and all low-priority background tasks are taken as the overload control object. Alternatively, the target background task is selected from all medium-priority background tasks, the target background task is selected from all low-priority background tasks, and the target background task is taken as the overload control object.

[0113] In one example, if the load level matches the control level of the second grade interval, and the foreground service object corresponding to the priority of the control level is multiple, the target resource identification point can be selected from all resource identification points based on the load data of each resource identification point, the target foreground service object is selected from the multiple foreground service objects based on the target resource identification point, and the target foreground service object is determined as the overload control object.

[0114] For example, the foreground service object sensitive to the target resource identification point can be taken as the target foreground service object. The foreground service object sensitive to the target resource identification point refers to that when the resource occupied by the foreground service object is adjusted, the load data of the target resource identification point changes greatly, and when the resource occupied by other foreground service objects other than the foreground service object is adjusted, the load data of the target resource identification point changes little.

[0115] Fourth, an overload handling (OLH) procedure of the distributed storage system. For example, for each resource object, such as a storage node or a storage pool, after obtaining an overload handling object of the resource object, the resources occupied by the overload handling object can be controlled, such as reducing the resources occupied by the overload handling object, so as to reduce the load of the resource object.

[0116] In one example, N resource gears can be pre-divided, and the resource gears represent the resource size that can be occupied by the background task or the foreground service object. The resource can be a token bucket resource, a processing resource, or other resources, and the resource type is not limited. The load of the resource object can be reduced by controlling the resource. For the token bucket resource, the resource gear represents the number of tokens (i.e., the number of tokens in the local token bucket) that can be occupied by the background task or the foreground service object, and the load of the resource object can be reduced by controlling the number of tokens. In addition, for the processing resource (concurrent processing resource), the resource gear represents the number of concurrent numbers that can be occupied by the background task or the foreground service object, and the load of the resource object can be reduced by controlling the number of concurrent numbers.

[0117] For each resource gear, the resource size (such as the number of tokens and the number of concurrent numbers) corresponding to the resource gear can be pre-determined. For example, resource gear a1, resource gear a2, resource gear a3, and the like can be pre-divided, and a total of 8 resource gears are divided. It is assumed that the resource size b1 (for example, the resource of the local token bucket represents b1 tokens occupied in the local token bucket) corresponding to the resource gear a1, the resource size b2 corresponding to the resource gear a2, and the resource size b2 can be greater than the resource size b1, the resource size b3 corresponding to the resource gear a3, and the like. Each resource gear corresponds to a resource size.

[0118] In order to determine the resource size corresponding to the resource notch, the resource size corresponding to the resource notch can be configured according to experience, or a certain strategy can be used to determine the resource size corresponding to the resource notch. For example, according to the storage capacity of the distributed storage system, the storage capacity is converted into IOPS (Input / Output Operations Per Second) resources under a standard IO length (such as 8K) (the IOPS resources under the standard IO length can determine the token quantity of the local token bucket), and the IOPS resources are divided into resource sizes of 8 notches (which can be average division or non-average division), and the resource sizes of the 8 notches are the resource sizes corresponding to the 8 resource notches. For example, if the IOPS resources under the standard IO length are 80, then the resource size b1 corresponding to the resource notch a1 is 10, the resource size b2 corresponding to the resource notch a2 is 20, and so on, and the resource size b8 corresponding to the resource notch a8 is 80.

[0119] For example, the standard capacity provided by the resource object (such as a storage node or a storage pool) can be calculated according to a certain read-write ratio (such as 7:3), and the resource size corresponding to the resource notch can be determined based on the capacity.

[0120] For the background tasks of the resource object, resource notches a1-a8 can be corresponded, and for the foreground business objects of the resource object, resource notches a1-a8 can also be corresponded.

[0121] For each background task associated with the resource object, the background task corresponds to a control resource notch and a running resource notch (i.e. the current resource notch). For each foreground business object associated with the resource object, the foreground business object corresponds to a control resource notch and a running resource notch (i.e. the current resource notch).

[0122] For example, assuming that the current resource notch of the background task is resource notch a6, it indicates that the background task is currently using the resource size corresponding to resource notch a6. Assuming that the control resource notch of the background task is resource notch a4, it indicates that the background task currently using resource notch a6 needs to be adjusted to resource notch a4, that is, the background task uses the resource size corresponding to resource notch a4, so as to reduce the load of the resource object.

[0123] For each background task associated with a resource object, a reference resource notch can be maintained for the background task, which serves as the initial control resource notch when the background task is created. For each foreground business object associated with a resource object, a reference resource notch can be maintained for the foreground business object, which serves as the initial control resource notch when the foreground business object is created. For example, assume that the reference resource notch of a background task is resource notch a6, then when the background task is created, the control resource notch of the background task is resource notch a6, i.e. the resource size corresponding to resource notch a6 is used when the background task is created. During the running of the background task, the control resource notch can be dynamically adjusted based on the load of the resource object.

[0124] When the resource object is overloaded, the reference resource notch of the background task can also be adjusted, so that the initial control resource notch when the background task is created is reduced, and the load of the resource object is reduced. For example, assume that the reference resource notch of the background task should be resource notch a6, however, due to the overload of the resource object, the reference resource notch of the background task is adjusted to resource notch a4. In this way, when the background task is created, the control resource notch of the background task is resource notch a4, i.e. the resource size corresponding to resource notch a4 is used when the background task is created, instead of the resource size corresponding to resource notch a6.

[0125] For the reference resource notch of a background task, the reference resource notch of a background task of different resource sensitivity type can be specified respectively, such as pre-specifying the following information: background task type, default reference resource notch 1 of a CPU-sensitive background task, and default reference resource notch 2 of a memory-sensitive background task. Based on this, for each background task, if the background task is a CPU-sensitive background task, the default reference resource notch 1 is taken as the reference resource notch of the background task, and if the background task is a memory-sensitive background task, the default reference resource notch 2 is taken as the reference resource notch of the background task.

[0126] In one example, when the resource occupied by an overload control object is controlled, the current resource notch (i.e. the running resource notch) of the overload control object can be reduced to obtain a control resource notch, and the resource occupied by the overload control object is controlled based on the control resource notch. The control resource notch is not less than the resource notch lower limit value of the overload control object, and if the current resource notch is the resource notch lower limit value, the current resource notch of the overload control object is not reduced any more, i.e. the resource occupied by the overload control object is not controlled.

[0127] When controlling the resources occupied by the overload control object, the token bucket resources occupied by the overload control object can be controlled based on the token quantity corresponding to the control resource gear, and / or the processing resources occupied by the overload control object can be controlled based on the concurrency quantity corresponding to the control resource gear.

[0128] The resource control process of the overload control object is described below in combination with several specific cases.

[0129] Case 1: The overload control object is a background task associated with a resource object.

[0130] For example, assuming that the current resource gear of the background task is resource gear a6, the control resource gear of the background task can be determined. The higher the load level, the lower the control resource gear, and the control resource gear is not less than the resource gear lower limit value (such as resource gear a2 as the resource gear lower limit value) of the background task. No limitation is imposed on the control resource gear, and it is assumed that the control resource gear is resource gear a4. When the control resource gear is determined, if the current resource gear has reached the resource gear lower limit value, it indicates that the resources occupied by the background task are no longer controlled, and the resources occupied by other overload control objects need to be adjusted.

[0131] After obtaining the control resource gear of the background task, the resources occupied by the background task can be controlled based on the control resource gear, such as adjusting the resource gear a6 to the resource gear a4, and the background task uses the resource size corresponding to the resource gear a4. For example, the token quantity corresponding to the resource gear a4 can be determined, and the token bucket resources occupied by the background task are controlled based on the token quantity corresponding to the resource gear a4, that is, the background task needs to use the token quantity corresponding to the resource gear a4 for processing. Obviously, the smaller the control resource gear, the smaller the token quantity, that is, by reducing the token quantity, the purpose of reducing the load is achieved.

[0132] In addition, the concurrency quantity corresponding to the resource gear a4 can be determined, and the processing resources occupied by the background task are controlled based on the concurrency quantity corresponding to the resource gear a4, that is, the background task needs to use the concurrency quantity corresponding to the resource gear a4 for processing (that is, concurrent processing of data). Obviously, the smaller the control resource gear, the smaller the concurrency quantity, that is, by reducing the concurrency quantity, the purpose of reducing the load is achieved.

[0133] Case 2: The overload control object is a foreground service object associated with a resource object.

[0134] For example, assuming that the current resource notch of the foreground service object is resource notch a6, the control resource notch of the foreground service object can be determined, the control resource notch is lower when the load level is higher, and the control resource notch is not less than the lower limit value of the resource notch of the foreground service object. When the control resource notch is determined, if the current resource notch has reached the lower limit value of the resource notch, it indicates that the resource occupied by the foreground service object is no longer controlled. After obtaining the control resource notch of the foreground service object, the resource occupied by the foreground service object can be controlled based on the control resource notch, that is, the resource size corresponding to the control resource notch is used.

[0135] For example, the number of tokens corresponding to the control resource notch can be determined, and the token bucket resource occupied by the foreground service object is controlled based on the number of tokens corresponding to the control resource notch, that is, the foreground service object needs to use the number of tokens corresponding to the control resource notch for processing, so as to achieve the purpose of reducing the load.

[0136] Case 3: The overload control object is a token bucket object for anti-pressure host IO.

[0137] For example, the control resource notch of the token bucket object can be determined, and the resource occupied by the token bucket object is controlled based on the control resource notch, that is, the resource size corresponding to the control resource notch is used. For example, the number of tokens of the token bucket object is determined based on the control resource notch, that is, the token bucket object uses the number of tokens, and then the data writing speed and the data reading speed are controlled based on the number of tokens.

[0138] For example, the number of tokens corresponding to the control resource notch can be determined, and the token bucket resource occupied by the token bucket object is controlled based on the number of tokens corresponding to the control resource notch, that is, the token bucket object needs to use the number of tokens corresponding to the control resource notch for processing, so as to achieve the purpose of reducing the load.

[0139] In one example, case 1 and case 2 are local overload control, and case 3 is global overload control. The local overload control and the global overload control are described below. Referring to FIG. 6, which is a schematic diagram of an overload control framework, a storage node includes a protocol process and a data engine process, the protocol process provides protocol access services, and the data engine process provides data services, various background tasks (such as value-added features, metadata merging, GC, data reconstruction, etc.) run in the data engine process, and the overload control relies on a distributed QoS framework.

[0140] The data engine process can include an overload_ctrl module and a local QoS client. The overload_ctrl module can identify resource object overload (e.g., determine a load level of a resource object based on load data, determine an overload control object based on the load level), and perform local overload control (e.g., perform local overload control using Case 1 or Case 2). The overload_ctrl module can send resource object overload information to an overload_ctrl leader module, and the overload_ctrl leader module can perform global overload control (e.g., perform global overload control using Case 3).

[0141] The QoS center can also include an overload_ctrl leader module and a QoS server. The overload_ctrl module can send resource object overload information to the local QoS client, the local QoS client can send the resource object overload information to the QoS server, and the QoS server can send the resource object overload information to the overload_ctrl leader module.

[0142] For Case 3, the overload_ctrl leader module can throttle host IO based on the number of tokens corresponding to the control resource gear. For example, by reducing the number of tokens in the token bucket (i.e., the global token bucket), the amount of data stored in the token bucket is reduced, and when the amount of data stored in the token bucket is reduced, the amount of data written by the compute node to the storage object is reduced, a process referred to as throttling host IO. When the amount of data written by the compute node to the resource object is reduced, the amount of data that the resource object needs to process is reduced, reducing the load on the resource object.

[0143] For Case 3, the scenario of global overload control can include storage pool overload, requiring control of all storage nodes. When a storage node is heavily overloaded, global throttling of host IO is required. If the current load level exceeds the sum of the levels that can be controlled locally, global overload control is relied on to throttle host IO.

[0144] In the above scenarios, Case 3 can be used to perform global overload control. When performing global overload control, host IO flow control is controlled in order of priority from low to high.

[0145] When the storage pool is overloaded or a storage node is heavily overloaded, overload information needs to be reported to the QoS center, and the QoS center performs global overload control on the overload control object associated with the resource object.

[0146] In one example, for case 2, if the overload control object is a foreground service object of a resource object, and multiple different foreground service objects correspond to multiple priority queues, the number of tokens corresponding to the overload control object (i.e., the number of tokens corresponding to the control resource notch) is determined, the control resource object reads data from the multiple priority queues based on the number of tokens (i.e., reads data matching the number of tokens), and the read data is processed. The multiple priority queues share one token bucket, and different priority queues are used to control dequeuing, i.e., when data in different priority queues is dequeued, tokens are taken from the token bucket.

[0147] For example, when a foreground service object corresponds to multiple priority queues, data can be read from the multiple priority queues based on the number of tokens. When reading data, the order value of each data in the multiple priority queues can be determined, and the data with the smallest order value is read. For example, when the order value of data A1 in priority queue 1 is the smallest, data A1 is read from priority queue 1, and then, when the order value of data A2 in priority queue 3 is the smallest, data A2 is read from priority queue 3, and so on.

[0148] For example, for each data, the order value of the data is determined based on the enqueue time of the data and the adjustment value of the priority queue in which the data is located. If the enqueue time of the data is earlier, the order value of the data is smaller, i.e., the order value of the data is proportional to the enqueue time of the data. If the priority of the priority queue in which the data is located is higher, the adjustment value of the priority queue is smaller, and the order value of the data is smaller, i.e., the order value of the data is inversely proportional to the priority queue in which the data is located.

[0149] For example, the order value of the data can be determined using the following formula: T+1 / Priority, which is only an example and does not limit the determination method. T represents the enqueue time of the data, i.e., the time when the data is written to the priority queue. Priority represents the priority of the priority queue in which the data is located, for example, the priority Priority of a low-priority queue is 1, the priority Priority of a medium-priority queue is 2, and the priority Priority of a high-priority queue is 3. 1 / Priority represents the adjustment value of the priority queue, and obviously, the adjustment value of a high-priority queue is 1 / 3, i.e., the higher the priority of the priority queue, the smaller the adjustment value.

[0150] Based on the above IO scheduling algorithm, the higher the priority and the earlier the enqueue time, the first to be scheduled, avoiding the situation that data is locked in the priority queue and some data cannot be read for a long time.

[0151] When the data engine process creates the token bucket based on the storage node and the storage pool, the IO request first acquires a token and then is dispatched. When the front and back stages of the storage resource cannot be balanced, the overload control notifies the data engine process to resist the host IO. The front and back stage IOs are put into the front high / medium / low and back priority queues. The IO queuing delay is controlled, and the unscheduled IOs in the queue are discarded before the host IO times out. The host IOs associated with the high / medium / low priority of the resource object and the back stage IOs of the back stage priority are flow controlled, and different priority flow control is realized through the IO dispatcher.

[0152] In one example, for case 3, if the overload control object is a token bucket object for resisting the host IO, and the token bucket object corresponds to multiple priority queues, the number of tokens corresponding to the overload control object is determined, and the resource object reads data from the multiple priority queues based on the number of tokens, and processes the data.

[0153] For example, when the token bucket object corresponds to multiple priority queues, data can be read from the multiple priority queues based on the number of tokens. When reading the data, the order value of each data in the multiple priority queues can be determined, and the data with the smallest order value is read. For each data, the order value of the data is determined based on the enqueue time of the data and the adjustment value of the priority queue where the data is located. If the enqueue time of the data is earlier, the order value of the data is smaller, if the priority of the priority queue where the data is located is higher, the adjustment value of the priority queue is smaller, and the order value of the data is smaller.

[0154] Fifth, the overload recovery (OLC) process of the distributed storage system. For example, after the resources occupied by the overload control object are controlled, the load level of the resource object can be further counted. If the load level is less than the preset control level (the preset control level can be the minimum control level in the first level interval, such as control level 1, and the control level 1 corresponds to the load level 4), such as the load level of the resource object is 1, 2, 3, in this case, it indicates that the resource object is lightly loaded, and the load of the overload control object can be recovered.

[0155] In one example, when the resource object is identified as lightly loaded, such as the load level is 1, 2, 3, etc., the load of each overload control object can be recovered in turn, and the order of first recovering the front-end business and then recovering the back-end task is adopted. Based on this, when the load of the overload control object is recovered, the load of the token bucket object for resisting the host IO can be recovered first, and the load level of the resource object is recalculated.

[0156] If the load level is still less than the preset control level (i.e., the resource object is identified as lightly loaded), the load of the foreground service object of the resource object can be continued to be recovered, and the load level of the resource object is re-counted. If the load level is still less than the preset control level (i.e., the resource object is identified as lightly loaded), the load of the background task of the resource object can be continued to be recovered.

[0157] In one example, when the load of the foreground service object of the resource object is recovered, the recovery can be gradually controlled in the order of priority from high to low. For example, the load of the foreground service object with high priority is recovered first, if the load level is less than the preset control level, the load of the foreground service object with medium priority is continued to be recovered, if the load level is less than the preset control level, the load of the foreground service object with low priority is continued to be recovered.

[0158] When the load of the background task of the resource object is recovered, the recovery can be gradually controlled in the order of priority from high to low. For example, the load of the background task with high priority is recovered first, if the load level is less than the preset control level, the load of the background task with medium priority is continued to be recovered, if the load level is less than the preset control level, the load of the background task with low priority is continued to be recovered.

[0159] As can be seen from the above technical solutions, in the embodiments of the present application, a storage node and a storage pool overload timely and accurately identification method is provided. A distributed storage overload control method is provided to ensure the reliability of critical services. A distributed storage overload recovery method is provided to ensure smooth service recovery. By providing a distributed storage overload identification, overload control, and overload recovery method, the I / O has a relatively stable time delay, the distributed storage system is avoided from being abnormal, and the operation stability and service reliability of the distributed storage system are effectively improved. The background task can be controlled to avoid the burst of background tasks from collapsing the distributed storage system, and the impact on the foreground IO is minimized, such as deleting snapshots, splitting volumes, background synchronization, GC, etc. The key background tasks requiring foreground and background balance are timely recovered to ensure service continuity and reliability and ensure the reliability of high-priority services.

[0160] Based on the same application concept as the above method, in the embodiments of the present application, a resource management device is provided, which is applied to a distributed storage system, the distributed storage system includes a plurality of resource objects, as shown in FIG. 7, which is a structural schematic diagram of the resource management device, the device can include:

[0161] The acquisition module 71 is configured to acquire, for each resource object including at least one resource identification point, load data of each resource identification point, the load data representing resource usage of the resource identification point; the determination module 72 is configured to determine a load level of the resource object based on the load data of each resource identification point; if it is determined that the resource object is overloaded based on the load level, determine an overload control object based on the load level, the overload control object including at least one of a background task of the resource object, a foreground service object of the resource object, and a token bucket object for backpressure host IO; wherein the background task and the foreground service object share a local token bucket of the resource object; the control module 73 is configured to control resources occupied by the overload control object.

[0162] In one example, if the resource object is a storage node, the resource object includes at least one of the following resource identification points: a CPU identification point, a memory identification point, a metadata database identification point, a write cache identification point, a read cache identification point, a token bucket identification point, a storage pool identification point, and a cache background task identification point; if the resource object is a storage pool, the resource object includes at least one of the following resource identification points: a memory identification point, a write cache identification point, a read cache identification point, a token bucket identification point, and a cache background task identification point; wherein the load data of the CPU identification point includes CPU usage of the storage node; the load data of the memory identification point includes remaining memory of the storage node; the load data of the metadata database identification point includes CPU usage of the metadata database; the load data of the write cache identification point includes water level and latency of the write cache; the load data of the read cache identification point includes read cache latency; the load data of the token bucket identification point includes IO operation token latency; the load data of the storage pool identification point includes foreground IO statistical latency; and the load data of the cache background task identification point includes background task read latency.

[0163] In one example, when the determination module 72 determines the overload control object based on the load level, a first level interval includes K control levels, the resource object includes K priority background tasks, the K priorities correspond to the K control levels one by one; if the load level matches a control level of the first level interval, the overload control object includes the priority background task corresponding to the control level;

[0164] Or, a second level interval includes P control levels, the resource object includes P priority foreground service objects, the P priorities correspond to the P control levels one by one; if the load level matches a control level of the second level interval, the overload control object includes the priority foreground service object corresponding to the control level and all priority background tasks;

[0165] Or, if the load level matches the third level interval, the overload control object includes the token bucket object, foreground service objects of all priorities, and background tasks of all priorities.

[0166] Wherein, the control level of the first level interval is less than the control level of the second level interval, and the control level of the second level interval is less than the control level of the third level interval.

[0167] In one example, when the control module 73 controls the resources occupied by the overload control object, it is specifically used for: reducing or increasing the control resource gear from the current resource gear of the overload control object;

[0168] The resources occupied by the overload control object are controlled based on the control resource gear; wherein the token bucket resources occupied by the overload control object are controlled based on the token quantity corresponding to the control resource gear, and / or the processing resources occupied by the overload control object are controlled based on the concurrency quantity corresponding to the control resource gear; wherein the smaller the control resource gear is, the smaller the token quantity and concurrency quantity corresponding to the control resource gear are.

[0169] In one example, when the determination module 72 determines the overload control object based on the load level, it is specifically used for: if the load level matches the control level of the first level interval, and the background tasks of the priority corresponding to the control level are multiple, selecting a target resource identification point from all resource identification points based on the load data of each resource identification point, and selecting a target background task from the multiple background tasks based on the target resource identification point; wherein the background task sensitive to the target resource identification point is the target background task; the background task sensitive to the target resource identification point refers to that when the resources occupied by the background task are adjusted, the load data of the target resource identification point changes greatly, and when the resources occupied by other background tasks except the background task are adjusted, the load data of the target resource identification point changes little.

[0170] The target background task is determined as the overload control object.

[0171] In one example, the control module 73 is specifically configured to control the resource occupied by the overload control object as follows: if the overload control object is a foreground service object of the resource object, and a plurality of foreground service objects correspond to a plurality of priority queues, the control module 73 determines a number of tokens corresponding to the overload control object, controls the resource object to read data from the plurality of priority queues based on the number of tokens, and processes the read data; wherein the plurality of priority queues share one token bucket; if the overload control object is a token bucket object for anti-pressure host IO, and the token bucket object corresponds to a plurality of priority queues, the control module 73 determines a number of tokens corresponding to the overload control object, controls the resource object to read data from the plurality of priority queues based on the number of tokens, and processes the read data.

[0172] In one example, the control module 73 is specifically configured to control the resource occupied by the overload control object as follows: if the overload control object is a foreground service object of the resource object, and a plurality of foreground service objects correspond to a plurality of priority queues, the control module 73 determines a number of tokens corresponding to the overload control object, controls the resource object to read data from the plurality of priority queues based on the number of tokens, and processes the read data; wherein the plurality of priority queues share one token bucket; if the overload control object is a token bucket object for anti-pressure host IO, and the token bucket object corresponds to a plurality of priority queues, the control module 73 determines a number of tokens corresponding to the overload control object, controls the resource object to read data from the plurality of priority queues based on the number of tokens, and processes the read data.

[0173] In one example, the control module 73 is specifically configured to control the resource occupied by the overload control object as follows: if the overload control object is a foreground service object of the resource object, and a plurality of foreground service objects correspond to a plurality of priority queues, the control module 73 determines a number of tokens corresponding to the overload control object, controls the resource object to read data from the plurality of priority queues based on the number of tokens, and processes the read data; wherein the plurality of priority queues share one token bucket; if the overload control object is a token bucket object for anti-pressure host IO, and the token bucket object corresponds to a plurality of priority queues, the control module 73 determines a number of tokens corresponding to the overload control object, controls the resource object to read data from the plurality of priority queues based on the number of tokens, and processes the read data.

[0174] Based on the same application concept as the above method, an electronic device is provided in the embodiments of the present application, as shown in FIG. 8, the electronic device comprises a processor 81 and a machine readable storage medium 82, the machine readable storage medium 82 stores machine executable instructions capable of being executed by the processor 81; the processor 81 is configured to execute the machine executable instructions to implement the resource management method disclosed in the above examples of the present application.

[0175] Based on the same application concept as the above method, the embodiments of the present application further provide a machine readable storage medium, the machine readable storage medium stores a plurality of computer instructions, and the computer instructions are executed by a processor to implement the resource management method disclosed in the above examples of the present application.

[0176] The machine readable storage medium can be any electronic, magnetic, optical, or other physical storage device, and can contain or store information such as executable instructions, data, and the like. For example, the machine readable storage medium can be a RAM (Random Access Memory), a volatile memory, a non-volatile memory, a flash memory, a storage drive (such as a hard disk drive), a solid state disk, any type of storage disk (such as an optical disk, a DVD, etc.), or similar storage medium, or a combination thereof.

[0177] Based on the same application concept as the above method, the embodiments of the present application further provide a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the resource management method disclosed in the above examples of the present application.

[0178] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can adopt a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the embodiments of the present application can adopt the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program codes.

[0179] The above only describes the embodiments of the present application and is not intended to limit the present application. Various modifications and changes can be made to the present application by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the scope of the claims of the present application.

Claims

1. A resource management method characterized by, The method is applied to a distributed storage system including a plurality of resource objects, and comprises: For each resource object, the resource object includes at least one resource identification point, load data of each resource identification point is obtained, and the load data represents resource usage of the resource identification point; A load level of the resource object is determined based on the load data of each resource identification point; If it is determined that the resource object is overloaded based on the load level, an overload control object is determined based on the load level, the overload control object includes at least one of a background task of the resource object, a foreground service object of the resource object, and a token bucket object for backpressure host IO; wherein the background task and the foreground service object share a local token bucket of the resource object; Resources occupied by the overload control object are controlled.

2. The method of claim 1, wherein: If the resource object is a storage node, the resource object includes at least one of the following resource identification points: a CPU identification point, a memory identification point, a metadata database identification point, a write cache identification point, a read cache identification point, a token bucket identification point, a storage pool identification point, and a cache background task identification point; If the resource object is a storage pool, the resource object includes at least one of the following resource identification points: a memory identification point, a write cache identification point, a read cache identification point, a token bucket identification point, and a cache background task identification point; The load data of the CPU identification point includes CPU usage of the storage node; The load data of the memory identification point includes remaining memory of the storage node; The load data of the metadata database identification point includes CPU usage of the metadata database; The load data of the write cache identification point includes water level and latency of the write cache; The load data of the read cache identification point includes read cache latency; The load data of the token bucket identification point includes IO operation token latency; The load data of the storage pool identification point includes foreground IO statistical latency; The load data of the cache background task identification point includes background task read latency.

3. The method of claim 2, wherein, The method further comprises one or more of the following: A target ratio between data length of an IO operation and a configured standard IO length is determined, and IO operation token latency is determined based on the target ratio and standard latency corresponding to the standard IO length; wherein the standard latency represents processing time of data of the standard IO length; If the storage node includes a plurality of CPU cores, usage of a CPU core bound to the IO operation is determined, and CPU usage of the storage node is determined based on the usage; If the storage node includes a plurality of CPU cores, usage of a CPU core bound to the metadata database is determined, and CPU usage of the metadata database is determined based on the usage.

4. The method of claim 1, wherein, The first level interval includes K control levels, the resource object includes K priority background tasks, and the K priorities correspond to the K control levels one by one; if the load level matches the control level of the first level interval, the overload control object includes the background task of the priority corresponding to the control level. Or, the second level interval includes P control levels, the resource object includes P priority foreground service objects, and the P priorities correspond to the P control levels one by one; if the load level matches the control level of the second level interval, the overload control object includes the foreground service object of the priority corresponding to the control level and the background task of all priorities. Or, if the load level matches the third level interval, the overload control object includes the token bucket object, the foreground service object of all priorities, and the background task of all priorities. The control level of the first level interval is less than the control level of the second level interval, and the control level of the second level interval is less than the control level of the third level interval.

5. The method of claim 1 or 4, wherein the control of the resource occupied by the overload control object comprises: reducing or increasing a control resource gear of the overload control object to obtain a control resource gear; controlling the resource occupied by the overload control object based on the control resource gear; wherein the token bucket resource occupied by the overload control object is controlled based on the number of tokens corresponding to the control resource gear, and / or the processing resource occupied by the overload control object is controlled based on the number of concurrent numbers corresponding to the control resource gear; wherein the smaller the control resource gear is, the smaller the number of tokens and the number of concurrent numbers corresponding to the control resource gear are.

6. The method of claim 4, wherein the determination of the overload control object based on the load level comprises: if the load level matches the control level of the first level interval, and the background task of the priority corresponding to the control level is multiple, selecting a target resource identification point from all resource identification points based on the load data of each resource identification point, and selecting a target background task from the multiple background tasks based on the target resource identification point; wherein the background task sensitive to the target resource identification point is the target background task; the background task sensitive to the target resource identification point refers to that when the resource occupied by the background task is adjusted, the load data of the target resource identification point changes greatly, and when the resource occupied by other background tasks except the background task is adjusted, the load data of the target resource identification point changes little; determining the target background task as the overload control object.

7. The method of claim 1, wherein the control of the resource occupied by the overload control object comprises: ​ ​ ​ If the overload control object is a foreground service object of the resource object, and multiple different foreground service objects correspond to multiple priority queues, a token quantity corresponding to the overload control object is determined, and the resource object is controlled to read data from the multiple priority queues based on the token quantity, and the read data is processed; wherein the multiple priority queues share one token bucket; If the overload control object is a token bucket object for anti-pressure host IO, and the token bucket object corresponds to multiple priority queues, a token quantity corresponding to the overload control object is determined, and the resource object is controlled to read data from the multiple priority queues based on the token quantity, and the read data is processed; Wherein, when reading data from the multiple priority queues based on the token quantity, the order value of each data in the multiple priority queues is determined, and the data with the smallest order value is read; For each data, the order value of the data is determined based on the enqueue time of the data and the adjustment value of the priority queue where the data is located; wherein, the earlier the enqueue time of the data is, the smaller the order value of the data is, the greater the priority of the priority queue is, the smaller the adjustment value is, and the smaller the order value of the data is.

8. The method of claim 1, wherein, After the resources occupied by the overload control object are controlled, the method further comprises: If it is determined that the resource object is lightly loaded based on the load level, the load of the overload control object is restored; wherein, when the load of the overload control object is restored, the load of the token bucket object for anti-pressure host IO is restored first, if it is determined that the resource object is lightly loaded based on the load level, the load of the foreground service object of the resource object is continued to be restored, and if it is determined that the resource object is lightly loaded based on the load level, the load of the background task of the resource object is continued to be restored.

9. A resource management apparatus characterized by comprising: The device is applied to a distributed storage system, and the distributed storage system comprises multiple resource objects, and the device comprises: An acquisition module is configured to, for each resource object, acquire load data of each resource identification point of the resource object, wherein the resource object comprises at least one resource identification point, and the load data represents resource usage of the resource identification point; A determination module is configured to determine a load level of the resource object based on the load data of each resource identification point, and if it is determined that the resource object is overloaded based on the load level, determine an overload control object based on the load level, wherein the overload control object comprises at least one of a background task of the resource object, a foreground service object of the resource object, and a token bucket object for anti-pressure host IO, and the background task and the foreground service object share a local token bucket of the resource object; A control module is configured to control resources occupied by the overload control object.

10. The device of claim 9, wherein, ​ If the resource object is a storage node, the resource object comprises at least one of the following resource identification points: a CPU identification point, a memory identification point, a metadata database identification point, a write cache identification point, a read cache identification point, a token bucket identification point, a storage pool identification point, and a cache background task identification point; if the resource object is a storage pool, the resource object comprises at least one of the following resource identification points: a memory identification point, a write cache identification point, a read cache identification point, a token bucket identification point, and a cache background task identification point; The load data of the CPU identification point comprises a CPU usage rate of the storage node. The load data of the memory identification point comprises a remaining memory of the storage node. The load data of the metadata database identification point comprises a CPU usage rate of the metadata database. The load data of the write cache identification point comprises a water level and a time delay of the write cache. The load data of the read cache identification point comprises a read cache time delay. The load data of the token bucket identification point comprises an IO operation token waiting time delay. The load data of the storage pool identification point comprises a foreground IO statistical time delay. The load data of the cache background task identification point comprises a background task read time delay.

11. The apparatus of claim 9, wherein, when the determination module determines the overload control object based on the load level, a first level interval comprises K control levels, the resource object comprises K background tasks of priorities, and the K priorities correspond to the K control levels one by one; if the load level matches a control level of the first level interval, the overload control object comprises the background tasks of the priorities corresponding to the control level; or, a second level interval comprises P control levels, the resource object comprises P foreground service objects of priorities, and the P priorities correspond to the P control levels one by one; if the load level matches a control level of the second level interval, the overload control object comprises the foreground service objects of the priorities corresponding to the control level and the background tasks of all priorities; or, if the load level matches a third level interval, the overload control object comprises the token bucket object, the foreground service objects of all priorities, and the background tasks of all priorities. The control level of the first level interval is less than the control level of the second level interval, and the control level of the second level interval is less than the control level of the third level interval.

12. The apparatus of claim 9 or 11, wherein, when the control module controls the resources occupied by the overload control object, the control module is specifically configured to: lower or raise a current resource gear of the overload control object to obtain a control resource gear. control resources based on the control resource gear; wherein the token bucket resources occupied by the overload control object are controlled based on the token quantity corresponding to the control resource gear, and / or the processing resources occupied by the overload control object are controlled based on the concurrency quantity corresponding to the control resource gear; wherein the smaller the control resource gear is, the smaller the token quantity and the concurrency quantity corresponding to the control resource gear are.

13. The apparatus of claim 11, wherein, The determining module is specifically configured to: if the load level matches the control level of the first level interval, and the background task corresponding to the priority of the control level is multiple, select a target resource identification point from all resource identification points based on the load data of each resource identification point, and select a target background task from the multiple background tasks based on the target resource identification point; wherein the background task sensitive to the target resource identification point is the target background task; the background task sensitive to the target resource identification point refers to that when the resources occupied by the background task are adjusted, the load data of the target resource identification point changes greatly, and when the resources occupied by other background tasks except the background task are adjusted, the load data of the target resource identification point changes little. The target background task is determined as the overload control object.

14. The apparatus of claim 9, wherein, The control module is specifically configured to control the resources occupied by the overload control object: if the overload control object is a foreground service object of the resource object, and multiple foreground service objects correspond to multiple priority queues, determine the token quantity corresponding to the overload control object, control the resource object to read data from the multiple priority queues based on the token quantity, and process the read data; wherein the multiple priority queues share one token bucket; if the overload control object is a token bucket object for anti-pressure host IO, and the token bucket object corresponds to multiple priority queues, determine the token quantity corresponding to the overload control object, control the resource object to read data from the multiple priority queues based on the token quantity, and process the read data; wherein, when the control module reads data from the multiple priority queues based on the token quantity, the order value of each data in the multiple priority queues is determined, and the data with the smallest order value is read; for each data, the order value of the data is determined based on the enqueue time of the data and the adjustment value of the priority queue in which the data is located; wherein the earlier the enqueue time of the data is, the smaller the order value of the data is, the greater the priority of the priority queue is, the smaller the adjustment value is, and the smaller the order value of the data is.

15. An electronic device, comprising: comprising: a processor and a machine readable storage medium storing machine executable instructions executable by the processor; the processor is configured to execute the machine executable instructions to implement the method of any one of claims 1-8.

Images