Flow control method of distributed storage system, and distributed storage system

Abstract

A flow control method of a distributed storage system, and a distributed storage system. The distributed storage system comprises a plurality of storage nodes and a plurality of clients. The method comprises: a client acquires a processing index related to a coroutine task in a local IO thread; the client acquires a delay index of an IO request sent to a storage node; and the client adjusts a control threshold of flow between the client and the storage node on the basis of the acquired processing index and delay index. The processing index related to the coroutine task in the IO thread and the delay index of the IO request that are collected by the client are used to evaluate the busyness of a cluster in a distributed manner, and how to the flow control threshold of the client is autonomously decided. According to the self-adaptive method, a response can be made to request accumulation in a more timely manner, thereby improving the overall resource utilization rate of the cluster.

Description

Flow control method of distributed storage system and distributed storage system

[0001] This disclosure claims priority to the Chinese patent application filed with the China Patent Office on September 1, 2023, with application number 202311128763.X and application name “Flow control method and distributed storage system of distributed storage system”, the entire content of which is incorporated by reference in this disclosure. Technical Field

[0002] The present disclosure relates to distributed technology, and in particular to a flow control method for a distributed storage system and a distributed storage system. Background Art

[0003] Distributed storage is a data storage technology that utilizes the disk space on every machine in an enterprise over a network, combining these dispersed storage resources into a virtual storage device, allowing data to be stored dispersedly throughout the enterprise. In practice, storage systems can be subject to intentional or unintentional attacks from upper-layer applications. Once the application traffic pressure exceeds the storage system's capacity, input and output (IO) requests from the application will continue to accumulate within the storage system, gradually reducing its service capabilities and even causing service outages.

[0004] Existing flow control methods often provide fixed flow control thresholds based on experience or testing, or require storage servers to perform flow control based on their own load. However, these methods cannot flexibly address the various problems faced by distributed storage systems in actual operations.

[0005] Therefore, there is a need for an improved flow control method for distributed storage systems.

[0006] Summary of the Invention

[0007] A technical problem to be solved by the present disclosure is to provide a flow control method and a distributed storage system for a distributed storage system. The method uses the processing indicators related to the coroutine tasks of the IO threads and the latency indicators of the IO requests collected by the client itself to distributely evaluate the busyness of the cluster and independently decide how to adjust the client's flow control threshold. Thus, under the premise of fully utilizing the storage cluster capacity, once request accumulation occurs, the cluster pressure can be quickly reduced by lowering the client's flow control threshold. This adaptive method can respond to request accumulation more promptly, thereby improving the overall resource utilization of the cluster.

[0008] According to a first aspect of the present disclosure, a flow control method for a distributed storage system is provided, wherein the distributed storage system includes multiple storage nodes and multiple clients, and the method includes: the client obtains a processing indicator related to a coroutine task in a local IO thread; the client obtains a latency indicator of an IO request sent to a storage node; and based on the obtained processing indicator and latency indicator, the client adjusts a flow control threshold with the storage node.

[0009] Optionally, the client obtains processing indicators related to coroutine tasks in the local IO thread, including: the client obtains information on the number of coroutine tasks to be processed in the local IO thread as the processing indicator; and / or the client obtains the proportion of coroutine task processing time in the local IO thread as the processing indicator.

[0010] Optionally, the client obtaining the latency indicator of the IO request sent to the storage node includes: obtaining a response status of the IO request sent to the storage node within a predetermined time to extract the IO request latency indicator.

[0011] Optionally, obtaining the response status of the IO requests sent to the storage node within the predetermined time to extract the latency index includes: obtaining the average latency and the proportion of timeout requests of multiple IO requests sent to the storage node within the predetermined time; and when the average latency exceeds the overload delay threshold and the proportion of timeout requests exceeds the overload proportion threshold, determining that the latency index indicates an overload.

[0012] Optionally, the client adjusts the flow control threshold with the storage node based on the acquired processing indicator and the delay indicator, including: when one of the processing indicator and the delay indicator exceeds the respective overload threshold, the client lowers the flow control threshold; and when the processing indicator and the delay indicator are both lower than the respective light load threshold, the client raises the flow control threshold.

[0013] Optionally, the flow control threshold is a control threshold of concurrency flow.

[0014] Optionally, the client may notify a corresponding storage node of the adjustment result, so as to adjust the flow values ​​of other clients connected to the storage node.

[0015] According to a second aspect of the present disclosure, a distributed storage system is provided, comprising a plurality of storage nodes and a plurality of clients, wherein the clients are configured to send IO requests to corresponding storage nodes and obtain IO responses; and the storage nodes are configured to send IO responses to corresponding clients based on the IO requests, wherein the clients include a load monitoring module configured to execute the method described in the first aspect.

[0016] According to a third aspect of the present disclosure, a computing device is provided, comprising: a processor; and a memory on which executable code is stored, and when the executable code is executed by the processor, the processor executes the method described in the first aspect above.

[0017] According to a fourth aspect of the present disclosure, a computer program product is provided, comprising an executable code. When the executable code is executed by a processor of an electronic device, the processor is caused to execute the method as described in the first aspect above.

[0018] According to a fifth aspect of the present disclosure, a non-transitory machine-readable storage medium is provided, on which executable code is stored. When the executable code is executed by a processor of an electronic device, the processor executes the method described in the first aspect above.

[0019] Therefore, the flow control method disclosed in the present invention periodically evaluates the storage nodes and then evaluates the busyness of the cluster based on the statistical processing indicators related to the coroutine tasks in the client IO thread and the statistical latency information from the IO request sent to the back-end storage node to the response of the request processing completed returned by the storage node. These two indicators cover the clients, networks, and storage nodes on the IO path, and can more reasonably judge whether the service capabilities of the storage cluster can withstand the current traffic pressure. Each client can evaluate the busyness of the cluster in a distributed manner and adaptively and dynamically adjust the client's concurrency flow control threshold. In this way, the resource utilization of the cluster can be improved while reducing the manpower cost of operation and maintenance. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The above and other objects, features and advantages of the present disclosure will become more apparent through a more detailed description of exemplary embodiments of the present disclosure with reference to the accompanying drawings, wherein like reference numerals generally represent like components in the exemplary embodiments of the present disclosure.

[0021] FIG1 shows a schematic flow chart of a flow control method for a distributed storage system according to an embodiment of the present disclosure.

[0022] FIG2 shows an example of a distributed storage system implementing the flow control method disclosed herein.

[0023] Figure 3 shows an example of the relationship between processes, threads, and coroutines.

[0024] FIG4 shows a schematic diagram of dynamic flow control performed by a client according to an embodiment of the present disclosure.

[0025] FIG5 shows a schematic structural diagram of a computing device that can be used to implement the flow control method of the distributed storage system according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0026] The preferred embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although preferred embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. Rather, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0027] Distributed storage is a data storage technology that uses the disk space on each machine in a storage system over a network, organizing these distributed storage resources into a virtual storage device, with data stored in multiple locations. Distributed storage systems employ a distributed architecture, utilizing multiple storage nodes to share the storage load and using location nodes to locate stored information. This improves the reliability, availability, and access efficiency of the storage system, while also facilitating scalability. In practice, storage systems may be subject to intentional or unintentional attacks from upper-layer applications. Once the application traffic pressure exceeds the storage system's capacity, I / O requests from the applications will continue to accumulate within the storage system, gradually reducing its service capabilities and even causing service outages.

[0028] Existing flow control methods often rely on storage nodes to control flow based on their own load or set fixed flow control thresholds based on experience or testing. However, these methods are unable to flexibly address the various challenges faced by distributed storage systems in actual operations.

[0029] Existing flow control methods on the storage server side prevent storage clusters from overloading and causing avalanches by refusing to accept excessive traffic on the storage node side. The core approach of this solution is to detect some indicators on the storage node that can represent the storage node load (such as CPU load, memory utilization, or the length of the request processing queue). When these monitoring indicators exceed the load threshold of the "safe" level set in advance based on experience or testing, the storage node will refuse to process part of the received traffic according to a certain strategy. As the storage node load returns to below the "safe" level, the rejected traffic will be reduced accordingly until it reaches 0.

[0030] Network congestion, client overload, and other conditions can also reduce the storage cluster's service capacity, leading to a crash. Therefore, the solution described above, which relies solely on monitoring the load on storage nodes to determine whether a storage cluster is overloaded, cannot diagnose all overload situations. Furthermore, simply rejecting traffic on the storage node side can cause clients to resend rejected requests because they haven't received a response confirming that these requests have been processed. This can lead to a doubling of traffic sent to the storage node.

[0031] Common fixed flow control thresholds based on experience or testing are often set based on test data or expert experience. This approach often suffers from the difficulty of optimizing flow control threshold configurations. Even before a cluster goes live, optimized concurrency flow control threshold configurations obtained through testing can cause fluctuations in the storage cluster's service capabilities after launch due to machine expansion and contraction, increases and decreases in the number of upper-layer applications, and underlying hardware wear and tear. This can render the previously configured concurrency flow control thresholds inoperable.

[0032] There are also solutions that rely on manual operations and maintenance for flow control. When cluster pressure increases, requests accumulate, and service performance is impacted, operations and maintenance personnel manually lower the concurrency flow control threshold. When cluster pressure decreases and cluster utilization is low, the concurrency flow control threshold is manually increased based on experience. However, manually adjusting the concurrency flow control threshold based on expert experience not only increases labor and maintenance costs but also slows down the overall system response.

[0033] In view of the many problems with existing flow control methods, the present disclosure proposes a solution in a distributed storage system in which the client adaptively adjusts its own flow control threshold according to the busyness of the cluster. In this solution, the indicators for considering whether the cluster is busy come from two aspects. One is the processing indicator related to the coroutine task in the client IO thread obtained by the client. This processing indicator is based on the current processing parameters of the coroutine task and is used to characterize whether the client's CPU thread is busy and whether the IO pressure received by the client exceeds the client's service capability. The second is the latency indicator of the IO request obtained by the client, such as the RPC latency information (i.e., the latency information from sending to the back-end storage node to receiving the response of the request processing completed returned by the storage node). This latency indicator is used to characterize whether the storage node is busy and whether the IO pressure received by the storage node exceeds the service capability of the storage node; it can also reflect whether the network from the client to the storage node is congested and whether the IO pressure received by the network exceeds the service capability of the network.

[0034] Based on these two statistical data points, each client can assess the cluster's workload in a distributed manner and independently decide how to adjust its concurrency flow control threshold. This approach allows full utilization of the storage cluster's capabilities. If the service capacity of a particular area in the storage system decreases and requests accumulate, cluster pressure can be quickly reduced by lowering the client flow control threshold. Once the storage system's service capacity recovers, the client flow control threshold can be raised to fully utilize the storage cluster's capabilities.

[0035] Figure 1 shows a schematic flow chart of a flow control method for a distributed storage system according to an embodiment of the present disclosure. Meanwhile, for ease of explanation, Figure 2 shows an example of a distributed storage system implementing the flow control method of the present disclosure.

[0036] As shown in Figure 2, a distributed storage system 200 may include multiple storage nodes 210 and multiple clients 220. In the server-client model, the storage node belongs to the server side. For example, a storage server may correspond to a storage node and is used to store data. The client in a distributed file system corresponds to the server and is a program that provides file system services to the client (i.e., installed on the client's computer). In a distributed file system, the client does not directly access the underlying storage, but communicates with the server through a network using a specific communication protocol to complete file storage operations.

[0037] The figure shows that M clients are obtaining storage services from a cluster including N storage nodes through the network. In actual operation, each client can make an IO request to the cluster, and the location node in the storage system (i.e., the location node, not shown in the figure) indicates the storage node where the request is located, and establishes a connection between the client and the storage node. For example, the connection established between client 220_1 and storage node 210_1 shown in Figure 2. After the connection is established, client 220_1 can continue to send IO requests to storage node 210_1, and storage node 210_1 can respond to the received IO requests. It should be understood that the connection between a specific client and a specific storage node can be disconnected after the current IO task is completed, and different clients can be connected to the same storage node (as shown, client 220_2 and client 220_3 are both connected to storage node 210_1) to establish a connection and perform data transmission.

[0038] In step S110, the client obtains processing metrics related to the coroutine task in the local I / O thread. In step S120, the client obtains latency metrics for I / O requests sent to the storage node. In step S130, based on the obtained processing metrics and latency metrics, the client adjusts the flow control threshold (hereinafter referred to as the "flow control threshold") with the storage node.

[0039] In a distributed storage system, because machines are deployed in a distributed manner, flow control can be implemented at the client entrance to prevent the traffic entering from the client from exceeding the capacity of the backend storage cluster. Flow control, or flow control, is a technology that uses software or hardware to protect limited downstream resources from being overwhelmed by traffic and ensure service availability. If the client's flow control threshold is fixed, then if the flow control threshold is set too low, the storage system's performance cannot be fully utilized, resulting in wasted storage resource capacity; conversely, if the flow control threshold is set too high, it will not protect the backend storage cluster. Even if an optimized flow control threshold configuration is obtained through testing before the cluster goes online, after going online, due to machine expansion and contraction, the increase or decrease in the number of upper-layer applications, and the loss of underlying hardware, the service capacity of the storage cluster will fluctuate, and the flow control threshold previously configured by the client will no longer be adapted.

[0040] To this end, each client in the system preferably performs steps S110-S130 above, obtaining processing metrics to assess whether the client's own CPU thread is busy and whether the client's current IO pressure exceeds the client's service capacity. Simultaneously, the client obtains latency metrics to assess whether the storage node connected to the client is busy and whether the storage node's current IO pressure exceeds the storage node's service capacity. The latency metric can also reflect whether the network from the client to the storage node is congested and whether the network's current IO pressure exceeds the network's service capacity. Thus, the client can dynamically adjust the flow control threshold with the corresponding storage node based on these two metrics.

[0041] Here, the client, the corresponding storage node, and the network between them can be considered to constitute a "subsystem" or "IO operation unit" of the distributed storage system. For example, in Figure 2, the dotted lines outline client 220_1, storage node 210_1, and the network connection between them. At this time, if the processing and latency indicators obtained by client 220_1 are both within the normal range, then it is considered that the processing capacity of client 220_1, the IO pressure of storage node 210_1, and the network congestion level are all within the normal range, and there is no need to adjust the flow control threshold (assuming that the "flow control threshold" at this time is originally set to a normal value). However, if either the processing or latency indicators indicate an overload, it is considered that there is a flow that the current subsystem cannot normally handle (for example, a high processing indicator indicates that the bottleneck is the client processing capacity, and a high latency indicator indicates that the bottleneck is the storage node or the network). Therefore, it is necessary to adaptively lower the flow control threshold from the normal value, that is, to reduce the flow rate between client 220_1 and storage node 210_1. When both the processing indicator and the latency indicator indicate a light load, the flow control threshold can be adaptively adjusted higher from the normal value, that is, data can be exchanged between the client 220_1 and the storage node 210_1 at a higher flow rate.

[0042] To this end, the client adjusting the flow control threshold with the storage node based on the acquired processing indicator and the latency indicator may include: when one of the processing indicator and the latency indicator exceeds the respective overload threshold, the client lowers the flow control threshold; and when the processing indicator and the latency indicator are both lower than the respective light load threshold, the client raises the flow control threshold.

[0043] In the present disclosure, processing indicators related to coroutine tasks are creatively used, rather than simply using CPU usage or thread-related indicators to evaluate the busyness of the client. It is known that a thread is the smallest unit of program execution, and a process is the smallest unit of resource allocation by the operating system. A process can be composed of one or more threads, and threads are different execution routes of the code in a process. Processes are independent of each other, but the threads under the same process share the program's memory space (including code segments, data sets, heaps, etc.) and some process-level resources (such as open files and signals). Threads within a process are not visible to other processes, and thread context switching is much faster than process context switching. In short, threads and processes are both abstract concepts. Threads are a smaller abstraction than processes, and both threads and processes can be used to achieve concurrency.

[0044] In conventional systems, each request occupies a thread to complete the entire service logic (including transactions). This means that the system's throughput depends on the operation time of each thread. When encountering time-consuming I / O operations, the overall system throughput immediately decreases because the threads are constantly blocked. Furthermore, if there are many threads, many threads may be idle (waiting for the thread to complete before they can execute). This results in inefficient resource utilization and increased context switching overhead.

[0045] Coroutines are based on threads but are more lightweight than threads. These lightweight threads, managed by programmers themselves, are also called "user-mode threads" and are invisible to the kernel. Just as a process can have multiple threads, a thread can also have multiple coroutines. Figure 3 shows an example of the relationship between processes, threads, and coroutines. As shown in the figure, multiple processes are currently running in the operating system, including process A and process B. Process A and B each contain multiple threads. Thread 1 of process A includes multiple coroutines.

[0046] Coroutines are particularly well-suited for scenarios involving blocked I / O operations that require extensive concurrency. When I / O is blocked, the coroutine scheduler handles the dispatch by immediately releasing the data flow and recording the data currently on the stack. After the block is resolved, the thread immediately restores the stack and returns the blocked result to the thread. This approach makes it appear identical to synchronous code. Since coroutine suspension is completely controlled by the program and occurs in user mode, whereas thread blocking is switched by the operating system kernel and occurs in kernel mode, coroutines have significantly less overhead than threads and eliminate the context switching overhead. Furthermore, coroutines utilize minimal resources and do not introduce significant performance issues. Furthermore, since there is only one thread, there are no conflicts over simultaneous variable writes. Controlling shared resources in coroutines requires no locking, only state verification, resulting in higher overall execution efficiency.

[0047] In one embodiment of the present disclosure, the client of a distributed storage system uses an R2C (Run-to-Completion) threading model. Read and write requests for the same file are always executed on the same client thread based on the file name hash, thereby resolving resource contention issues between multiple client threads. Upon receiving a request, the client's IO thread schedules its execution as a coroutine.

[0048] Figure 4 shows a schematic diagram of dynamic flow control performed by a client according to an embodiment of the present disclosure. As shown in the figure, the client may include multiple IO threads, i.e., threads used to perform IO interaction with storage nodes for reading and writing. Each IO thread may include multiple coroutines. Each thread includes a ResourceGuard ("resource guard", for example, implemented as an IO throttle (IO throttling algorithm)), which is an execution unit for IO flow control, and may also be referred to as a flow limiter. The load monitoring module shown in the figure can be deployed on the client background thread. The load monitoring module can collect processing indicators and latency indicators periodically (for example, 1s).

[0049] In one embodiment, the processing indicator related to the coroutine task disclosed in the present invention can be the number of coroutine tasks waiting to be scheduled for execution on each IO thread of the client, that is, the number of coroutine tasks to be processed. The number of coroutine tasks to be processed can reflect the busyness of the client. On the one hand, when the IO thread executes a relatively time-consuming coroutine task, the number of coroutine tasks that the IO thread can process per unit time will decrease, resulting in a gradual increase in the number of coroutine tasks waiting to be scheduled for execution. On the other hand, when the service pressure increases, it will also cause the number of coroutine tasks waiting to be scheduled for execution in the IO thread to gradually increase.

[0050] Alternatively or in addition, the percentage of time spent processing coroutine tasks in the client's IO thread can be used as a processing metric. This is because if the processing time of each coroutine task is short, the total processing latency required by the thread for a large number of coroutines will not be too long. In other words, a high number of coroutine tasks may not indicate that the client is busy. Therefore, the percentage of time spent processing coroutine tasks in the client's IO thread can be used as a processing metric to more accurately measure whether the IO thread is busy in some cases.

[0051] Since the bottleneck of subsystem traffic in the client is the processing of coroutines in the client IO thread, using processing indicators related to coroutine tasks to measure whether the client is busy is better than using indicators of thread execution or CPU usage in existing technologies.

[0052] In addition to using processing indicators related to coroutine tasks to characterize the busyness of the client itself, the flow rate regulation method disclosed in the present invention also utilizes another indicator that the client itself can obtain, namely the latency indicator of the IO request. In one embodiment, the client obtains the latency indicator of the IO request sent to the storage node, including: obtaining the response status of the IO request sent to the storage node within a predetermined time to extract the latency indicator. The client usually sends multiple IO requests to the storage node within a predetermined time, such as 1s. The response status of these IO requests can be counted, and the latency indicator indicating light load, balance or overload can be extracted.

[0053] In a distributed storage system where clients communicate with storage nodes, clients set a timeout before sending each request, such as an RPC (remote procedure call) timeout. This RPC timeout indicates the time within which the client expects to receive a response from the storage node. RPC timeouts are typically set based on network conditions and the storage node's media type. If a client does not receive a response after waiting for the RPC timeout, it will resend the request.

[0054] In one embodiment, the latency indicator of the IO request obtained by the client can be the RPC latency information of the IO request counted by the client (i.e., the latency from sending the request to the back-end storage node to receiving the response from the storage node indicating that the request processing is completed). The RPC latency information can reflect the busyness of the storage node and the congestion level of the network from the client to the storage node. If the traffic pressure exceeds the service capacity of the storage node / network, the requests will accumulate on the storage node / network link, and the time it takes for the client to receive the request response will be longer. In severe cases, the client will count a large proportion of request sending timeouts.

[0055] In a preferred embodiment, obtaining the response status of IO requests sent to the storage node within a predetermined time period to extract the latency indicator may include obtaining an average latency and a percentage of timed-out requests for multiple IO requests sent to the storage node within the predetermined time period; and determining that the latency indicator indicates an overload when the average latency exceeds an overload latency threshold and the percentage of timed-out requests exceeds an overload percentage threshold. In other words, whether the latency indicator is overloaded can be determined based on both the average latency and the percentage of timed-out requests.

[0056] It should be understood that whether it's the number of coroutine tasks, the processing time percentage, or RPC latency information, as shown in Figure 4, these are all load statistics obtained by the background load monitoring module through interaction with the I / O thread. After obtaining this statistical information, the load status can be evaluated and the flow control threshold adjusted accordingly.

[0057] There are usually two control methods for flow control. One is to control the flow rate (including iops / throughput, where iops refers to the number of I / O requests that the system can process per unit time), and the other is to control the concurrency of the flow (including the number of inflight requests / the amount of inflight request data, where inflight refers to the amount of requests currently being processed). In one embodiment of the present disclosure, flow control is preferably performed based on concurrency. To this end, based on the obtained processing index and the latency index, the flow control threshold that the client adjusts with the storage node can be a concurrency flow control threshold.

[0058] Concurrency flow control is preferred because, compared to rate flow control, when the underlying storage system experiences a resource bottleneck, resulting in a decrease in service capacity, upper-layer applications will continue to issue requests at a certain rate of iops / throughput, further increasing the burden on underlying resources and causing a gradual accumulation of requests. However, concurrency control controls the number and data volume of inflight requests, preventing requests from accumulating even when the underlying processing rate slows.

[0059] To further enhance understanding, the following describes an example of client-side metric collection and adaptive flow control, using Figure 4. As shown, the load monitoring module is deployed on the client's background thread and periodically (e.g., every 1 second) collects the number of pending coroutine tasks waiting to be scheduled on each client's I / O thread as a processing metric. It also collects RPC latency information (including average RPC latency and the percentage of timed-out requests) for requests sent from the client to the storage node as a latency metric.

[0060] Based on the above statistical information, the load monitoring module evaluates the load status of the storage cluster. Each statistical information can be divided into multiple intervals based on the value range, and each interval corresponds to a different load status of the storage cluster: overloaded, balanced, and lightly loaded. For example, if the number of coroutine tasks exceeds 1000, the client is considered to be overloaded; if the number is between 500 and 1000, the client is considered to be in a balanced state; and if the number is below 500, the task client is in a lightly loaded state. For another example, if the RPC latency of requests sent by the client to the storage node reaches more than 90% of the set RPC timeout period, and more than 20% of the requests sent have timed out, the network link / storage node is considered to be overloaded; if the RPC latency of requests sent to the storage node is less than 50% of the RPC timeout period, the network link / storage node is considered to be lightly loaded; in other cases, the network link / storage node is considered to be in a balanced state.

[0061] After evaluating the load status of the client, network link, and storage node based on each statistical information, the client will comprehensively evaluate the load status of the cluster:

[0062] 1) As long as one subsystem is in overload state, the storage is considered to be in overload state.

[0063] 2) If all subsystems are in a balanced state, the storage cluster is considered to be in a balanced state.

[0064] 3) In other cases (no subsystem is in an overloaded state and at least one subsystem is in a lightly loaded state), the storage cluster is considered to be in a lightly loaded state.

[0065] In some embodiments, the evaluation and adjustment results of the client for each subsystem can be summarized by the storage node or even the storage cluster. In one embodiment, the flow control method of the present disclosure may also include the client notifying the corresponding storage node of the adjustment results; and based on the adjustment results, the flow values ​​of other clients connected to the storage node can be adjusted. The above adjustment can be performed by the storage node side, preferably by the client side. Taking Figure 2 as an example, if the client 220_2 determines that there is an overload based on its own coroutine indicators and the latency indicators obtained based on the response returned by the storage node 210_2, and lowers the concurrent flow, the client 220_2 can inform the storage node 210_2 of the above indicators and adjustment results. The storage node 210_2 can then inform the client 220_3, which is also connected to it, of the above adjustment, and the latter can adjust the flow accordingly. In one embodiment, if client 220_2 reduces its concurrent traffic only due to excessive processing metrics related to its own coroutine tasks, then since the bottleneck lies with client 220_2, client 220_3 can appropriately increase the traffic limit with storage node 210_2, or not adjust the traffic limit at all. If client 220_2 is overloaded only due to latency metrics, then since the bottleneck lies with the network or storage node, client 220_3 can appropriately decrease the traffic limit with storage node 210_2, or not adjust the traffic limit at all, based on its own situation.

[0066] Furthermore, for client-based upper-layer storage services, the real-time flow control thresholds determined by each client's dynamic adjustments can be aggregated. The aggregated result can be considered the cluster's current traffic capacity, such as concurrency. Based on this aggregated value, the upper-layer storage service can proactively reduce its own concurrency pressure to prevent excessive traffic pressure from being transmitted to the cluster, or appropriately increase it to prevent underutilization of cluster capacity.

[0067] In one embodiment, the present disclosure can also be implemented as a distributed storage system as shown in Figure 2. The system includes multiple storage nodes and multiple clients. The clients are configured to send I / O requests to corresponding storage nodes and obtain I / O responses; the storage nodes are configured to send I / O responses to corresponding clients based on the I / O requests. The clients include a load monitoring module configured to execute the method described above for adaptively adjusting traffic thresholds based on processing metrics and latency metrics.

[0068] FIG5 shows a schematic diagram of a computing device that can be used to implement the above-mentioned flow control method for a distributed storage system according to an embodiment of the present disclosure. Referring to FIG5 , a computing device 500 includes a memory 510 and a processor 520 .

[0069] Processor 520 may be a multi-core processor or may include multiple processors. In some embodiments, processor 520 may include a general-purpose main processor and one or more specialized coprocessors, such as a graphics processing unit (GPU) or a digital signal processor (DSP). In some embodiments, processor 520 may be implemented using customized circuits, such as application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs).

[0070] The memory 510 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. ROM may store static data or instructions required by the processor 520 or other modules of the computer. The permanent storage device may be a readable and writable storage device. The permanent storage device may be a non-volatile storage device that does not lose stored instructions and data even when the computer is powered off. In some embodiments, the permanent storage device uses a large-capacity storage device (such as a magnetic or optical disk, flash memory) as the permanent storage device. In other embodiments, the permanent storage device may be a removable storage device (such as a floppy disk, optical drive). The system memory may be a readable and writable storage device or a volatile readable and writable storage device, such as dynamic random access memory. The system memory may store some or all instructions and data required by the processor during operation. In addition, the memory 510 may include any combination of computer-readable storage media, including various types of semiconductor memory chips (DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and magnetic disks and / or optical disks may also be used. In some embodiments, the memory 510 may include a readable and / or writable removable storage device, such as a compact disc (CD), a read-only digital versatile disc (e.g., DVD-ROM, dual-layer DVD-ROM), a read-only Blu-ray disc, an ultra-density optical disc, a flash memory card (e.g., SD card, mini SD card, Micro-SD card, etc.), a magnetic floppy disk, etc. Computer-readable storage media do not include carrier waves and transient electronic signals transmitted wirelessly or wired.

[0071] The memory 510 stores executable codes. When the executable codes are processed by the processor 520 , the processor 520 can execute the flow control method of the distributed storage system mentioned above.

[0072] The flow control method of the distributed storage system and the corresponding distributed storage system according to the present disclosure have been described in detail above with reference to the accompanying drawings. The flow control method of the present disclosure is performed by the client based on the statistical processing indicators related to the coroutine task in the client IO thread, and the statistical latency indicators from the IO request sent to the back-end storage node to the response of the request processing completed returned by the storage node, to periodically evaluate the storage node, and then evaluate the busyness of the cluster. These two indicators cover the client, network, and storage nodes on the IO path, and can more reasonably judge whether the service capacity of the storage cluster can withstand the current traffic pressure. Each client can evaluate the busyness of the cluster in a distributed manner and adaptively and dynamically adjust the client's concurrency flow control threshold. In this way, the resource utilization of the cluster can be improved while reducing the manpower cost of operation and maintenance.

[0073] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0074] In addition, the method according to the present disclosure may also be implemented as a computer program or a computer program product, which includes computer program code instructions for executing the above steps defined in the above method of the present disclosure.

[0075] Alternatively, the present disclosure may also be implemented as a non-transitory machine-readable storage medium (or computer-readable storage medium, or machine-readable storage medium) on which executable code (or computer program, or computer instruction code) is stored. When the executable code (or computer program, or computer instruction code) is executed by a processor of an electronic device (or computing device, server, etc.), the processor is caused to perform the various steps of the above-mentioned method according to the present disclosure.

[0076] Those skilled in the art will further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or combinations of both.

[0077] The flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems and methods according to multiple embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment or part of code, and the part of the module, program segment or code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0078] While various embodiments of the present disclosure have been described above, the above descriptions are illustrative, non-exhaustive, and not intended to be limiting of the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or improvements to existing technologies, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A flow control method for a distributed storage system, wherein the distributed storage system comprises a plurality of storage nodes and a plurality of clients, the method comprising: The client obtains the processing indicators related to the coroutine task in the local input and output IO thread; The client obtains a latency indicator of an IO request sent to a storage node; as well as According to the acquired processing index and the latency index, the client adjusts a flow control threshold with the storage node.

2. The method of claim 1, wherein: The client obtains the processing indicators related to the coroutine task in the local IO thread including: The client obtains information about the number of pending coroutine tasks in the local IO thread as the processing indicator; and / or The client obtains the proportion of coroutine task processing time in the local IO thread as the processing indicator.

3. The method according to claim 1 or 2, wherein: The latency index of the IO request sent by the client to the storage node includes: The response status of the IO request sent to the storage node within a predetermined time is obtained to extract the latency indicator.

4. The method of claim 3, wherein: The obtaining of the response status of the IO request sent to the storage node within a predetermined time to extract the latency indicator includes: Obtain the average latency and percentage of timed-out requests for multiple IO requests sent to the storage node within a predetermined time; and When the average delay exceeds an overload delay threshold and the timeout request ratio exceeds an overload ratio threshold, it is determined that the delay indicator indicates an overload.

5. The method according to any one of claims 1 to 4, wherein: The client adjusting the flow control threshold with the storage node according to the acquired processing index and the delay index comprises: When one of the processing indicator and the delay indicator exceeds a respective overload threshold, the client lowers the flow control threshold; and When the processing indicator and the delay indicator are both lower than their respective light load thresholds, the client increases the flow control threshold.

6. The method according to any one of claims 1 to 5, wherein: The flow control threshold is a control threshold of concurrent flow.

7. The method according to any one of claims 1 to 6, further comprising: The client notifies the corresponding storage node of the adjustment result, wherein the flow values ​​of other clients connected to the storage node are adjusted based on the adjustment result.

8. A distributed storage system, comprising a plurality of storage nodes and a plurality of clients, wherein: The client is used to: Send input and output IO requests to the corresponding storage node and obtain IO responses; The storage node is used for: Sending an IO response to the corresponding client based on the IO request, The client comprises a load monitoring module for executing the method as claimed in any one of claims 1 to 7.

9. A computing device comprising: processor; as well as A memory having executable codes stored thereon, which, when executed by the processor, causes the processor to execute the method according to any one of claims 1 to 7.

10. A computer program product, comprising an executable code, which, when executed by a processor of an electronic device, causes the processor to execute the method according to any one of claims 1 to 7.

11. A non-transitory machine-readable storage medium having executable codes stored thereon, which, when executed by a processor of an electronic device, causes the processor to execute the method according to any one of claims 1 to 7.

Images