Control method, control system, computing device, storage medium and program product

Abstract

The embodiment of the invention provides a control method, a control system, computing equipment, a storage medium and a program product. The method comprises the steps of obtaining a load state of a server in a storage system; wherein the load state is determined based on index data of at least one operation index; updating a flow control threshold according to the load state; and performing flow limiting control on a processing request to be sent to the server according to the flow control threshold. According to the technical scheme provided by the embodiment of the invention, the storage performance is ensured, and the resource utilization rate is improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a control method, control system, computing device, storage medium, and program product. Background Technology

[0002] A current storage system employs a client-server architecture. The server can be implemented as a metadata storage node or multiple data storage nodes to achieve distributed storage. The client interacts with applications using the storage service. Each application can have a corresponding client deployed to receive access requests from the application and interact with the metadata and data storage nodes to ultimately perform data processing operations.

[0003] In practical applications, storage systems may be subject to intentional or unintentional traffic attacks from applications, which can lead to a decline in the service capabilities of the storage system and thus affect storage performance. Summary of the Invention

[0004] This application provides a control method, a control system, a computing device, a storage medium, and a program product to solve the technical problems of reduced service capabilities and impacted storage performance in the prior art.

[0005] In a first aspect, this application provides a control method applied to a client in a storage system, the method comprising:

[0006] Obtain the load status of the server in the storage system; wherein the load status is determined based on the metric data of at least one operating metric;

[0007] Update the flow control threshold based on the load status;

[0008] According to the flow control threshold, the processing requests to be sent to the server are subject to flow limiting control.

[0009] Secondly, this application provides a control method, including:

[0010] Determine the server's load status;

[0011] The load status is provided to the client so that the client can update the flow control threshold according to the load status of the server. The flow control threshold is used to perform flow limiting control on the requests to be sent to the server.

[0012] Thirdly, this application provides a control method, including:

[0013] Detect the indicator data of at least one operational indicator in the data storage node;

[0014] Based on the indicator data of the at least one operating indicator, determine the load status of the data storage node;

[0015] The load status of the data storage nodes is provided to the load assessment node, so that the load assessment node can combine the load status of multiple data storage nodes in the storage system to determine the load status of the data storage node set composed of the multiple data storage nodes; the load status of the data storage node set is used to trigger the update operation of the flow control threshold, and the flow control threshold is used to control the request sending operation for the data storage node.

[0016] Fourthly, this application provides a control system, a client, a metadata storage node, a data storage node, and a load assessment node.

[0017] The client is used to execute the control method as described in the first aspect above; the data storage node is used to execute the control method as described in the third aspect above; the metadata storage node is used to execute the control method as described in the second aspect above, or the load assessment node is used to execute the control method as described in the second aspect above.

[0018] Fifthly, this application provides a computing device, including a processing component and a storage component;

[0019] The storage component stores one or more computer instructions; the one or more computer instructions are to be invoked and executed by the processing component to implement the control method as described in the first aspect above, or the control method as described in the second aspect above, or the control method as described in the third aspect above.

[0020] Sixthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a computer processing component, implements the control method as described in the first aspect, the second aspect, or the third aspect above.

[0021] In a seventh aspect, this application provides a computer program product, including a computer program / instruction, which, when executed by a processing component, implements the control method as described in the first aspect, the second aspect, or the third aspect above.

[0022] In this embodiment, traffic control is performed on the client side. By combining the data of at least one operating metric, the load status of the server in the storage system is determined. The flow control threshold can be updated according to the load status. That is, the processing requests to be sent to the server can be rate-limited according to the flow control threshold. Rate limiting through the flow control threshold can constrain the number of requests sent to the server, thereby balancing the traffic pressure on the server and ensuring storage service capacity. Furthermore, dynamic updates based on the server load status ensure both storage performance and improved resource utilization.

[0023] These or other aspects of this application will become more apparent in the following description of the embodiments. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 The diagram illustrates a system architecture diagram to which the technical solution of this application is applicable;

[0026] Figure 2 This illustration shows another system architecture diagram to which the technical solution of the present application is applicable;

[0027] Figure 3 A flowchart of one embodiment of the control method provided in this application is shown;

[0028] Figure 4 This illustration shows a queue scheduling diagram in a practical application based on an embodiment of this application.

[0029] Figure 5 This illustration shows a request sending diagram in a practical application according to an embodiment of this application;

[0030] Figure 6 A flowchart of yet another embodiment of the control method provided in this application is shown;

[0031] Figure 7 A flowchart of yet another embodiment of the control method provided in this application is shown;

[0032] Figure 8 This illustration shows a scenario interaction diagram of an embodiment of this application in a practical application;

[0033] Figure 9This illustration shows another scenario interaction diagram of an embodiment of this application in a practical application;

[0034] Figure 10 A schematic diagram of one embodiment of a computing device provided in this application is shown. Detailed Implementation

[0035] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0036] In some of the processes described in the specification, claims, and accompanying drawings of this application, multiple operations appearing in a specific order are included. However, it should be clearly understood that these operations may not be executed in the order they appear herein, or may be executed in parallel. The operation numbers, such as 101, 102, etc., are merely used to distinguish different operations and do not themselves represent any execution order. Furthermore, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should be noted that the descriptions such as "first," "second," etc., in this document are used to distinguish different messages, devices, modules, etc., and do not represent a chronological order, nor do they limit "first" and "second" to different types.

[0037] The technical solutions of this application embodiment can be applied to storage systems. In one implementation scenario, they can be specifically applied to distributed storage systems, especially distributed storage systems for cloud computing scenarios.

[0038] like Figure 1 The diagram shown is a system architecture diagram of a storage system to which the technical solution of this application embodiment can be applied. The system may include a client 101, a metadata storage node (Master node) 102, and a data storage node 103.

[0039] Client 101 is used to interact with application 100 at the upper layer. In a distributed storage system, applications are often deployed on physical machines or in containers, and each application can have its own client. Therefore, the number of clients is usually related to the number of applications.

[0040] In a storage system, multiple metadata storage nodes and multiple data storage nodes can be deployed. Typically, there are more data storage nodes and relatively fewer metadata storage nodes.

[0041] In this embodiment, the metadata storage node 102 and the data storage node 103 constitute a storage cluster.

[0042] Data storage node 103 is a physical or virtual server that actually stores data. Data can be divided into data chunks and distributed across these data storage nodes. Data storage node 103 can be configured with a ChunkServer (storage engine) to achieve data management and performance monitoring.

[0043] Metadata storage node 102 is responsible for managing and maintaining metadata information for all data, such as storage space allocation, size, and storage location information. In the storage system, the number of clients and data storage nodes is relatively large, usually proportional to the number of physical machines, while the number of metadata storage nodes 102 is relatively small. A cluster composed of consensus algorithms such as Paxos (a distributed consensus protocol) or Raft (a distributed consensus protocol) can be used, so that even if some metadata storage nodes fail, the system can still continue to provide services.

[0044] In storage systems, data flow and control flow are typically separated. There is no data traffic between client 101 and metadata storage node 102, or between metadata storage node 102 and data storage node 103, to reduce the traffic load on metadata storage node 102. Client 101 interacts with metadata storage node 102 to perform corresponding metadata operations, such as creating or deleting files or directories, reading or modifying metadata, etc. When client 101 receives a data access request from an application, it first queries metadata storage node 102 to determine the data location information, and then accesses the corresponding data storage node 103 to perform data processing operations, such as data reading or data storage.

[0045] As described above, metadata storage nodes receive metadata operation requests from clients, while data storage nodes receive data access requests from clients. The traffic pressure on the server comes from these metadata operation or data access requests. Once the traffic exceeds the storage system's capacity, the storage system may experience a complete service outage. Even if the traffic pressure is reduced or stopped, the storage system may not be able to quickly restore service because it has already crashed.

[0046] In developing this application, the inventors conceived of using flow control (hereinafter referred to as "flow control") to prevent downstream storage resources from being overwhelmed by traffic, thus ensuring the availability of storage services. For example, protecting data storage nodes could involve setting flow control thresholds for accessing them, and protecting metadata storage nodes could involve setting flow control thresholds for accessing them. However, the inventors further discovered that, taking metadata storage nodes as an example, different types of metadata operations exert different levels of pressure on them. Therefore, it is difficult to quantify the system consumption of a single metadata operation on a metadata storage node. For instance, typically, retrieving metadata information about files and subdirectories within a directory consumes more resources than creating a file. Similarly, for the same metadata operation—retrieving information about files and subdirectories within a directory—retrieving metadata about a directory with multiple levels of subdirectories and hundreds of files obviously consumes more resources than retrieving information about an empty directory. Therefore, the flow control threshold setting will affect the operation of the metadata storage node. If the threshold is set too low, resources will not be fully utilized; if it is set too high, the metadata storage node will not be effectively protected. Similarly, for data storage nodes, setting the flow control threshold too low will prevent the storage system from fully utilizing its performance, resulting in wasted storage resources. Conversely, setting the flow control threshold too high will fail to protect the storage cluster. Furthermore, after the cluster goes live, factors such as machine scaling up and down, increases or decreases in the number of applications, and hardware wear and tear can all cause fluctuations in the storage cluster's service capacity, potentially rendering the flow control threshold inappropriate. Therefore, relying solely on expert experience is insufficient for setting accurate flow control thresholds.

[0047] Therefore, in order to ensure the service capacity and storage performance of the storage system, the inventors have proposed the technical solution of the embodiments of this application after a series of studies. By dynamically updating the flow control threshold according to the load status of the server, and performing flow limiting control on the processing requests to be sent to the server according to the flow control threshold, the traffic pressure on the server can be balanced, ensuring the storage service capacity. Moreover, the dynamic updating of the flow control threshold not only ensures storage performance but also improves resource utilization.

[0048] In practical applications, in distributed storage systems, the server can be implemented as a metadata storage node, or as multiple data storage nodes within the storage system. For ease of description, in this article, multiple data storage nodes in the storage system are described as a set of data storage nodes. In this case, the server's load status represents the overall load status of the data storage nodes in the storage system. When the server is a metadata storage node, its own load status can be detected by the metadata storage node. When the server is a set of data storage nodes, to detect the load status of the data storage nodes, such as... Figure 2 In the system architecture shown, with Figure 1The difference in the system architecture shown is that, in this embodiment of the application, a load assessment node 104 can be added to the storage system. The data storage node 103 can be responsible for detecting its own load status and providing it to the load assessment node 104. The load assessment node 104 summarizes the load status of different data storage nodes 103 in the data storage node set to obtain the load status of the data storage node set, etc.

[0049] Data storage node 103, metadata storage node 102, and load assessment node 104 can be deployed in computing devices, which can be physical devices or elastic computing hosts provided by cloud computing platforms, etc. This application does not limit them.

[0050] Client 101 and application 100 can be deployed in a computing device, which can be an electronic device. An electronic device can refer to a user-used device with the computing, internet access, and communication functions required by the user, such as a mobile phone, tablet computer, personal computer, or wearable device. An electronic device typically includes at least one processing component and at least one storage component. The electronic device may also include basic configurations such as a network interface card (NIC) chip, I / O bus, and audio / video components; this application does not limit its scope. Optionally, depending on the implementation of the electronic device, it may also include some peripheral devices, such as a keyboard, mouse, pen, and printer; this application does not limit its scope either.

[0051] It should be noted that the embodiments of this application may involve the use of user data. In practical applications, user-specific personal data may be used in the scheme described herein within the scope permitted by applicable laws and regulations, provided that it complies with the applicable laws and regulations of the country (e.g., with the user's explicit consent, with the user being properly notified, etc.).

[0052] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0053] It should be noted that the technical solutions of this application embodiment are applicable to the network virtual environment, and the described users generally refer to "virtual users". Real users can register user accounts on the server through registration to obtain user identity in the network environment.

[0054] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0055] Figure 3 A flowchart illustrating an embodiment of a data processing method provided in this application is shown. The technical solution of this embodiment can be executed by a client, and the method may include the following steps:

[0056] 301: Get the load status of the server in the storage system.

[0057] The server's load status can be determined based on at least one operational metric.

[0058] The server can refer to a metadata storage node in a storage system or a collection of data storage nodes consisting of multiple data storage nodes in a storage system.

[0059] 302: Update flow control thresholds based on load status.

[0060] 303: Rate limiting is applied to requests to be sent to the server based on the flow control threshold.

[0061] In the case where the server is a metadata storage node, the processing request refers to a metadata operation request; in the case where the server is a set of data storage nodes, the processing request refers to a data access request, such as a data read request or a data storage request. Of course, it can also be other types of data access requests, such as data verification requests or data deletion requests. This application does not limit the types of requests.

[0062] In this embodiment, by dynamically updating the flow control threshold according to the server's load status, and performing flow limiting control on the processing requests to be sent to the server according to the flow control threshold, the traffic pressure on the server can be balanced, ensuring storage service capabilities. Moreover, the dynamic updating of the flow control threshold not only ensures storage performance but also improves resource utilization.

[0063] As described above, in a practical application, the server can refer to the metadata storage node in the storage system. Obtaining the server's load status can include: receiving a response signal from the metadata storage node in response to any processing request; and determining the load status of the metadata storage node carried in the response signal.

[0064] The load status of the metadata storage node can be determined based on the metric data of at least one operating metric obtained from the detected metadata storage node.

[0065] The at least one performance metric may include software and / or hardware metrics that reflect whether the metadata storage node has reached a performance bottleneck. For example, the at least one performance metric may include one or more of the following: resource utilization, request processing queue length, request processing latency, and cache hit rate. Resource utilization may include, for example, storage device utilization, memory utilization, and processing unit utilization. Storage devices may refer to disks, and processing units may refer to CPUs (Central Processing Units). Disk utilization refers to the degree to which a disk is used at a specific point in time or within a given time period, usually expressed as a percentage of utilization. It reflects the proportion of time the disk spends processing requests. High disk utilization may indicate a heavy disk workload and continuous high activity, which may affect performance.

[0066] The request processing queue stores requests waiting to be processed. The length of the request processing queue can be used to assess processing performance. If the queue length is too long, it means that there is a bottleneck in processing performance, which leads to increased latency.

[0067] Request processing latency refers to the time interval between sending a request and receiving a response. A longer latency indicates a heavier workload.

[0068] Cache hit rate can refer to the proportion of successfully retrieved requests out of the total number of requests.

[0069] Memory utilization can refer to the ratio of currently used physical memory to total physical memory.

[0070] CPU utilization can be defined as the proportion of time a CPU is active. When a CPU is executing instructions, it is considered "active"; while when it is waiting for input / output (I / O) operations to complete or running in idle mode, it can be considered "idle".

[0071] Metric data from at least one operational metric can be used to measure the current load and resource usage of metadata storage nodes.

[0072] Since the number of clients in the storage system can be particularly large, and the metadata storage node sends a response signal to the client after processing each request, the load status determined by the metadata storage node can be included in the response signal and sent to the client to reduce the load on the metadata storage node.

[0073] By adding a target field to the response signal to write the load status, the load status can be included in the response signal for any processing request and sent to the client. Providing the load status to the client will not put processing pressure on the metadata storage node.

[0074] The metadata storage node can periodically detect the metric data of at least one operating metric to determine the load status of the metadata storage node, etc.

[0075] In one implementation, the load state can include overload, balanced, and light load states. Under overload conditions, it can be assumed that the metadata storage node may experience a decline in service quality.

[0076] Based on the data of each operational metric, the load status of each operational metric can be determined first. Then, by combining the load status of at least one operational metric, the load status of the metadata storage node can be determined.

[0077] The threshold range corresponding to different load states of each operating metric can be predefined, so that the load state of each operating metric can be determined based on the load state corresponding to the threshold range hit by the metric data of each operating metric.

[0078] When the server is a metadata storage node, the load status determined by the metadata storage node is sent to the client in the response signal. In some embodiments, the client can update the flow control threshold periodically, that is, it can update the flow control threshold at regular intervals based on the most recently received load status, so as to avoid resource consumption caused by frequent updates.

[0079] In another practical application, the server can refer to a collection of data storage nodes in a distributed storage system. Since a distributed storage system consists of multiple distributed data storage nodes, the load status of the data storage node collection can be determined by the load status of each individual data storage node. Obtaining the server's load status can include:

[0080] The load status of the data storage node set is obtained from the load assessment node; wherein the load status of the data storage node set is determined based on the load status of the data storage nodes; and the load status of the data storage nodes can be determined based on the indicator data of at least one operating indicator of the data storage nodes obtained by detection.

[0081] The data storage node may include at least one of the following operational metrics: resource utilization, request processing queue length, request processing latency, and cache hit rate. Resource utilization may include, for example, storage device utilization, memory utilization, and processing unit utilization. Storage devices may refer to disks, and processing units may refer to CPUs. Detailed explanations of each operational metric can be found in the embodiments described above.

[0082] Among them, the data storage node can periodically detect the indicator data of at least one operating indicator and determine the corresponding load status.

[0083] The load assessment node can periodically collect the load status of each data storage node in the data storage node set, and determine the load status of the data storage node set accordingly.

[0084] The client can also periodically obtain the load status of the data storage node set from the load platform node, or the load assessment node can actively push the load status of the data storage node set to the client after determining it. This application does not limit this.

[0085] The load status of the data storage node set can include overload, balanced, and light load states. Under overload conditions, the data storage node set may experience a decline in service quality.

[0086] The data storage node can first determine the load status of each operating indicator based on the indicator data of each operating indicator, and then combine the load status of at least one operating indicator to determine the load status of the data storage node.

[0087] The threshold range corresponding to different load states of each operating metric can be predefined, so that the load state of each operating metric can be determined based on the load state corresponding to the threshold range hit by the metric data of each operating metric.

[0088] The load assessment node can determine the load status of the data storage node set by combining the load status of multiple data storage nodes in the storage system. For example, if more than a certain number or a certain proportion of data storage nodes are overloaded, the data storage node set can be considered to be overloaded. If more than a certain number or a certain proportion of data storage nodes are lightly loaded, the data storage node set can be considered to be lightly loaded. If a certain number or a certain proportion of data storage nodes are in a balanced state, the data storage node set can be considered to be in a balanced state.

[0089] In some embodiments, updating the flow control threshold based on the load state may include: lowering the flow control threshold when the load state is overloaded; and raising the flow control threshold when the load state is lightly loaded.

[0090] The initial value of the flow control threshold can be set according to actual needs.

[0091] Optionally, lowering the flow control threshold may mean, for example, lowering the flow control threshold by a certain percentage, such as 20%; raising the flow control threshold may mean, for example, raising the flow control threshold by a certain percentage, such as 10%.

[0092] Optionally, the flow control threshold can be adjusted upwards under an upper limit threshold constraint, and downwards under a lower limit threshold constraint. For example, if the upper limit threshold is 1000, meaning the maximum value of the flow control threshold is 1000, then adjusting the flow control threshold upwards cannot exceed the upper limit threshold. Similarly, adjusting the flow control threshold downwards cannot be lower than the lower limit threshold, etc.

[0093] If the load is balanced, the flow control threshold can remain unchanged.

[0094] In some embodiments, the above-mentioned rate limiting control of the processing requests to be sent to the server according to the rate control threshold can be:

[0095] Store the processing requests to be sent into the processing queue; determine whether the request sending status meets the rate limiting conditions based on the flow control threshold; determine whether the number of sent but unprocessed processing requests has reached the flow control threshold; if the request sending status does not meet the rate limiting conditions, retrieve the processing request from the processing queue and send it to the server.

[0096] In other words, processing requests to be sent can first be stored in the processing queue.

[0097] In one implementation, the rate limiting condition can refer to the number of sent but unprocessed processing requests reaching a flow control threshold, where the flow control threshold represents the maximum allowed number of requests. Therefore, if the number of requests is less than the flow control threshold, processing requests can be retrieved from the processing queue and sent to the server, and the request count can be updated after the processing requests are sent.

[0098] Here, "sent but unprocessed processing requests" can refer to processing requests that have been sent but for which no response signal has been received. If the number of requests is less than the flow control threshold, processing requests can be retrieved from the processing queue according to dequeue rules such as first-in-first-out and scheduled to the server. The number of processing requests can be retrieved from the processing queue based on the difference between the number of requests and the flow control threshold, and sent to the server accordingly. Alternatively, processing requests can be retrieved one by one, sent to the server, and then the request count judgment operation can be performed repeatedly. This application does not limit the specific approach.

[0099] If the number of requests reaches the flow control threshold, it indicates that the server's service capacity has reached its limit, and the sending and processing of requests can be paused at this time.

[0100] In another implementation, the rate limiting condition can refer to the traffic rate reaching the flow control threshold. In this case, the flow control threshold represents the maximum allowable traffic rate. Therefore, if the traffic rate is less than the flow control threshold, a processing request can be retrieved from the processing queue and sent to the server.

[0101] Traffic rate can represent the request sending rate, the number of requests sent per unit time, or the data flow per unit time. In practical applications, when processing data access requests, traffic rate can be expressed as IOPS (Input / Output Operations Per Second) or Throughput (the amount of data processed per unit time); when processing metadata operation requests, traffic rate can be expressed as QPS (Queries Per Second).

[0102] Among them, the flow rate can reflect the number of requests that the server can receive and process per unit of time, and can reflect the overall throughput capacity. If the flow rate reaches the flow control threshold, it indicates that the server's service capacity has reached its limit, and at this time, sending and processing requests can be paused.

[0103] In some embodiments, storing the processing request to be sent into the processing queue may include:

[0104] Identify the processing requests to be sent; determine if the request sending status meets the rate limiting conditions; if yes, store the processing request in the processing queue; if no, send the processing request to the server and update the request count.

[0105] The processing request to be sent can refer to a newly generated processing request. When the server is a set of data storage nodes, the newly generated processing request is also the data access request sent by the application using the storage service. When the server is a metadata storage node, the newly generated processing request can refer to a metadata operation request.

[0106] For a processing request to be sent, we can first determine whether the request sending status meets the rate limiting conditions, such as whether the number of requests has reached the rate control threshold or whether the traffic rate has reached the rate control threshold. If so, we can put it into the processing queue. Otherwise, we can send it directly to the server. When the rate limiting condition is that the number of requests has reached the rate control threshold, after the processing request is sent to the server, we can also update the number of requests.

[0107] Furthermore, in some embodiments, when the rate limiting condition is that the number of requests reaches the flow control threshold, the method may further include: updating the request count in response to a response signal sent by the server for any processing request. Receiving a response signal indicates that the processing request has been processed, at which point the request count can be updated, for example, by decrementing it by 1.

[0108] In some embodiments, since the request types for processing requests may be different and the requirements for processing latency may also be different, multiple processing queues corresponding to different request types can be configured, and the priority of the request type can be determined, and the weight value of the processing queue can be determined according to the priority of the request type.

[0109] The above-mentioned storage of the processing request to be sent into the processing queue may include: storing the processing request into the processing queue corresponding to the request type, determining the priority of the request type, and determining the weight value corresponding to the processing queue based on the priority;

[0110] The above-mentioned method of sending processing requests in the processing queue to the server when the request sending status does not meet the rate limiting conditions may include:

[0111] If the request sending status does not meet the rate limiting condition, the processing requests in multiple processing queues are scheduled to the server in descending order of priority and weight value.

[0112] Request types include, for example, read requests and storage requests.

[0113] The weight value can be used to indicate the number of requests retrieved from the processing queue for each scheduling operation.

[0114] For ease of understanding, let's take the server as a set of data storage nodes, and the rate limiting condition as the number of sent but unprocessed requests reaching the flow control threshold as an example. Figure 4The diagram illustrates the scheduling of the processing queue. Processing requests to be sent can originate from upper-layer applications, meaning data access requests such as data read requests or data storage requests. In practical applications, a WRR (Weighted Round Robin) algorithm can be used for scheduling. For data access requests sent from upper-layer applications, it is first determined whether the number of sent but unprocessed data access requests has reached the flow control threshold. If not, the requests can be directly sent to the corresponding data storage node 103. If so, the requests are added to the corresponding processing queue based on their request type, such as read / write or recycling. Each processing queue for each request type has a weight value corresponding to its priority. Then, if the number of requests is less than the flow control threshold, the corresponding number of data access requests can be dequeued from multiple processing queues according to their weight values ​​and sent to the data storage node 103. The weight value corresponding to each priority level can, for example, start from the target value and decrease sequentially by a predetermined value according to the priority level from high to low. For example, if the target value is 5 and the predetermined value is 1, then the weight value corresponding to the highest priority can be 5, the weight value corresponding to the next priority can be 4, and so on. Of course, the weight values ​​corresponding to different priorities can also be pre-configured, with the weight value of higher priority being greater than that of lower priority. This application does not limit the specific setting method of the weight values. Assume there are 3 processing queues: queue 1, queue 2, and queue 3. The priority of queue 1 is higher than that of queue 2, and the priority of queue 2 is higher than that of queue 3. Assume the weight value of queue 1 is 5, the weight value of queue 2 is 3, and the weight value of queue 3 is 1. If the number of requests is less than the flow control threshold (assuming the flow control threshold is 10 and the number of requests is 3), then 5 data access requests can be retrieved from queue 1 and sent to the data storage node. At this point, the number of requests is 8. Then, 2 more data access requests can be retrieved from queue 2 and sent to the data storage node. At this point, the number of requests is 10, which reaches the flow control threshold. Therefore, sending is paused, and the number of requests can continue to be monitored.

[0115] Furthermore, as another embodiment, storing the processing request to be sent into the processing queue includes: storing the processing request into the processing queue corresponding to the request type, and determining the priority of the request type;

[0116] The above-mentioned method of sending processing requests in the processing queue to the server when the request sending status does not meet the rate limiting conditions may include: scheduling processing requests in multiple processing queues to the server in descending order of priority when the request sending status does not meet the rate limiting conditions.

[0117] In other words, requests can be scheduled based solely on priority. Once all requests in the high-priority processing queue have been processed, requests in the next-priority processing queue will be scheduled.

[0118] In practical applications, multiple processing queues may not be fully traversed in a single scheduling round. To ensure that processing requests are processed, as an optional approach, if the request sending status does not meet the rate limiting conditions, processing requests from multiple processing queues can be scheduled to the server sequentially according to their priority from high to low and their weight values. This can include:

[0119] If the request sending status does not meet the rate limiting conditions, the processing requests in multiple processing queues are scheduled to the server in descending order of priority, starting from the first processing queue that was not scheduled in the previous round of scheduling.

[0120] In other words, for the current round of scheduling, scheduling may not start from the highest priority processing queue, but rather from the first processing queue that was not scheduled in the previous round, to ensure that all processing queues can be scheduled. For example, in the example above, let's assume the processing request is a data access request, with queues 1, 2, and 3. The priority of queue 1 is higher than that of queue 2, and the priority of queue 2 is higher than that of queue 3. Let's assume the weight of queue 1 is 5, the weight of queue 2 is 3, and the weight of queue 3 is 1. If the number of requests is less than the flow control threshold (assuming the threshold is 10 and the number of requests is 3), then 5 data access requests can be retrieved from queue 1 and sent to the data storage node, bringing the total number of requests to 8. Then, 2 more data access requests can be retrieved from queue 2 and sent to the data storage node, bringing the total number of requests to 10. This reaches the flow control threshold, so sending is paused and one round of scheduling ends. The number of requests can continue to be monitored. If the number of requests is less than the flow control threshold again, then the process can start from queue 3, which was not scheduled in the previous round, and retrieve data access requests from queue 3 to schedule the data storage node. If the number of requests is still less than the flow control threshold, then the scheduling can continue from queue 1.

[0121] As another optional approach, if the request sending status does not meet the rate limiting conditions, the processing requests in multiple processing queues can be scheduled to the server in descending order of priority and weight values. This can include:

[0122] If the request sending status does not meet the rate limiting conditions, check if there is at least one processing queue among the multiple processing queues that has not been scheduled for a predetermined time; if so, retrieve the processing request from at least one processing queue and send it to the server; if not, schedule the processing requests in the multiple processing queues to the server in order of priority from high to low and weight values.

[0123] In other words, when performing the current round of scheduling, we can first determine whether there are processing queues that have not been scheduled for a predetermined time. If so, we can first schedule the processing requests in the processing queues that have not been scheduled for a predetermined time.

[0124] Optionally, processing requests can be retrieved from at least one processing queue according to their priority from high to low and their weight values ​​and sent to the server. After at least one processing queue has been scheduled, scheduling can then begin from the highest priority processing queue.

[0125] In some embodiments, to improve processing efficiency, the client is typically divided into multiple processing threads. Processing requests to be sent can be scheduled to the corresponding processing threads according to principles such as average allocation or load balancing. Therefore, storing the processing requests to be sent in the processing queue can include:

[0126] The processing requests to be sent are assigned to the target processing thread, so that the target processing thread can store the processing requests in the processing queue.

[0127] The above-mentioned determination of whether the request sending status meets the rate limiting conditions may include: determining whether the request sending status corresponding to the target processing thread meets the rate limiting conditions, such as determining whether the number of sent but unprocessed processing requests corresponding to the target processing thread reaches the flow control threshold or whether the traffic rate corresponding to the target processing thread reaches the flow control threshold.

[0128] The above-mentioned process of retrieving a processing request from the processing queue and sending it to the server includes: retrieving a processing request from the processing queue of the target processing thread and sending it to the server.

[0129] Flow control thresholds can be used to specifically constrain the request sending behavior of each processing thread.

[0130] For ease of understanding, let's take the server as a set of data storage nodes, and the rate limiting condition as the number of sent but unprocessed requests reaching the flow control threshold as an example. Figure 5 The diagram illustrates the scheduling of the processing queue. Processing requests to be sent can originate from upper-layer applications; these requests are essentially data access requests. Client 101 can send data access requests from the application to processing thread 501 according to either an average distribution principle or a load balancing principle.

[0131] The flow control threshold updated by client 101 can be used to constrain the number of requests sent by each processing thread to data storage node 103. For each processing thread, if the number of its corresponding sent but unprocessed data access requests does not reach the flow control threshold, the data access request can be sent directly to the data storage node. If the number of requests reaches the flow control threshold, the data access request can be added to the processing queue corresponding to the processing thread. If the number of requests is less than the flow control threshold, the data access request can then be retrieved from the processing queue and sent to the data storage node.

[0132] Figure 6 A flowchart illustrating yet another embodiment of a control method provided in this application is provided. The method may include the following steps:

[0133] 601: Determines the server's load status.

[0134] As an optional approach, when the server is a set of data storage nodes, the technical solution of this embodiment can be executed by a load assessment node. Therefore, determining the load status of the server includes: collecting the load status of multiple data storage nodes in the storage system; and combining the load status of the multiple data storage nodes to determine the load status of the set of data storage nodes.

[0135] Among them, data storage nodes can periodically report their own load status to load assessment nodes, or load assessment nodes can periodically obtain the load status of data storage nodes from data storage nodes.

[0136] As another optional approach, when the server is a metadata storage node, the technical solution of this embodiment can be executed by the metadata storage node. In this case, determining the load status of the server may include: detecting the metric data of at least one operating metric in the metadata storage node; and determining the load status of the metadata storage node based on the metric data of the at least one operating metric.

[0137] 602: Provides the load status to the client so that the client can update the flow control threshold based on the server's load status. The flow control threshold is used to limit the flow of requests to be sent to the server for processing.

[0138] For details on the specific execution operations on the client side, please refer to [link / reference]. Figure 1 The embodiments shown are described herein and will not be repeated here.

[0139] In this embodiment, the client updates the flow control threshold according to the server's load status. That is, the client can limit the flow of requests to be sent to the server according to the flow control threshold. By limiting the flow of requests or the flow rate sent to the server, the client can balance the traffic pressure on the server and ensure the storage service capacity. Furthermore, the client can dynamically update the threshold based on the server's load status, which not only ensures storage performance but also improves resource utilization.

[0140] As described above, the load state can include overload state, light load state, and balanced state.

[0141] In the case where the server is a set of data storage nodes, in one optional approach, determining the load status of the data storage node set by combining the load status of multiple data storage nodes may include:

[0142] If the number of data storage nodes in an overloaded state among multiple data storage nodes meets the first overload condition, the data storage node set is determined to be in an overloaded state; if the number of data storage nodes in a lightly loaded state among multiple data storage nodes meets the first lightly loaded condition, the data storage node set is determined to be in a lightly loaded state.

[0143] The first overload condition can be, for example, that the number of data storage nodes in an overloaded state is greater than a certain number, or that the proportion of data storage nodes in an overloaded state is greater than a certain percentage, such as more than 50% (percentage) of the data storage nodes being in an overloaded state; the first light load condition can be, for example, that the number of data storage nodes in a light load state is greater than a certain number, or that the proportion of data storage nodes in a light load state is greater than a certain number.

[0144] In another alternative approach, determining the load status of the data storage node set by combining the load status of multiple data storage nodes may include:

[0145] Compared to the number of data storage nodes that were overloaded in the previous period, the number of newly added data storage nodes that were overloaded in the current period meets the second overload condition, thus determining that the data storage node set is in an overloaded state; compared to the number of data storage nodes that were lightly loaded in the previous period, the number of newly added data storage nodes that were lightly loaded in the current period meets the second lightly loaded condition, thus determining that the data storage node set is in an overloaded state.

[0146] The second overload condition can be, for example, that the number of newly added data storage nodes in an overloaded state is greater than a certain number, or that the number of newly added nodes exceeds a certain proportion, such as more than 10% of the data storage nodes being in an overloaded state; the second light load condition can be, for example, that the number of newly added data storage nodes in an overloaded state is greater than a certain number, or that the number of newly added nodes exceeds a certain proportion, such as more than 10% of the data storage nodes being in a light load state.

[0147] For those states that are neither overloaded nor lightly loaded, the data storage node set can be determined to be in a balanced state.

[0148] When the server is a metadata storage node, determining the load status of the metadata storage node based on at least one operational metric includes:

[0149] Based on the indicator data of at least one operating indicator and the threshold range of at least one operating indicator for different load states, determine the load state of at least one operating indicator.

[0150] The load status of the metadata storage node is determined by combining the load status of at least one operational metric.

[0151] In other words, based on the data of each operating metric, the load status of each operating metric can be determined first. Then, by combining the load status of at least one operating metric, the load status of the metadata storage node can be determined.

[0152] The load status of each operational indicator can be determined based on the load status corresponding to the threshold range hit by the indicator data of each operational indicator.

[0153] As an optional approach, determining the load status of a metadata storage node by combining the load status of at least one operational metric may include:

[0154] If any one of the operating metrics is overloaded, the metadata storage node is determined to be overloaded; if at least one operating metric is in a balanced state, the metadata storage node is determined to be in a balanced state; if at least one operating metric is not overloaded and one or more operating metrics are in a light-loaded state, the metadata storage node is determined to be in a light-loaded state.

[0155] In other words, if any one operating metric is overloaded, the metadata storage node can be considered to be overloaded. If all operating metrics are in a balanced state, the metadata storage node can be considered to be in a balanced state. In other cases, if no operating metric is overloaded and at least one operating metric is underloaded, the metadata storage node can be considered to be underloaded.

[0156] As an alternative approach, determining the load status of a metadata storage node by combining the load status of at least one operational metric can include:

[0157] Based on the load status of at least one operational metric, determine the load score of at least one operational metric; weight the load scores of at least one operational metric to obtain a total load score; and determine the load status of the metadata storage node based on the total load score.

[0158] For example, different load states can be pre-set with corresponding load scores. Different operating indicators can be assigned corresponding weight coefficients, so that the total load score can be obtained through a weighted summation method. Of course, in a practical application, the weight coefficients of different operating indicators can all be 1, and the total load can be the average of the load scores of all operating indicators.

[0159] Among them, the load score of the overload state can be greater than the load score of the balanced state, and the load score of the balanced state can be greater than the load score of the light load state. Therefore, if the total load score is greater than the first value, the metadata storage node is determined to be in an overload state. If the total load score is less than the second value, the metadata storage node is determined to be in a light load state. In other cases, the metadata storage node can be considered to be in a balanced state.

[0160] Among them, the overload state can be used to trigger a downward adjustment of the flow control threshold; the light load state can be used to trigger an upward adjustment of the flow control threshold; and the balanced state can be used to trigger the maintenance of the flow control threshold unchanged.

[0161] Figure 7 This is a flowchart illustrating yet another embodiment of a control method provided by this application. The technical solution of this embodiment can be executed by a data storage node, and the method may include the following steps:

[0162] 701: Detects the indicator data of at least one running indicator in the data storage node.

[0163] 702: Determine the load status of the data storage node based on the data of at least one operational metric.

[0164] 703: Provide the load status of the data storage nodes to the load assessment node so that the load assessment node can combine the load status of multiple data storage nodes in the storage system to determine the load status of the data storage node set; the load status of the data storage node set is used to trigger the adjustment operation of the flow control threshold, which is used to control the request sending operation for the data storage node.

[0165] Based on the data of each operational metric, the load status of each operational metric can be determined first. Then, by combining the load status of at least one operational metric, the load status of the data storage node can be determined.

[0166] The threshold range corresponding to different load states of each operating metric can be predefined, so that the load state of each operating metric can be determined based on the load state corresponding to the threshold range hit by the metric data of each operating metric.

[0167] In one alternative approach, determining the load status of a data storage node based on the load status of at least one operational metric may include:

[0168] If any one of the operating metrics is overloaded, the data storage node is determined to be overloaded; if at least one operating metric is in a balanced state, the data storage node is determined to be in a balanced state; if at least one operating metric is not overloaded and one or more operating metrics are in a light-load state, the data storage node is determined to be in a light-load state.

[0169] In other words, if any one operating indicator is overloaded, the data storage node can be considered to be overloaded. If all operating indicators are in a balanced state, the data storage node can be considered to be in a balanced state. In other cases, if no operating indicator is overloaded and at least one operating indicator is underloaded, the data storage node can be considered to be underloaded.

[0170] In another alternative approach, determining the load status of a data storage node based on the load status of at least one operational metric may include:

[0171] Based on the load status of at least one operational indicator, determine the load score of at least one operational indicator; weight the load scores of at least one operational indicator to obtain a total load score; and determine the load status of the data storage node based on the total load score.

[0172] For example, different load states can be pre-set with corresponding load scores. Different operational metrics can be assigned corresponding weighting coefficients, allowing the total load score to be obtained through a weighted summation. Of course, in a practical application, the weighting coefficients for different operational metrics can all be 1, and the total load score can be the average of the load scores for all operational metrics.

[0173] Among them, the load score of the overload state can be greater than the load score of the balanced state, and the load score of the balanced state can be greater than the load score of the light load state. Therefore, if the total load score is greater than the third value, the data storage node can be determined to be in an overload state. If the total load score is less than the fourth value, the data storage node can be determined to be in a light load state. In other cases, the data storage node can be considered to be in a balanced state.

[0174] Among them, the data storage node can periodically detect the indicator data of at least one operating indicator and determine the corresponding load status.

[0175] The load assessment node can periodically collect the load status of each data storage node in the storage system and determine the load status of the data storage node set accordingly.

[0176] The client can also periodically obtain the load status of the data storage node set from the load platform node, or the load assessment node can actively push the load status of the data storage node set to the client after determining it. This application does not limit this.

[0177] The load assessment node can determine the load status of the data storage node set by combining the load status of multiple data storage nodes in the storage system. For example, if more than a certain number or a certain proportion of data storage nodes are overloaded, the data storage node set can be considered to be overloaded. If more than a certain number or a certain number of data storage nodes are lightly loaded, the data storage node set can be considered to be lightly loaded. If a certain number or a certain number of data storage nodes are in a balanced state, the data storage node set can be considered to be in a balanced state. The specific implementation method can be found in the corresponding embodiments described above, and will not be repeated here.

[0178] To make it easier to understand, the following example uses a storage system. Figure 8 The diagram shown is a scenario interaction diagram in which the server is a set of data storage nodes, according to an embodiment of this application.

[0179] In this system, multiple distributed data storage nodes 103 constitute a data storage node set. Therefore, the load status of the data storage node set is determined by the load status of each data storage node. In this embodiment, the storage engine in the data storage node 103 can be used to periodically detect software / hardware indicators that reflect whether the data storage node has reached a performance bottleneck, such as disk utilization, request processing queue length, request processing latency, memory utilization, etc.

[0180] Each operational metric can be divided into multiple intervals according to its value range. Each interval represents a different load state of the operational metric, such as overload, balanced, and light load. For example, if the disk utilization reaches 95% or more, the disk is considered to be in an overload state; if the disk utilization is between 30% and 95%, the disk is considered to be in a balanced state; and if the disk utilization is below 30%, the disk is considered to be in a light load state.

[0181] Optionally, if any operational metric is overloaded or remains overloaded for a certain period, the data storage node may experience a service degradation, and therefore can be considered overloaded. If all operational metrics are in a balanced state, the data storage node is considered to be in a balanced state; otherwise, it can be considered to be underloaded.

[0182] The load assessment node 104 can periodically collect the load status of all data storage nodes in the data storage node set and assess the load status of the data storage node set by combining the load status of all data storage nodes. For example, if more than a certain percentage of data storage nodes in the data storage node set are in an overloaded state, it can be considered that the stability of the storage system will have problems, and eventually a system avalanche and service outage may occur. Therefore, for example, if more than 50% of the data storage nodes are in an overloaded state, the load status of the data storage node set can be considered as overloaded; or if more than 10% of the data storage nodes are in an overloaded state compared to the previous period, the load status of the data storage node set is considered as overloaded, and so on.

[0183] Client 101 can periodically query the load status of the data storage node set from the load assessment node 104, and adjust the flow control threshold according to the load status. For example, if the data storage node set is overloaded, the flow control threshold is lowered by a certain percentage, such as 20%; if the data storage node set is in a balanced state, the flow control threshold remains unchanged; if the data storage node set is in a light-load state, the flow control threshold is increased by a certain percentage, such as 10%, but the maximum cannot exceed the upper limit of the concurrency flow control threshold, such as 1000.

[0184] Client 101 can receive data access requests sent by its corresponding application and perform flow control according to the adjusted flow control threshold. For details of the implementation, please refer to the previous embodiments, which will not be repeated here.

[0185] In this implementation, the load assessment node collects the load status of each data storage node in the storage system, and then summarizes the results to assess the load status of the data storage node set. Clients periodically query the load status of the data storage node set maintained by the load assessment node, and adaptively adjust their flow control thresholds according to a certain strategy. In this way, the resource utilization of the storage cluster can be improved while reducing the manpower costs of operation and maintenance.

[0186] Furthermore, the client supports prioritizing data access requests based on different request types. By differentiating traffic types in this way, when rate limiting is implemented for data access requests, the performance of high-priority data streams can be guaranteed. When the pressure on high-priority data streams decreases, low-priority data streams can preempt idle traffic, avoiding resource waste.

[0187] like Figure 9 The diagram illustrates another scenario interaction diagram in which the server is a metadata storage node, according to an embodiment of this application.

[0188] Different types of metadata operations result in different system pressures on the metadata storage node 102. The metadata storage node 102 detects software / hardware operating indicators that can reflect whether the metadata storage node 102 has reached the service capacity bottleneck, such as request processing queue length, request processing latency, cache hit rate, CPU utilization, memory utilization, etc.

[0189] Each operational metric can be divided into multiple intervals according to its value range. Each interval can represent the load status of the operational metric, such as overload, balanced, and light load. For example, if the CPU utilization reaches more than 4000%, the CPU is considered to be in an overload state; if the CPU utilization is between 3000% and 4000%, the CPU is considered to be in a balanced state; and if the CPU utilization is below 3000%, the metadata storage node is considered to be in a light load state.

[0190] Optionally, if any operational metric is overloaded or remains overloaded for a certain period, the metadata storage node may experience a service degradation, and therefore can be considered overloaded. If all operational metrics are in a balanced state, the metadata storage node is considered a balanced node; otherwise, it can be considered lightly loaded.

[0191] The load status of metadata storage node 102 can be carried in a response signal and sent to client 101. Client 101 can adjust the flow control threshold according to the load status of metadata storage node 102. For example, if the metadata storage node is overloaded, the client can lower the flow control threshold by a certain percentage, such as 20%, but not below the lower limit of the flow control threshold. If the metadata storage node is in a balanced state, the client keeps the flow control threshold unchanged. If the metadata storage node is under light load, the metadata storage node can increase the flow control threshold by a certain percentage, such as 10%, but not exceeding the upper limit of the flow control threshold.

[0192] Each time client 101 receives a metadata operation request, such as creating or opening a file, it will perform rate limiting control according to the adjusted flow control threshold. The specific implementation method can be found in the previous embodiment, and will not be repeated here.

[0193] After a metadata operation request is sent to and processed by the metadata storage node 102, the metadata storage node 102 sends a response signal back to the client 101, carrying with it a real-time assessed load status. The client 101 can then adjust the flow control threshold based on the load status of the metadata storage node 102. Furthermore, each time the client 101 adjusts the flow control threshold based on the load status of the metadata storage node 102, it can record the timestamp of this adjustment. Only after a certain time period (e.g., 1 second) is the flow control threshold allowed to be updated again. In other words, the client 101 can periodically adjust the flow control threshold based on the latest received load status.

[0194] In this implementation scheme, the metadata storage node assesses its load status by detecting whether its software / hardware performance indicators have reached service capacity bottlenecks. After transmitting the load status of the metadata storage node to the client via response signals to metadata operation requests, the client can adaptively and periodically adjust its flow control thresholds according to certain strategies. This collaboration between the client and the metadata storage node improves the service availability of the metadata storage node while reducing operational manpower costs.

[0195] Combination Figure 1 or Figure 2 As described in the illustrated embodiments, this application also provides a control system, which can be composed of clients and metadata storage nodes in a distributed storage system;

[0196] The client is used to execute, for example Figure 3 The control method in the illustrated embodiment; the metadata storage node is used to perform, as shown in the example. Figure 6 The control method described in the illustrated embodiment.

[0197] In addition, embodiments of this application also provide a control system, which may include a client in a distributed storage system, a storage cluster consisting of data storage nodes and metadata storage nodes, and a load assessment node.

[0198] The client is used to execute, for example Figure 3 The control method described in the illustrated embodiment uses a data storage node to execute, for example... Figure 7 The control method described in the illustrated embodiment. The load assessment node can be used to perform, for example... Figure 6 The control method described in the illustrated embodiment.

[0199] Furthermore, embodiments of this application also provide a computing device, such as... Figure 10 As shown, the device may include a storage component 1001 and a processing component 1002;

[0200] Storage component 1001 stores one or more computer instructions, wherein the one or more computer instructions are invoked and executed by processing component 1002 to achieve, for example, Figure 3 The control method described in the illustrated embodiment or as shown in the figure Figure 6 The control method described in the illustrated embodiment or as shown in the figure Figure 7 The control method described in the illustrated embodiment.

[0201] Of course, computing devices may also include other components, such as input / output interfaces, display components, communication components, etc.

[0202] Input / output interfaces provide interfaces between processing components and peripheral interface modules, which can be output devices, input devices, etc. Communication components are configured to facilitate wired or wireless communication between computing devices and other devices.

[0203] The processing component 1002 may include one or more processors to execute computer instructions to complete all or part of the steps in the above-described method. Alternatively, the processing component may be implemented as one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above-described method.

[0204] Storage component 1001 is configured to store various types of data to support operations at the terminal. The storage component can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0205] It should be noted that the aforementioned computing devices can be physical devices or elastic computing hosts provided by cloud computing platforms. They can be implemented as a distributed cluster of multiple servers or terminal devices, or as a single server or a single terminal device.

[0206] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a computer, can perform the following functions: Figure 3The control method described in the illustrated embodiment or as shown in the figure Figure 6 The control method described in the illustrated embodiment or as shown in the figure Figure 7 The control method described in the illustrated embodiment. This computer-readable medium may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device.

[0207] This application also provides a computer program product, which includes a computer program / instructions carried on a computer-readable storage medium, wherein the computer program / instructions, when executed by a computer, can perform the following: Figure 3 The control method described in the illustrated embodiment or as shown in the figure Figure 6 The control method described in the illustrated embodiment or as shown in the figure Figure 7 The control method described in the illustrated embodiment. In such an embodiment, the computer program / instructions may be downloaded and installed from a network, and / or installed from a removable medium. When the computer program / instructions are executed by the processing component, they perform various functions defined in the system of this application.

[0208] It should be noted that the embodiments of this application may involve the use of user data. In practical applications, user-specific personal data may be used in the scheme described herein within the scope permitted by applicable laws and regulations, provided that it complies with the applicable laws and regulations of the country (e.g., with the user's explicit consent, with the user being properly notified, etc.).

[0209] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0210] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0211] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0212] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A control method characterized by, The method applied to a client in a storage system comprises: obtaining a load state of a server in the storage system, wherein the load state is determined based on index data of at least one running index; updating a flow control threshold according to the load state; controlling flow of processing requests to be sent to the server according to the flow control threshold.

2. The method of claim 1, wherein, The controlling flow of processing requests to be sent to the server according to the flow control threshold comprises: storing the processing requests to be sent in a processing queue; judging whether a request sending condition meets a flow control condition according to the flow control threshold, wherein the flow control condition comprises that a number of processing requests that have been sent and not processed reaches the flow control threshold or a flow rate reaches the flow control threshold; in a case where the request sending condition does not meet the flow control condition, obtaining processing requests from the processing queue and sending the processing requests to the server.

3. The method of claim 2, wherein, The storing the processing requests to be sent in the processing queue comprises: determining the processing requests to be sent; judging whether the request sending condition meets the flow control condition; if yes, storing the processing requests in the processing queue; if no, sending the processing requests to the server and updating the number of requests.

4. The method of claim 2, wherein, The storing the processing requests to be sent in the processing queue comprises: storing the processing requests in a processing queue corresponding to a request type and determining a priority of the request type and a weight value corresponding to the processing queue according to the priority; The obtaining processing requests from the processing queue and sending the processing requests to the server in the case where the request sending condition does not meet the flow control condition comprises: in the case where the request sending condition does not meet the flow control condition, sequentially dispatching processing requests in a plurality of processing queues to the server according to an arrangement order from high to low of the priority and the weight value.

5. The method of claim 4, wherein, The obtaining processing requests from the processing queue and sending the processing requests to the server in the case where the request sending condition does not meet the flow control condition comprises: in the case where the request sending condition does not meet the flow control condition, sequentially dispatching processing requests in a plurality of processing queues to the server according to an arrangement order from high to low of the priority and the weight value, starting from a first processing queue that has not been dispatched in a previous round of dispatching; or, in the case where the request sending condition does not meet the flow control condition, detecting whether there is at least one processing queue that has not been dispatched for a predetermined time length in the plurality of processing queues; if yes, obtaining processing requests from the at least one processing queue and sending the processing requests to the server; if no, sequentially dispatching processing requests in the plurality of processing queues to the server according to the arrangement order from high to low of the priority and the weight value.

6. The method of claim 2, wherein, The storing the processing requests to be sent in the processing queue comprises: allocating the processing requests to be sent to a target processing thread, so that the target processing thread stores the processing requests in the processing queue; The judging whether the request sending condition meets the flow control condition comprises: judging whether a request sending condition corresponding to the target processing thread meets the flow control condition. The obtaining the processing request from the processing queue and sending to the server comprises: obtaining the processing request from the processing queue of the target processing thread and sending to the server.

7. The method of claim 1, wherein, The server is a data storage node set composed of a plurality of data storage nodes in the storage system. The obtaining the load state of the server comprises: Obtaining the load state of the data storage node set from the load evaluation node; wherein the load state of the data storage node set is determined according to the load state of the data storage node; the load state of the data storage node is determined based on the index data of at least one running index of the data storage node detected.

8. The method of claim 1, wherein, The server comprises a metadata storage node. The obtaining the load state of the server comprises: Receiving the response signal of any processing request sent by the metadata storage node; Determining the load state of the metadata storage node carried in the response signal; the load state of the metadata storage node is determined based on the index data of at least one running index of the metadata storage node detected.

9. The method of claim 1, wherein, The load state comprises an overload state, a balanced state and a light load state. The updating the flow control threshold according to the load state comprises: In the case of the load state being an overload state, the flow control threshold is lowered; In the case of the load state being a light load state, the flow control threshold is raised.

10. A control method characterized by, Comprise: Determining the load state of the server; Providing the load state to the client, so that the client updates the flow control threshold according to the load state of the server, and the flow control threshold is used to control the flow of the request to be sent to the server.

11. The method of claim 10, wherein, The server is a data storage node set composed of a plurality of data storage nodes in the storage system. The determining the load state of the server comprises: Collecting the load state of a plurality of data storage nodes in the storage system; Combining the load state of the plurality of data storage nodes, the load state of the data storage node set is determined.

12. The method of claim 11, wherein, The combining the load state of the plurality of data storage nodes, the load state of the data storage node set is determined comprises: If the number of data storage nodes in the overload state in the plurality of data storage nodes meets the first overload condition, it is determined that the data storage node set is in an overload state; if the number of data storage nodes in the light load state in the plurality of data storage nodes meets the first light load condition, it is determined that the data storage node set is in a light load state; Or, compared with the number of data storage nodes in the overload state in the previous period, the number of newly added data storage nodes in the overload state in the current period meets the second overload condition, and it is determined that the data storage node set is in an overload state; compared with the number of data storage nodes in the light load state in the previous period, the number of newly added data storage nodes in the light load state in the current period meets the second light load condition, and it is determined that the data storage node set is in a light load state.

13. The method of claim 10, wherein, The server is a metadata storage node; The determining the load state of the server comprises: Detecting the index data of at least one running index of the metadata storage node; According to index data of the at least one running index, a load state of the metadata storage node is determined.

14. The method of claim 13, wherein, The determining of the load state of the metadata storage node according to the index data of the at least one running index comprises: According to the index data of the at least one running index and a threshold range corresponding to different load states of the at least one running index, a load state of the at least one running index is determined; In combination with the load state of the at least one running index, a load state of the metadata storage node is determined.

15. The method of claim 14, wherein, The load state comprises an overload state, a balanced state and a light load state; the determining of the load state of the metadata storage node in combination with the load state of the at least one running index comprises: When any one of the running indexes is in the overload state, it is determined that the metadata storage node is in the overload state; when all of the at least one running index is in the balanced state, it is determined that the metadata storage node is in the balanced state; and when the at least one running index is in the non-overload state and one or more running indexes are in the light load state, it is determined that the metadata storage node is in the light load state; Or, according to the load state of the at least one running index, a load score of the at least one running index is calculated; the load score of the at least one running index is weighted to obtain a total load score; and according to the total load score, the load state of the metadata storage node is determined. The overload state is used to trigger a decrease of the flow control threshold; the light load state is used to trigger an increase of the flow control threshold; and the balanced state is used to trigger a maintenance of the flow control threshold.

16. A control method characterized by, Comprise: Detecting index data of at least one running index in a data storage node; According to the index data of the at least one running index, a load state of the data storage node is determined; The load state of the data storage node is provided to a load evaluation node, so that the load evaluation node determines a load state of a data storage node set composed of a plurality of data storage nodes in combination with load states of the plurality of data storage nodes; and the load state of the data storage node set is used to trigger an update operation of a flow control threshold, which is used to control a request sending operation to the data storage node.

17. A control system characterized by, Comprise a client, a metadata storage node, a data storage node and a load evaluation node; The client is used to perform the control method according to any one of claims 1-9; the data storage node is used to perform the control method according to claim 16; the metadata storage node is used to perform the control method according to any one of claims 10, 13-15; or the load evaluation node is used to perform the control method according to any one of claims 10, 11-12.

18. A computing device, comprising: Comprise a processing component and a storage component; The storage component stores one or more computer instructions; and the one or more computer instructions are used to be called and executed by the processing component to implement the control method according to any one of claims 1-9 or the control method according to any one of claims 10-15 or the control method according to claim 16.

19. A computer-readable storage medium, characterized in that, A computer program product having stored thereon a computer program which, when executed by a computer processing component, implements the control method according to any one of claims 1 to 9 or the control method according to any one of claims 10 to 15 or the control method according to claim 16.

20. A computer program product, characterised in that, A computer program product having stored thereon a computer program which, when executed by a computer processing component, implements the control method according to any one of claims 1 to 9 or the control method according to any one of claims 10 to 15 or the control method according to claim 16.

Images