Resource scheduling method and device, medium, electronic equipment and program product
By obtaining and utilizing the user-side request retry failure information, the resources on the storage node are scheduled to other normal nodes, and the IO jitter and misscheduling problems caused by storage node failures and network failures in distributed storage systems are solved, and the system is improved Stability and high availability.
Patent Information
- Application Number
- CN202411205350.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-29
- Publication Date
- 2024-11-15
AI Technical Summary
Under the cloud computing architecture, the excessive IO jitter caused by storage node failure in distributed storage systems and the error scheduling of batch storage resources caused by network failures affect the stability and high availability of storage services.
By obtaining the user-side request retry failure information, and when the storage node is heartbroken and loses contact, the target storage resource is scheduled to other storage nodes with normal operating status based on this information, reducing IO jitter during failure and avoiding network failures caused by network failures. misscheduling.
It realizes accurate identification and self-healing of abnormal storage nodes, reduces IO jitter when storage node failures, avoids missed scheduling of batch storage resources, and improves the stability and high availability of distributed storage clusters.
Smart Images

Figure CN118963676A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of distributed storage systems, and in particular, to a resource scheduling method, device, medium, electronic device and program product. Background Art
[0002] In the cloud computing architecture, users deploy and run applications by purchasing virtual machines (VMs) of Elastic Computing Service (ECS), which usually rely on distributed storage systems to provide related storage services. Therefore, the stability and high availability of storage services provided by distributed storage systems directly affect the quality of user cloud services. Therefore, how to provide more stable and highly available storage services based on distributed storage systems and reduce input / output (IO) jitter caused by storage node failures is a technical problem that needs to be solved in the current cloud computing field. Summary of the invention
[0003] This summary is provided to introduce concepts in a brief form that will be described in detail in the detailed description below. This summary is not intended to identify key features or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.
[0004] In a first aspect, the present disclosure provides a resource scheduling method based on a distributed storage system, comprising: obtaining request retry failure information generated by a user side in a distributed storage cluster when a request retry fails, and storing the request retry failure information, wherein the request retry failure information is at least used to indicate a target storage resource where the request retry failure occurs; when it is monitored that a heartbeat loss occurs at a target storage node in the distributed storage cluster, scheduling the target storage resource where the request retry failure occurs on the target storage node to other storage nodes in the distributed storage cluster that are in normal operating status according to the request retry failure information related to the target storage node.
[0005] In a second aspect, the present disclosure provides a resource scheduling device based on a distributed storage system, comprising: an acquisition module, used to obtain request retry failure information generated by a user side in a distributed storage cluster when a request retry fails, and store the request retry failure information, wherein the request retry failure information is at least used to indicate a target storage resource where the request retry failure occurs; a scheduling module, used to, when monitoring that a heartbeat loss occurs in a target storage node in the distributed storage cluster, schedule the target storage resource where the request retry failure occurs on the target storage node to other storage nodes in the distributed storage cluster that are in normal operating status according to the request retry failure information related to the target storage node.
[0006] In a third aspect, the present disclosure provides a computer-readable medium having a computer program stored thereon, which implements the steps of any one of the methods described in the first aspect when the computer program is executed by a processing device.
[0007] In a fourth aspect, the present disclosure provides an electronic device, including: a storage device having a computer program stored thereon; A processing device is used to execute the computer program in the storage device to implement the steps of any one of the methods in the first aspect.
[0008] In a fifth aspect, the present disclosure provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of any one of the methods in the first aspect.
[0009] By adopting the above technical solution, it is possible to obtain the request retry failure information generated by the user side in the distributed storage cluster when the request retry fails, and the request retry failure information is at least used to indicate the target storage resource where the request retry failure occurs. Then, when it is monitored that the heartbeat of the target storage node in the distributed storage cluster is lost, the target storage resource on the target storage node where the request retry failure occurs is scheduled to other storage nodes in the distributed storage cluster with normal operating status according to the request retry failure information related to the target storage node. Therefore, the abnormal storage node can be accurately identified according to the request retry failure information, and because only the target storage resource on the target storage node where the request retry fails occurs is scheduled when the heartbeat is lost, the IO jitter caused by the storage node failure can be reduced, and the batch storage resource mis-scheduling caused by network failures between the management node and the storage node in the related technology can be avoided.
[0010] Other features and advantages of the present disclosure will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS
[0011] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale. In the drawings: Figure 1 It is a schematic diagram of a distributed storage cluster architecture in related technologies.
[0012] Figure 2 It is a schematic diagram of cloud disk shard scheduling on storage nodes in the related art.
[0013] Figure 3This is another schematic diagram of cloud disk shard scheduling on storage nodes in the related art.
[0014] Figure 4 The present invention is a flowchart of a resource scheduling method based on a distributed storage system according to an embodiment of the present invention.
[0015] Figure 5 is a schematic diagram of a multi-level state machine according to an embodiment of the present disclosure.
[0016] Figure 6 This is a schematic diagram of pre-scheduling after heartbeat recovery according to an embodiment of the present disclosure.
[0017] Figure 7 It is a schematic diagram of resource scheduling in a scenario where a network anomaly occurs between a management node and a storage node according to an embodiment of the present disclosure.
[0018] Figure 8 It is a schematic block diagram of a resource scheduling device based on a distributed storage system according to an embodiment of the present disclosure.
[0019] Fig. 9 A schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present disclosure is shown. DETAILED DESCRIPTION
[0020] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.
[0021] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.
[0022] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.
[0023] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.
[0024] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".
[0025] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.
[0026] Figure 1 Schematic diagram of distributed storage cluster architecture in related technologies. Figure 1 As shown, the storage cluster 10 is a distributed storage cluster. The storage cluster 10 includes storage nodes, such as storage node 0, storage node 1, and storage node n. The storage cluster 10 also includes a management node. The management node is used to manage the storage nodes. The storage node is used to process IO read and write requests of virtual machines (VMs). Multiple cloud disk shards, i.e., storage resources, are scheduled on each storage node.
[0027] Currently, the industry usually adopts the heartbeat mechanism to ensure that the management node identifies storage node anomalies: the storage node periodically exchanges heartbeats with the management node. If the management node does not receive a heartbeat request from a storage node within a certain period, it considers that the storage node is in an abnormal state, and converts the state of the storage node from a normal operating state (Up) to a stopped state (Down). All cloud disk shards on the storage node are migrated out, and the storage node is finally isolated. When the storage node resumes heartbeats, the management node reverses the state of the storage node back to the Up state, so that the storage node continues to provide services to the outside world.
[0028] The above abnormal storage node detection and scheduling method based on the heartbeat mechanism has the following problems.
[0029] (1) When a storage node fails, it will cause excessive IO jitter.
[0030] When the heartbeat of a storage node is disconnected, the management node will migrate out all cloud disk shards on the abnormal storage node. However, the actual situation is often that only some cloud disk shards on the abnormal storage node receive IO requests. The behavior of the management node to migrate out all cloud disk shards will also cause the cloud disk shards without IO requests on the abnormal storage node to be migrated out. Figure 2As shown in the figure, cloud disk shards 1_0 and n_2 on abnormal storage node 0 have IO requests, and other cloud disk shards have no IO requests. However, when scheduling, the management node will migrate all cloud disk shards on abnormal storage node 0. This type of invalid scheduling will occupy the scheduling capacity of the storage cluster, prolong the recovery time of the cloud disk shards that are receiving IO requests, and ultimately cause excessive IO jitter during fault recovery.
[0031] (2) Due to network failures between the management node and the storage node, batches of cloud disk shards are mis-scheduled.
[0032] When the storage node is in normal service, if the network between the management node and the storage node is abnormal (for example, network partition, high network latency, network packet loss, etc.), the heartbeat of the storage node will be disconnected. In this case, due to the heartbeat disconnection, the management node will misjudge the status of the storage node, such as misjudging that the storage node is faulty, which will eventually trigger large-scale misscheduling of storage resources, seriously affecting the stability of the storage cluster. Figure 3 As shown in the figure, the heartbeat of storage node 0 is lost due to a network anomaly between storage node 0 and the management node. Although storage node 0 is not abnormal at this time and can still provide normal IO read and write services, the management node misjudged that storage node 0 is abnormal due to the heartbeat loss. Therefore, the management node migrated all cloud disk shards on storage node 0 to storage node 1 and storage node n.
[0033] Figure 4 1 is a flow chart of a resource scheduling method based on a distributed storage system according to an embodiment of the present disclosure. The resource scheduling method based on a distributed storage system can be applied to various distributed storage clusters to perform storage resource scheduling on storage nodes in the distributed storage cluster. Figure 4 As shown, the resource scheduling method based on the distributed storage system according to the embodiment of the present disclosure may include the following steps S41 to S42. The resource scheduling method based on the distributed storage system according to the embodiment of the present disclosure may be executed by a management node in a distributed storage cluster.
[0034] In step S41, request retry failure information generated by the user side in the distributed storage cluster when the request retry fails is obtained and stored, wherein the request retry failure information is at least used to indicate the target storage resource where the request retry failure occurs.
[0035] When IO hang occurs on the user side, the virtual block device will feedback IO hang information to the management node of the distributed storage cluster, that is, IO hang, which means that IO has not been successful during the retry process. The IO hang information represents the request retry failure information, which can indicate that the user side has a failure when requesting the target storage resource. In this way, resource scheduling can be performed dynamically based on the user-side IO feedback. Among them, the storage resource can be the smallest resource unit of the distributed storage cluster. Taking the Elastic Block Storage (EBS) cluster as an example, the storage resource can be a cloud disk shard. The cloud disk refers to the virtual block device provided by EBS, and the cloud disk shard refers to the partition of the cloud disk divided into a fixed size as the smallest resource unit of the EBS cluster to provide IO read and write services on the storage node.
[0036] The management node can obtain the request retry failure information from the user side in real time or periodically or when a network failure occurs, or the user side can actively report the request retry failure information when a request retry failure occurs, such as reporting immediately or periodically. In addition to the request retry failure information, the management node can also actively collect the IO count accumulation value of the storage resources on the storage node and summarize the traffic information of each storage resource.
[0037] In step S42, when it is monitored that the heartbeat of the target storage node in the distributed storage cluster is lost, the target storage resource on the target storage node where the request retry failure occurs is scheduled to other storage nodes in the distributed storage cluster that are in normal operating status based on the request retry failure information related to the target storage node.
[0038] The target storage node is a storage node in the distributed storage cluster where the heartbeat is lost. The number of the target storage node can be one or more.
[0039] For example, in step S42, a storage node in normal operation state may be selected from other storage nodes in the distributed storage cluster except the target storage node, and then a first storage node whose schedulable capacity is greater than the number of target storage resources where the request retry failure occurred is selected from the selected storage nodes in normal operation state, and the target storage resources where the request retry failure occurred on the target storage node are scheduled to the first storage node. Alternatively, according to the load balancing principle, several storage nodes with relatively small loads may be selected from the selected storage nodes in normal operation state, and the target storage resources where the request retry failure occurred on the target storage node are evenly scheduled to these storage nodes with relatively small loads.
[0040] By adopting the above technical solution, it is possible to obtain the request retry failure information generated by the user side in the distributed storage cluster when the request retry fails, and the request retry failure information is at least used to indicate the target storage resource where the request retry failure occurs. Then, when it is detected that the heartbeat of the target storage node in the distributed storage cluster is disconnected, according to the request retry failure information related to the target storage node, the target storage resource where the request retry failure occurs on the target storage node is scheduled to other storage nodes in the distributed storage cluster with normal operating status. Therefore, the abnormal storage node can be accurately identified according to the request retry failure information, and because only the target storage resource where the request retry failure occurs on the target storage node is scheduled when the heartbeat is disconnected, the IO jitter caused by the storage node failure can be reduced, and the batch storage resource mis-scheduling caused by the network failure between the management node and the storage node in the related technology can be avoided. For example, if it is determined according to the request retry failure information that the storage resources on the target storage node can perform normal IO reading and writing, the target storage node will not be judged to be abnormal, and the storage resources on the target storage node will not be scheduled. In addition, when the target storage resource whose request retry fails disappears on the target storage node, the impact of the failure of the target storage node is eliminated, and the self-healing of the target storage node is achieved.
[0041] The disclosed embodiment configures a multi-level state machine for each storage node in a distributed storage cluster. Figure 5 is a schematic diagram of a multi-level state machine according to an embodiment of the present disclosure.
[0042] like Figure 5As shown, the multi-level state machine includes a normal operating state, a scheduling inhibition state, an isolation state, a pre-stop state, a stop operation state, and a pre-operation state. When a storage node is in a normal operating state, storage resources are allowed to be migrated in and out for the storage node. When a storage node is in a scheduling inhibition state, scheduling freeze is performed on the storage node, that is, the migration in and out of storage resources is not allowed. When a storage node is in an isolation state, storage resources with failed request retries are allowed to be migrated out for the storage node, and storage resources are not allowed to be migrated in. When a storage node is in a stop operation state, all storage resources are forced to be migrated out for the storage node. If a storage node loses heartbeat connection in a normal operating state, and the state of the distributed storage cluster meets the scheduling inhibition condition (described in detail below), the state of the storage node is converted from the normal operating state to the scheduling inhibition state. If a storage node loses heartbeat connection in a normal operating state, and the state of the distributed storage cluster does not meet the scheduling inhibition condition, the state of the storage node is converted from the normal operating state to the isolation state. If a storage node is in a scheduling inhibition state, but the scheduling inhibition condition is no longer met, the state of the storage node is converted from the scheduling inhibition state to the isolation state. If a storage node is in an isolated state, if all storage resources on the storage node that have failed request retries are scheduled, the state of the storage node is converted from the isolated state to the pre-stop state. If a heartbeat recovery occurs on a storage node in the isolated state, the state of the storage node is converted from the isolated state to the normal operation state. If a storage node is in a pre-stop state, if the number of scheduled storage resources on the storage node that have failed request retries reaches a preset threshold, the state of the storage node is converted from the pre-stop state to the stopped operation state. If a heartbeat recovery occurs on a storage node in the pre-stop state, the state of the storage node is converted from the pre-stop state to the pre-operation state. If a heartbeat recovery occurs on a storage node in the stopped operation state, the state of the storage node is converted from the stopped operation state to the pre-operation state. If a storage node is in the pre-operation state, if the request retry failure information for the probe cloud disk storage resource on the storage node is obtained, the state of the storage node is rolled back from the pre-operation state to the state before the pre-operation state, for example, if the state before the pre-operation state is the stopped state, it is rolled back to the stopped state, and if the state before the pre-operation state is the pre-stop state, it is rolled back to the pre-stop state. If a storage node is in the pre-operation state, if the request retry failure information for the probe cloud disk storage resource on the storage node is not obtained, the state of the storage node is converted from the pre-operation state to the normal operation state.
[0043] Through the above-mentioned multi-level state machine, the IO jitter caused by storage node failure can be reduced, and the batch storage resource misscheduling caused by network failures between management nodes and storage nodes in related technologies can be avoided, thereby avoiding the distributed storage cluster avalanche caused by batch scheduling in large-scale failure scenarios; in addition, a pre-check mechanism after heartbeat recovery is provided to avoid the risk of secondary failure of storage resources and improve the stability of the distributed storage cluster.
[0044] Furthermore, in order to prevent the distributed storage cluster from avalanching due to unreasonable storage resource scheduling, when scheduling the target storage resource on the target storage node where the request retry failure occurs to other storage nodes in the distributed storage cluster that are in normal operating state according to the request retry failure information related to the target storage node, it is necessary to determine whether it is necessary to protect the entire distributed storage cluster (that is, whether it is necessary to suppress the scheduling of the target storage resource on the target storage node where the request retry failure occurs), that is, it is necessary to determine whether the scheduling suppression conditions are met. Based on this, the scheduling of the target storage resource on the target storage node where the request retry failure occurs to other storage nodes in the distributed storage cluster that are in normal operating state as described in step S42 can be achieved in the following way.
[0045] First, get the status of the distributed storage cluster.
[0046] The status of the distributed storage cluster may include: the total number of storage nodes in the distributed storage cluster (total), that is, the scale of the distributed storage cluster; the total number of storage resources in the distributed storage cluster (segments), taking the storage resources as cloud disk shards as an example, the total number of storage resources refers to the total number of cloud disk shards in the distributed storage cluster; the maximum number of storage resources that a single storage node can carry (limit); the optimized maximum number of storage resources that a single storage node can carry (reserved), that is, in order to enable the storage node to run stably, the maximum number of storage resources carried on the storage node during actual operation will not be based on the maximum number of storage resources, but on the maximum number of storage resources multiplied by the preset reserved value; the average storage node memory usage in the distributed storage cluster (memory_used); the minimum number of storage nodes allowed for fault scheduling (min_fault), that is, the minimum number of faulty storage nodes allowed for scheduling.
[0047] Then, based on the state of the distributed storage cluster, determine whether the scheduling suppression condition is met. That is, the number of storage nodes that allow for large-scale fault scheduling can be determined based on the state of the distributed storage cluster. If the number of storage nodes that allow for large-scale fault scheduling is greater than or equal to the target number of storage nodes where the request retry fails, it is determined that the scheduling suppression condition is not met; if the number of storage nodes that allow for large-scale fault scheduling is less than the target number of storage nodes where the request retry fails, it is determined that the scheduling suppression condition is met.
[0048] The number of storage nodes (faults) that allow for scaled fault scheduling can be determined using the following formula: faults=Max(total-(segments / (limit*(reserved-0.1*memory_used))), min_faults) Assuming that total is 50, segments is 200000, limit is 5120, reserved is 0.9, memory_used is 80%, and min_fault is 2, the calculated number of storage nodes faults that can be scheduled with scale faults is 2.
[0049] Then, if the number of storage nodes that allow for large-scale fault scheduling is greater than or equal to the number of target storage nodes where the request retry fails, the target storage resource on the target storage node where the request retry fails is scheduled to other storage nodes in the distributed storage cluster that are in normal operating state. In this operation, the state transition of the target storage node where the heartbeat is lost is "from normal operating state in the multi-level state machine → isolated state".
[0050] If the number of storage nodes that allow for large-scale fault scheduling is less than the number of target storage nodes where the request retry fails, it is prohibited to perform storage resource migration in and out for the target storage node, which is actually a scheduling freeze for the target storage node. In this operation, the state transition of the target storage node where the heartbeat is lost is "from the normal operation state in the multi-level state machine → the scheduling inhibition state".
[0051] In addition, while the target storage node where the heartbeat loss occurs is in a scheduling inhibition state, it is possible to determine again whether the scheduling inhibition condition is met. If the scheduling inhibition condition is no longer met, the state of the target storage node where the heartbeat loss occurs can be converted from the scheduling inhibition state to the isolation state. In this way, the management node can schedule the target storage resource on the target storage node where the request retry failed to other storage nodes in the distributed storage cluster that are in normal operating status.
[0052] By adopting the above-mentioned scheduling suppression strategy, it is possible to avoid a distributed storage cluster avalanche due to unreasonable storage resource scheduling. For example, it is possible to avoid a distributed storage cluster avalanche caused by batch scheduling in a large-scale failure scenario.
[0053] In some embodiments, the resource scheduling method based on a distributed storage system according to an embodiment of the present disclosure may also include: when the number of target storage resources on the target storage node where the request retry fails is greater than or equal to a preset threshold, after the scheduling of the target storage resources on the target storage node where the request retry fails is completed, the service of the target storage node is stopped, and other storage resources on the target storage node are scheduled (for example, migrated out in batches) to other storage nodes in normal operating status. In this operation, the state transition of the target storage node where the heartbeat is lost is "from the isolation state in the multi-level state machine → pre-stop state → stop running state".
[0054] The preset threshold may be the number of storage resources with traffic on the target storage node where the heartbeat loss occurs multiplied by a fixed ratio.
[0055] The number of target storage resources that have failed to request retries reaches a preset threshold, indicating that the corresponding target storage node has failed. Therefore, by converting the state of such a target storage node to a stopped state and migrating the remaining storage resources on it in batches to other storage nodes in normal operation, the self-healing of the distributed storage cluster is achieved, ensuring the stable operation of the distributed storage cluster. In addition, after the heartbeat of the target storage node is lost, the resource scheduling method according to the embodiment of the present disclosure will not immediately schedule all the storage resources on the target storage node, but will first schedule the target storage resources on the target storage node that have failed to request retries, and then after the number of target storage resources that have failed to request retries reaches a preset threshold, the remaining storage resources on the target storage node will be scheduled in batches, which greatly reduces the IO jitter caused by storage node failures.
[0056] In some embodiments, the resource scheduling method based on a distributed storage system according to an embodiment of the present disclosure may also include: when it is monitored that the target storage node has recovered its heartbeat, scheduling the storage resources with traffic in the probe cloud disk to the target storage node, wherein the probe cloud disk is a cloud disk used to detect whether the target storage node can recover its service; when it is monitored that no request retry failure information for the storage resource with traffic has been obtained within a period of time, performing a management and control operation on the storage resource with traffic, wherein the management and control operation is used to manage the storage resource with traffic; if the management and control operation is executed normally, the service of the target storage node is restored.
[0057] Management and control operations can be performed in batches. Management and control operations can include, for example, modifying properties on storage resources with traffic, actively scheduling storage resources with traffic, and so on.
[0058] For example, after the heartbeat of the target storage node that has lost connection is restored, the target storage node will not be immediately allowed to provide services to the outside world. Instead, the state of the target storage node is converted to a pre-operation state, and the storage resources of the probe cloud disk are pre-scheduled to the target storage node for observation, and then a determination is made as to whether the target storage node should be restored to provide services to the outside world.
[0059] If the request retry failure information for the storage resource with traffic is obtained within the period of time, the storage resource with traffic in the probe cloud disk is rescheduled to the target storage node after a preset time interval, and after the number of rescheduling reaches a preset number, the target storage node is set to a stopped state and the rescheduling is stopped. In addition, in the preset number of rescheduling, the value of the preset time interval between rescheduling can be gradually increased.
[0060] Figure 6 This is a schematic diagram of pre-scheduling after heartbeat recovery according to an embodiment of the present disclosure.
[0061] Assume that before the heartbeat is restored, the state of the target storage node is in the pre-stop state or the stop running state. After the heartbeat of the target storage node is restored, the management node can first convert the state of the target storage node to the pre-running state. Then, the management node can schedule the storage resources with traffic in the probe cloud disk (the management node can determine the storage resources with traffic by actively collecting the real-time traffic information of the storage resources on the probe cloud disk) to the target storage node in the pre-running state, and observe whether these storage resources with traffic will be reported with request retry failure information. If the request retry failure information is reported, the management node will roll back the target storage node to the previous state, that is, the stop running state / pre-stop state; if the user side does not feedback the request retry failure information, the management node will perform batch control operations on the storage resources with traffic. When the batch control operations are also normal, the management node will restore the target storage node to normal operation. At this point, the initial pre-scheduling process is completed.
[0062] In addition, after the first rollback to the previous state, the management node can execute the above pre-scheduling process again after an interval of x minutes. If the pre-scheduling fails, the above pre-scheduling process will be executed again after an interval of 2x minutes. If the pre-scheduling fails, the above pre-scheduling process will be executed again after an interval of 4x minutes. If the management node still receives a report of the request retry failure, the management node will directly roll back the state of the target storage node to the stopped running state and will no longer attempt to recover.
[0063] Through the above-mentioned pre-scheduling process, it can be ensured that the target storage node can provide IO services normally after the heartbeat is restored, avoiding the risk of secondary failure and improving the stability of the distributed storage cluster.
[0064] Next, examples are given to illustrate the beneficial effects of the resource scheduling method based on a distributed storage system according to an embodiment of the present disclosure.
[0065] Technical effect 1: It can reduce the IO jitter caused by storage node failure.
[0066] In an embodiment of the present disclosure, after the heartbeat of the target storage node is lost, the management node will first schedule the target storage resources on the target storage node where request retry failed, and the remaining storage resources on the target storage node will be gradually migrated out in batches after the target storage node is subsequently set to a stopped running state.
[0067] Assume that a distributed storage cluster has 30 storage nodes, a single storage node has 3,000 storage resources, 1,000 of which have IO requests, and a single storage node can migrate storage resources at 10 QPS (Queries-per-second). If a single storage node in the distributed storage cluster goes down: The maximum IO suspension time of the resource scheduling method based on the heartbeat mechanism in the related art is: Maximum IO suspension time = 3000 / (29*10) = 10.344s The maximum IO suspension time of the resource scheduling method based on the distributed storage system according to the embodiment of the present disclosure is: Maximum IO suspension time = 1000 / (29*10) = 3.448s The resource scheduling method based on the distributed storage system according to the embodiment of the present disclosure is superior to the resource scheduling method based on the heartbeat mechanism in the related art.
[0068] Technical effect 2: It can avoid misscheduling of batch storage resources due to network failures between management nodes and storage nodes.
[0069] According to an embodiment of the present disclosure, in a scenario where only the network between the management node and the storage node is abnormal: the management node checks whether the user side has fed back the request retry failure information. If the user side has not fed back the request retry failure information, the scheduling of storage resources will not be triggered. Figure 7 shown.
[0070] Assume that a distributed storage cluster has 30 storage nodes, a single storage node has 3,000 storage resources, 1,000 of which have IO requests, and a single storage node can migrate storage resources at 10 QPS. Then, in the scenario where the management node and a single storage node in the distributed storage cluster lose network connection: The maximum IO suspension time of the resource scheduling method based on the heartbeat mechanism in the related art is: Maximum IO suspension time = 3000 / (29*10) = 10.344s The maximum IO suspension time of the resource scheduling method based on the distributed storage system according to the embodiment of the present disclosure is: Maximum IO suspension time = 0s.
[0071] Technical Effect 3: The resource scheduling method based on the distributed storage system according to the embodiment of the present disclosure provides a large-scale fault scheduling suppression strategy, which can avoid the distributed storage cluster avalanche caused by batch scheduling in large-scale fault scenarios. The resource scheduling method based on the heartbeat mechanism in the related art cannot avoid the distributed storage cluster avalanche caused by batch scheduling in large-scale fault scenarios.
[0072] Technical Effect 4: The resource scheduling method based on the distributed storage system according to the embodiment of the present disclosure provides a pre-scheduling mechanism after the heartbeat of the storage node is restored, which can avoid the risk of secondary failure. The resource scheduling method based on the heartbeat mechanism in the related art directly restores the IO service of the storage node after the heartbeat of the storage node is restored, which cannot avoid the risk of secondary failure.
[0073] Figure 8 1 is a schematic block diagram of a resource scheduling device based on a distributed storage system according to an embodiment of the present disclosure. The resource scheduling device based on a distributed storage system can be applied to various distributed storage clusters to perform storage resource scheduling on storage nodes in the distributed storage cluster. Figure 8 As shown, the resource scheduling device based on the distributed storage system includes: an acquisition module 81, which is used to obtain the request retry failure information generated by the user side in the distributed storage cluster when the request retry fails, and store the request retry failure information, wherein the request retry failure information is at least used to indicate the target storage resource where the request retry failure occurs; a scheduling module 82, which is used to, when it is monitored that the heartbeat loss of the target storage node in the distributed storage cluster occurs, schedule the target storage resource where the request retry failure occurs on the target storage node to other storage nodes in the distributed storage cluster that are in normal operating status according to the request retry failure information related to the target storage node.
[0074] By adopting the above technical solution, it is possible to obtain the request retry failure information generated by the user side in the distributed storage cluster when the request retry fails, and the request retry failure information is at least used to indicate the target storage resource where the request retry failure occurs. Then, when it is monitored that the heartbeat of the target storage node in the distributed storage cluster is lost, the target storage resource on the target storage node where the request retry failure occurs is scheduled to other storage nodes in the distributed storage cluster with normal operating status according to the request retry failure information related to the target storage node. Therefore, the abnormal storage node can be accurately identified according to the request retry failure information, and because only the target storage resource on the target storage node where the request retry fails occurs is scheduled when the heartbeat is lost, the IO jitter caused by the storage node failure can be reduced, and the batch storage resource mis-scheduling caused by network failures between the management node and the storage node in the related technology can be avoided.
[0075] Optionally, the scheduling module 82 schedules the target storage resource on the target storage node where the request retry failure occurs to other storage nodes in the distributed storage cluster that are in normal operation according to the request retry failure information related to the target storage node, including: Selecting a storage node in a normal operating state from other storage nodes in the distributed storage cluster except the target storage node; and From the selected storage nodes in normal operating status, select a first storage node whose schedulable capacity is greater than the number of target storage resources where the request retry failure occurs, and schedule the target storage resources where the request retry failure occurs on the target storage node to the first storage node.
[0076] Optionally, the scheduling module 82 schedules the target storage resource on the target storage node where the request retry fails to other storage nodes in the distributed storage cluster that are in normal operation, including: Obtaining the status of the distributed storage cluster; Based on the state of the distributed storage cluster, determining the number of storage nodes that allow for large-scale fault scheduling; If the number of storage nodes that allow for large-scale fault scheduling is greater than or equal to the number of target storage nodes where the request retry fails, the target storage resource on the target storage node where the request retry fails is scheduled to other storage nodes in the distributed storage cluster that are in normal operating status; If the number of storage nodes that allow for large-scale fault scheduling is less than the number of target storage nodes where the request retry fails, storage resource migration in and out of the target storage nodes is prohibited.
[0077] Optionally, the scheduling module 82 is also used to: when the number of target storage resources on the target storage node where the request retry fails is greater than or equal to a preset threshold, after the scheduling of the target storage resources on the target storage node where the request retry fails is completed, stop the service of the target storage node, and schedule other storage resources on the target storage node to other storage nodes in normal operating status.
[0078] Optionally, the scheduling module 82 is further used for: When it is detected that the target storage node has recovered its heartbeat, the storage resources with traffic in the probe cloud disk are scheduled to the target storage node, wherein the probe cloud disk is a cloud disk used to detect whether the target storage node can recover its service; When it is monitored that no request retry failure information for the storage resource with traffic is obtained within a period of time, a control operation is performed on the storage resource with traffic, wherein the control operation is used to control the storage resource with traffic; If the management and control operation is performed normally, the service of the target storage node is restored.
[0079] Optionally, the scheduling module 82 is further used for: When it is monitored that request retry failure information for the storage resource with traffic is obtained within the period of time, the storage resource with traffic in the probe cloud disk is scheduled to the target storage node again after a preset time interval, and after the number of re-scheduling reaches a preset number, the target storage node is set to a stopped running state and the re-scheduling is stopped.
[0080] The specific implementation of the operations performed by each module in the resource scheduling device according to the embodiment of the present disclosure has been described in detail in the relevant resource scheduling method, and will not be repeated here.
[0081] The present disclosure also provides a computer-readable medium having a computer program stored thereon, and when the computer program is executed by a processing device, the steps of any method described in the present disclosure are implemented.
[0082] The present disclosure also provides an electronic device, comprising: a storage device having a computer program stored thereon; A processing device is used to execute the computer program in the storage device to implement the steps of any method described in the present disclosure.
[0083] The present disclosure also provides a computer program product, comprising a computer program, which implements the steps of any one of the methods of the present disclosure when executed by a processor.
[0084] Reference below Fig. 9 , which shows a schematic diagram of the structure of an electronic device 600 suitable for implementing the embodiment of the present disclosure. The terminal device in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig. 9 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.
[0085] like Fig. 9 As shown, the electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.
[0086] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Although Fig. 9 The electronic device 600 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.
[0087] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.
[0088] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than computer readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.
[0089] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.
[0090] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.
[0091] The above-mentioned computer-readable medium carries one or more programs. When the above-mentioned one or more programs are executed by the electronic device, the electronic device: obtains the request retry failure information generated by the user side in the distributed storage cluster when the request retry fails, and stores the request retry failure information, wherein the request retry failure information is at least used to indicate the target storage resource where the request retry failure occurs; when it is monitored that the heartbeat of the target storage node in the distributed storage cluster is lost, according to the request retry failure information related to the target storage node, the target storage resource where the request retry failure occurs on the target storage node is scheduled to other storage nodes in the distributed storage cluster with normal operating status.
[0092] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).
[0093] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0094] The modules involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of the module does not limit the module itself in some cases. For example, the acquisition module may also be described as "a module for acquiring request retry failure information generated by the user side in the distributed storage cluster when the request retry fails and storing the request retry failure information".
[0095] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.
[0096] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0097] According to one or more embodiments of the present disclosure, Example 1 provides a resource scheduling method based on a distributed storage system, including: obtaining request retry failure information generated by a user side in a distributed storage cluster when a request retry fails, and storing the request retry failure information, wherein the request retry failure information is at least used to indicate a target storage resource where the request retry failure occurs; when it is monitored that a heartbeat loss occurs at a target storage node in the distributed storage cluster, scheduling the target storage resource where the request retry failure occurs on the target storage node to other storage nodes in the distributed storage cluster that are in normal operating status according to the request retry failure information related to the target storage node.
[0098] According to one or more embodiments of the present disclosure, Example 2 provides the method of Example 1, wherein, according to the request retry failure information related to the target storage node, the target storage resource where the request retry failure occurs on the target storage node is scheduled to other storage nodes in the distributed storage cluster that are in normal operation, including: Selecting a storage node in a normal operating state from other storage nodes in the distributed storage cluster except the target storage node; and From the selected storage nodes in normal operating status, select a first storage node whose schedulable capacity is greater than the number of target storage resources where the request retry failure occurs, and schedule the target storage resources where the request retry failure occurs on the target storage node to the first storage node.
[0099] According to one or more embodiments of the present disclosure, Example 3 provides the method of Example 1, wherein the step of scheduling the target storage resource on the target storage node where the request retry fails to other storage nodes in the distributed storage cluster that are in normal operation includes: Obtaining the status of the distributed storage cluster; Based on the state of the distributed storage cluster, determining the number of storage nodes that allow for large-scale fault scheduling; If the number of storage nodes that allow for large-scale fault scheduling is greater than or equal to the number of target storage nodes where the request retry fails, the target storage resource on the target storage node where the request retry fails is scheduled to other storage nodes in the distributed storage cluster that are in normal operating status; If the number of storage nodes that allow for large-scale fault scheduling is less than the number of target storage nodes where the request retry fails, storage resource migration in and out of the target storage nodes is prohibited.
[0100] According to one or more embodiments of the present disclosure, Example 4 provides the method of Example 1, wherein the method further includes: When the number of target storage resources on the target storage node where the request retry fails is greater than or equal to a preset threshold, after the scheduling of the target storage resources on the target storage node where the request retry fails is completed, the service of the target storage node is stopped, and other storage resources on the target storage node are scheduled to other storage nodes in normal operating status.
[0101] According to one or more embodiments of the present disclosure, Example 5 provides the method of any one of Examples 1 to 4, wherein the method further includes: When it is detected that the target storage node has recovered its heartbeat, the storage resources with traffic in the probe cloud disk are scheduled to the target storage node, wherein the probe cloud disk is a cloud disk used to detect whether the target storage node can recover its service; When it is monitored that no request retry failure information for the storage resource with traffic is obtained within a period of time, a control operation is performed on the storage resource with traffic, wherein the control operation is used to control the storage resource with traffic; If the management and control operation is performed normally, the service of the target storage node is restored.
[0102] According to one or more embodiments of the present disclosure, Example 6 provides the method of Example 5, wherein the method further includes: When it is monitored that request retry failure information for the storage resource with traffic is obtained within the period of time, the storage resource with traffic in the probe cloud disk is scheduled to the target storage node again after a preset time interval, and after the number of re-scheduling reaches a preset number, the target storage node is set to a stopped running state and the re-scheduling is stopped.
[0103] According to one or more embodiments of the present disclosure, Example 7 provides a resource scheduling device based on a distributed storage system, including: An acquisition module, used to acquire request retry failure information generated by a user side in a distributed storage cluster when a request retry fails, and store the request retry failure information, wherein the request retry failure information is at least used to indicate a target storage resource where the request retry failure occurs; The scheduling module is used to schedule the target storage resource where the request retry failure occurs on the target storage node to other storage nodes in the distributed storage cluster that are in normal operating status, based on the request retry failure information related to the target storage node, when it is monitored that the heartbeat of the target storage node in the distributed storage cluster is lost.
[0104] According to one or more embodiments of the present disclosure, Example 8 provides a computer-readable medium having a computer program stored thereon, which implements the steps of any of the methods described in Examples 1-6 when executed by a processing device.
[0105] According to one or more embodiments of the present disclosure, Example 9 provides an electronic device, including: a storage device having a computer program stored thereon; A processing device is used to execute the computer program in the storage device to implement the steps of any one of the methods described in Examples 1-6.
[0106] According to one or more embodiments of the present disclosure, Example 10 provides a computer program product, including a computer program, which implements the steps of any one of the methods of Examples 1 to 6 when executed by a processor.
[0107] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.
[0108] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.
[0109] Although the subject matter has been described in language specific to structural features and / or method logic actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims. Regarding the device in the above embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.
Claims
1. A resource scheduling method based on a distributed storage system, It is characterized in that include: Obtaining request retry failure information generated by a user side in a distributed storage cluster when a request retry fails, and storing the request retry failure information, wherein the request retry failure information is at least used to indicate a target storage resource where the request retry failure occurs; When it is monitored that the heartbeat of the target storage node in the distributed storage cluster is lost, the target storage resource on the target storage node where the request retry failure occurs is scheduled to other storage nodes in the distributed storage cluster that are in normal operating status based on the request retry failure information related to the target storage node.
2. The method according to claim 1, It is characterized in that The step of scheduling the target storage resource on the target storage node where the request retry failure occurs to other storage nodes in the distributed storage cluster that are in normal operation according to the request retry failure information related to the target storage node includes: Selecting a storage node in a normal operating state from other storage nodes in the distributed storage cluster except the target storage node; and From the selected storage nodes in normal operating status, select a first storage node whose schedulable capacity is greater than the number of target storage resources where the request retry failure occurs, and schedule the target storage resources where the request retry failure occurs on the target storage node to the first storage node.
3. The method according to claim 1, It is characterized in that The step of scheduling the target storage resource on the target storage node where the request retry failure occurs to other storage nodes in the distributed storage cluster that are in normal operation includes: Obtaining the status of the distributed storage cluster; Based on the state of the distributed storage cluster, determining the number of storage nodes that allow for large-scale fault scheduling; If the number of storage nodes that allow for large-scale fault scheduling is greater than or equal to the number of target storage nodes where the request retry fails, the target storage resource on the target storage node where the request retry fails is scheduled to other storage nodes in the distributed storage cluster that are in normal operating status; If the number of storage nodes that allow for large-scale fault scheduling is less than the number of target storage nodes where the request retry fails, storage resource migration in and out of the target storage nodes is prohibited.
4. The method according to claim 1, It is characterized in that The method further comprises: When the number of target storage resources on the target storage node where the request retry fails is greater than or equal to a preset threshold, after the scheduling of the target storage resources on the target storage node where the request retry fails is completed, the service of the target storage node is stopped, and other storage resources on the target storage node are scheduled to other storage nodes in normal operating status.
5. The method according to any one of claims 1 to 4, It is characterized in that The method further comprises: When it is detected that the target storage node has recovered its heartbeat, the storage resources with traffic in the probe cloud disk are scheduled to the target storage node, wherein the probe cloud disk is a cloud disk used to detect whether the target storage node can recover its service; When it is monitored that no request retry failure information for the storage resource with traffic is obtained within a period of time, a control operation is performed on the storage resource with traffic, wherein the control operation is used to control the storage resource with traffic; If the management and control operation is performed normally, the service of the target storage node is restored.
6. The method according to claim 5, It is characterized in that The method further comprises: When it is monitored that request retry failure information for the storage resource with traffic is obtained within the period of time, the storage resource with traffic in the probe cloud disk is scheduled to the target storage node again after a preset time interval, and after the number of re-scheduling reaches a preset number, the target storage node is set to a stopped running state and the re-scheduling is stopped.
7. A resource scheduling device based on a distributed storage system, It is characterized in that include: An acquisition module, used to acquire request retry failure information generated by a user side in a distributed storage cluster when a request retry fails, and store the request retry failure information, wherein the request retry failure information is at least used to indicate a target storage resource where the request retry failure occurs; The scheduling module is used to schedule the target storage resource where the request retry failure occurs on the target storage node to other storage nodes in the distributed storage cluster that are in normal operating status, based on the request retry failure information related to the target storage node, when it is monitored that the heartbeat of the target storage node in the distributed storage cluster is lost.
8. A computer readable medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processing device, the steps of the method according to any one of claims 1 to 6 are implemented.
9. An electronic device, It is characterized in that include: a storage device having a computer program stored thereon; A processing device, configured to execute the computer program in the storage device to implement the steps of the method according to any one of claims 1 to 6.
10. A computer program product comprising a computer program, It is characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.
Citation Information
Cited By
Storage volume processing method and device, electronic equipment and storage medium
CN119759264A
Resource scheduling method, apparatus, medium, electronic device, and program product
EP4703861A1
