Resource fragmentation consolidation method, controller, control node, and computer cluster

Abstract

The application provides a resource fragment consolidation method, a controller, a control node and a computer cluster, and relates to the technical field of data processing. The method simulates resource scheduling according to a first resource capacity and a first task currently being executed to obtain a plurality of candidate resource scheduling paths; determines a target resource scheduling path from the plurality of candidate resource scheduling paths; and schedules the first task based on the target resource scheduling path to perform resource fragment consolidation. After each candidate resource scheduling path is used to perform resource fragment consolidation, the residual resource capacity of at least one node can reach the first resource capacity. That is, the method can actively eliminate resource fragments in the computer cluster by scheduling the first task to perform resource fragment consolidation. Compared with the method of slowing down the generation of new resource fragments from the source, the scheduling reliability of a task with high resource demand can be effectively improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of data processing, and particularly relates to a resource fragment arrangement method, a controller, a control node and a computer cluster. BACKGROUND

[0002] Resource fragments (i.e. fragmented resources) refer to small-capacity resources provided by a computer cluster that are scattered and not fully utilized. The resource fragments are difficult to meet the resource requirements of a large new task, resulting in the large new task being unable to be scheduled. The large new task refers to a task that has rigid requirements for continuity and concentration of resources and cannot be executed by using scattered resources through cross-node splitting. For example, a task that requires 8 graphics processing units (GPUs) to run cannot be run on multiple nodes each of which has 2 to 3 idle GPUs.

[0003] Currently, when performing resource allocation, a new task is usually preferentially allocated to a node with less remaining resource capacity to be executed, so as to concentrate idle resources in the computer cluster to a few nodes that are not fully utilized, thereby slowing down the generation of new resource fragments. That is, the current method aims to slow down the generation of new resource fragments from the source to meet the resource requirements of a large new task as much as possible.

[0004] However, the reliability of this method of slowing down the generation of new resource fragments from the source to meet the resource requirements of a large new task is low. SUMMARY

[0005] The present application provides a resource fragment arrangement method, a controller, a control node and a computer cluster, which can solve the problem of low reliability of the method of slowing down the generation of new resource fragments from the source to meet the resource requirements of a large new task in the related art.

[0006] In a first aspect, the present application provides a resource fragment arrangement method, which is applied to a computer cluster including a plurality of nodes, and includes:

[0007] Simulating resource scheduling according to a first resource capacity and a first task that is currently being executed to obtain a plurality of candidate resource scheduling paths, wherein after each candidate resource scheduling path is used for resource fragment arrangement, the remaining resource capacity of at least one node can reach the first resource capacity;

[0008] Determining a target resource scheduling path from the plurality of candidate resource scheduling paths;

[0009] Scheduling the first task based on the target resource scheduling path to perform resource fragment arrangement.

[0010] Optionally, the simulating resource scheduling according to the first resource capacity and the first task currently being executed to obtain a plurality of candidate resource scheduling paths comprises:

[0011] obtaining a second resource capacity occupied by the first task currently being executed by each node in the plurality of nodes;

[0012] simulating resource scheduling based on the second resource capacity, the first resource capacity, and a characteristic parameter of the first task to obtain a plurality of candidate resource scheduling paths, wherein the characteristic parameter comprises: priority, interruptability, and recoverability of the first task.

[0013] Optionally, the simulating resource scheduling based on the second resource capacity, the first resource capacity, and a characteristic parameter of the first task to obtain a plurality of candidate resource scheduling paths comprises:

[0014] determining at least two target first tasks from the plurality of first tasks based on the characteristic parameter of the first task, wherein each target first task has a priority lower than a priority threshold, is interruptable, and is recoverable;

[0015] determining a scheduling mode of the target first task based on the second resource capacity occupied by each target first task, the first resource capacity, and an initial resource capacity of each node;

[0016] obtaining a plurality of candidate resource scheduling paths by arranging and combining the scheduling modes of all target first tasks.

[0017] Optionally, the obtaining a second resource capacity occupied by the first task currently being executed by each node in the plurality of nodes comprises:

[0018] taking a snapshot of the plurality of nodes to obtain the second resource capacity occupied by the first task currently being executed by each node in the plurality of nodes.

[0019] Optionally, the determining a target resource scheduling path from the plurality of candidate resource scheduling paths comprises:

[0020] obtaining a scheduling cost of each candidate resource scheduling path;

[0021] determining the candidate resource scheduling path with the minimum scheduling cost as the target resource scheduling path.

[0022] Optionally, each candidate resource scheduling path comprises: a destination node of the first task to be scheduled; and the obtaining a scheduling cost of each candidate resource scheduling path comprises:

[0023] For each of the candidate resource scheduling paths, obtain the number of the first tasks to be scheduled, the interruption sensitivity of the first tasks to be scheduled, the difference of the destination nodes of any two of the first tasks to be scheduled with the correlation degree greater than the degree threshold, and the scheduling cost of the first tasks to be scheduled.

[0024] Based on the number of the first tasks to be scheduled, the interruption sensitivity of the first tasks to be scheduled, the difference of the destination nodes of any two of the first tasks to be scheduled with the correlation degree greater than the degree threshold, and the scheduling cost of the first tasks to be scheduled, obtain the scheduling cost of the candidate resource scheduling path.

[0025] The scheduling cost of the candidate resource scheduling path is positively correlated with the number of the first tasks to be scheduled, the interruption sensitivity of the first tasks to be scheduled, and the scheduling cost of the first tasks, and the scheduling cost of the candidate resource scheduling path is lower in the case that the destination nodes of any two of the first tasks to be scheduled with the correlation degree greater than the degree threshold are the same than in the case that the destination nodes are different.

[0026] Optionally, the target resource scheduling path indicates that a first task in a first node is scheduled from the first node to a second node in the plurality of nodes, so that the remaining resource capacity of the first node reaches the first resource capacity; before the first task is scheduled based on the target resource scheduling path, the method further comprises:

[0027] locking resources of the first resource capacity in the first node and resources of a second resource capacity in the second node based on the target resource scheduling path;

[0028] The second resource capacity is the resource capacity occupied by the first task in the first node.

[0029] Optionally, locking resources of the first resource capacity in the first node and resources of a second resource capacity in the second node based on the target resource scheduling path comprises:

[0030] creating a first placeholder instance of the first resource capacity in the first node to lock the resources of the first resource capacity in the first node based on the target resource scheduling path;

[0031] and creating a second placeholder instance in the second node to lock the resources of the second resource capacity in the second node.

[0032] Optionally, the locking, based on the target resource scheduling path, the resource of the first resource capacity in the first node and the resource of the second resource capacity in the second node, comprises:

[0033] In a case where it is determined that there is a second task to be executed, the locking, based on the target resource scheduling path, the resource of the first resource capacity in the first node and the resource of the second resource capacity in the second node;

[0034] The resource capacity required by the second task is less than or equal to the first resource capacity.

[0035] Optionally, the number of the first tasks is a plurality, and each of the first tasks is executed in a node; after the scheduling of the first tasks based on the target resource scheduling path, the method further comprises:

[0036] If at least one of the first tasks is not successfully scheduled, the successfully scheduled first task is scheduled to a source node of the first task, and the resource occupied by the first task in a destination node of the first task is released.

[0037] Optionally, the simulating of the resource scheduling according to the first resource capacity and the currently executed first task comprises:

[0038] In a case where a resource fragmentation consolidation condition is met, the simulating of the resource scheduling according to the first resource capacity and the currently executed first task is performed.

[0039] The consolidation condition comprises that a waiting time length of a second task to be executed is greater than a first time threshold, and / or a dispersion degree of resources in the computer cluster is greater than a degree threshold.

[0040] Optionally, the consolidation condition comprises that the dispersion degree of resources in the computer cluster is greater than the degree threshold; before the simulating of the resource scheduling according to the first resource capacity and the currently executed first task, the method further comprises:

[0041] Obtaining a proportion of unavailable resources of the computer cluster;

[0042] If it is determined that the proportion of the unavailable resources is greater than a proportion threshold, it is determined that the dispersion degree of resources in the computer cluster is greater than the degree threshold.

[0043] Optionally, the method further comprises:

[0044] If a meeting time length of the consolidation condition reaches a second time threshold, the resource consolidation is not completed, and / or if an obtaining time length of the candidate resource scheduling path is greater than a third time threshold, the resource fragmentation consolidation is ended.

[0045] In a second aspect, the present application provides a resource fragmentation consolidation device, comprising:

[0046] a simulation module, configured to simulate resource scheduling according to a first resource capacity and a first task currently being executed by each of the plurality of nodes, to obtain a plurality of candidate resource scheduling paths, wherein each of the candidate resource scheduling paths is capable of making a residual resource capacity of at least one of the nodes reach the first resource capacity after resource fragmentation consolidation is performed by using the candidate resource scheduling path;

[0047] a determination module, configured to determine a target resource scheduling path from the plurality of candidate resource scheduling paths;

[0048] a scheduling module, configured to schedule the first task based on the target resource scheduling path, to perform resource fragmentation consolidation.

[0049] Optionally, the simulation module is specifically configured to obtain a second resource capacity occupied by the first task currently being executed by each of the plurality of nodes; simulate resource scheduling based on the second resource capacity, the first resource capacity, and a characteristic parameter of the first task, to obtain the plurality of candidate resource scheduling paths, wherein the characteristic parameter comprises a priority, interruptability, and recoverability of the first task.

[0050] Optionally, the simulation module is specifically configured to determine at least two target first tasks from the plurality of first tasks based on the characteristic parameter of the first task, wherein a priority of each of the target first tasks is lower than a priority threshold, each of the target first tasks is interruptable and recoverable; determine a scheduling mode of each of the target first tasks based on a second resource capacity occupied by each of the target first tasks, the first resource capacity, and an initial resource capacity of each of the nodes; and arrange and combine the scheduling modes of all the target first tasks, to obtain the plurality of candidate resource scheduling paths.

[0051] Optionally, the simulation module is further configured to take a snapshot of the plurality of nodes, to obtain the second resource capacity occupied by the first task currently being executed by each of the plurality of nodes.

[0052] Optionally, the determination module is specifically configured to obtain a scheduling cost of each of the candidate resource scheduling paths; and determine the candidate resource scheduling path with the minimum scheduling cost as the target resource scheduling path.

[0053] Optionally, the determining module is specifically configured to: for each candidate resource scheduling path, acquire the number of the first tasks to be scheduled, the interruption sensitivity of the first tasks to be scheduled, the difference of the destination nodes of any two first tasks to be scheduled with the correlation degree greater than the degree threshold, and the scheduling cost of the first tasks to be scheduled; and based on the number of the first tasks to be scheduled, the interruption sensitivity of the first tasks to be scheduled, the difference of the destination nodes of any two first tasks to be scheduled with the correlation degree greater than the degree threshold, and the scheduling cost of the first tasks to be scheduled, acquire the scheduling cost of the candidate resource scheduling path; wherein the scheduling cost of the candidate resource scheduling path is positively correlated with the number of the first tasks to be scheduled, the interruption sensitivity of the first tasks to be scheduled, and the scheduling cost of the first tasks, and the scheduling cost of the candidate resource scheduling path is lower in the case that the destination nodes of the two first tasks to be scheduled are the same than in the case that the destination nodes of the two first tasks to be scheduled are different.

[0054] Optionally, the target resource scheduling path indicates that a first task in a first node is scheduled from the first node to a second node in the plurality of nodes, so that the remaining resource capacity of the first node reaches the first resource capacity; and the scheduling module is further configured to: based on the target resource scheduling path, lock the resources of the first resource capacity in the first node, and lock the resources of a second resource capacity in the second node; wherein the second resource capacity is the resource capacity occupied by the first task in the first node.

[0055] Optionally, the scheduling module is specifically configured to: based on the target resource scheduling path, create a first placeholder instance of the first resource capacity in the first node, so as to lock the resources of the first resource capacity in the first node; and create a second placeholder instance in the second node, so as to lock the resources of the second resource capacity in the second node.

[0056] Optionally, the scheduling module is further configured to: in a case that it is determined that there is a second task to be executed, based on the target resource scheduling path, lock the resources of the first resource capacity in the first node, and lock the resources of a second resource capacity in the second node; wherein the resource capacity required by the second task is less than or equal to the first resource capacity.

[0057] Optionally, the number of the first tasks is a plurality, and each first task runs in a node; and the scheduling module is further configured to: if at least one first task fails to be scheduled successfully, schedule the first tasks that have been successfully scheduled to the source node of the first tasks, and release the resources occupied by the first tasks in the destination nodes of the first tasks.

[0058] Optionally, the scheduling module is further configured to simulate resource scheduling according to the first resource capacity and the first task currently being executed, if a resource fragmentation consolidation condition is met; the consolidation condition includes that a waiting time length of a second task to be executed is greater than a first time threshold, and / or a dispersion degree of resources in the computer cluster is greater than a degree threshold.

[0059] Optionally, the consolidation condition includes that the dispersion degree of resources in the computer cluster is greater than the degree threshold; the scheduling module is specifically configured to obtain a proportion of unavailable resources of the computer cluster; if it is determined that the proportion of unavailable resources is greater than a proportion threshold, it is determined that the dispersion degree of resources in the computer cluster is greater than the degree threshold.

[0060] Optionally, the scheduling module is specifically configured to, if a meeting time length of the consolidation condition reaches a second time threshold, resource consolidation has not been completed, and / or if an obtaining time length of the candidate resource scheduling path is greater than a third time threshold, end resource fragmentation consolidation.

[0061] In a third aspect, the present application provides a controller, comprising a processor and a memory, the memory stores programs or instructions executable on the processor, and the programs or instructions are executed by the processor to implement the steps of the resource fragmentation consolidation method according to any one of the first aspect.

[0062] In a fourth aspect, the present application provides a control node, comprising the controller according to the third aspect.

[0063] In a fifth aspect, the present application provides a computer cluster, comprising the control node according to the fourth aspect.

[0064] In a sixth aspect, the present application provides a computer readable storage medium, the readable storage medium stores programs or instructions, and the programs or instructions are executed by a processor to implement the steps of the resource fragmentation consolidation method according to any one of the first aspect.

[0065] In a seventh aspect, the present application provides a computer program product, the program product is executed by a processor of a vehicle or a cloud server to implement the steps of the resource fragmentation consolidation method according to any one of the first aspect.

[0066] This application provides a resource fragmentation defragmentation method, controller, control node, and computer cluster. The method simulates resource scheduling based on a first resource capacity and a currently executing first task, obtaining multiple candidate resource scheduling paths. From these candidate paths, a target resource scheduling path is determined. Based on the target resource scheduling path, the first task is scheduled for resource fragmentation defragmentation. After defragmentation using each candidate resource scheduling path, at least one node's remaining resource capacity reaches the first resource capacity. In other words, this method can proactively eliminate resource fragmentation in the computer cluster by scheduling the first task for resource fragmentation defragmentation, effectively consolidating scattered resource fragments into contiguous large blocks of resources. Compared to mitigating the generation of new resource fragments at the source, the method provided in this application effectively avoids situations where excessive resource fragmentation prevents task scheduling, thereby significantly improving the scheduling reliability of tasks with high resource demands. Attached Figure Description

[0067] Figure 1 A flowchart illustrating a resource fragmentation defragmentation method provided in an embodiment of this application;

[0068] Figure 2 A flowchart illustrating another resource fragmentation defragmentation method provided in this application embodiment;

[0069] Figure 3 A flowchart illustrating a method for obtaining dispersion degree provided in an embodiment of this application;

[0070] Figure 4 A flowchart illustrating another resource fragmentation defragmentation method provided in this application embodiment;

[0071] Figure 5 A flowchart illustrating yet another resource fragmentation defragmentation method provided in this application embodiment;

[0072] Figure 6 A flowchart illustrating yet another resource fragmentation defragmentation method provided in this application embodiment;

[0073] Figure 7 This is a schematic diagram of the structure of a resource fragmentation sorting device provided in an embodiment of this application. Detailed Implementation

[0074] The technical solutions in the embodiments of this application will be clearly described below with reference to the accompanying drawings. All other embodiments obtained by those skilled in the art based on the embodiments in this application are within the scope of protection of this application.

[0075] The terms "first", "second", and the like in this application are used to distinguish similar objects, and are not used to describe a particular order or sequence. It should be understood that the terms used in this way can be interchanged, so that the embodiments of the present application can be implemented in an order other than that illustrated or described herein, and the objects distinguished by "first", "second" are generally a class, not limited to the number of objects.

[0076] The following embodiments of the present application can be applied to a computer cluster, such as a control node in the computer cluster. For example, it can be executed by a controller in the control node. Specifically, the controller can be installed with a scheduler, and the following embodiments can be executed by the scheduler. The following describes embodiments of the resource fragmentation consolidation method in several specific embodiments, Figure 1 A flowchart of a resource fragmentation consolidation method provided by an embodiment of the present application is shown in FIG. 1. Figure 1

[0077] S11: Simulate resource scheduling according to the first resource capacity and the first task currently being executed, to obtain a plurality of candidate resource scheduling paths.

[0078] Wherein, after resource fragmentation consolidation is performed using each candidate resource scheduling path, the remaining resource capacity of at least one node reaches the first resource capacity, i.e., is greater than or equal to the first resource capacity. The first resource capacity can be pre-stored by the control node. Alternatively, the first resource capacity can be the capacity of the resources required by the second task to be executed received by the computer cluster. Resource fragmentation consolidation refers to: by scheduling (i.e., migrating) the first task currently being executed by a node to another node, the resources occupied by the first task in the node are released, so that the fragmented resources released can be aggregated into large continuous resources.

[0079] The candidate resource scheduling path refers to a resource reallocation mode, which aims to schedule the first task currently running on a node to another node, so that the remaining resource capacity of the node reaches the first resource capacity. As can be seen, the method provided by the embodiments of the present application can, without adding resources, enable at least one node to execute a task of the first resource capacity by resource relocation.

[0080] S12: Determine a target resource scheduling path from the plurality of candidate resource scheduling paths.

[0081] The control node can calculate the scheduling cost of the plurality of candidate resource scheduling paths, and determine the candidate resource scheduling path with the minimum scheduling cost as the target resource scheduling path.

[0082] S13: Schedule the first task based on the target resource scheduling path to perform resource fragmentation consolidation.

[0083] ​The control node schedules the first task to concentrate originally multiple scattered resource fragments, to obtain a first resource capacity that is continuously available, and to perform resource fragment consolidation.

[0084] In this embodiment, the method simulates resource scheduling according to the first resource capacity and the first task that is currently being executed, to obtain multiple candidate resource scheduling paths; determines a target resource scheduling path from the multiple candidate resource scheduling paths; and schedules the first task based on the target resource scheduling path, to perform resource fragment consolidation. After each candidate resource scheduling path is used to perform resource fragment consolidation, the remaining resource capacity of at least one node can reach the first resource capacity. That is, the method can actively eliminate resource fragments in the computer cluster by scheduling the first task to perform resource fragment consolidation, so that the scattered resource fragments in the computer cluster can be effectively integrated into a large block of continuous resources. Compared with the method of slowing down the generation of new resource fragments from the source, the method provided in this application can effectively avoid the situation that the task cannot be scheduled due to too many resource fragments, so as to effectively improve the scheduling reliability of the task with high resource demand.

[0085] Figure 2 Another resource fragment consolidation method provided in this embodiment is shown in the flowchart. Figure 2 As shown in the figure, Figure 2 On the basis of the embodiment shown in the figure, Figure 1 A possible implementation of S11 is shown as follows:

[0086] S111: Obtain the second resource capacity occupied by the first task currently being executed by each node in the multiple nodes.

[0087] Specifically, the control node takes a snapshot of the multiple nodes to obtain the second resource capacity occupied by the first task currently being executed by each node in the multiple nodes. The snapshot refers to a momentary record of the current state of the nodes in the computer cluster. The current state includes: the first task running in the node, the capacity of the resource occupied by each first task, and the initial resource capacity of the node. The initial resource capacity is the maximum resource capacity of the node.

[0088] Optionally, the number of first tasks is multiple, and each first task runs in a node.

[0089] S112: Simulate resource scheduling based on the second resource capacity, the first resource capacity, and the characteristic parameters of the first task, to obtain multiple candidate resource scheduling paths.

[0090] Each candidate resource scheduling path includes: a first task to be scheduled, and a destination node of the first task. The destination node is the node that executes the first task after the first task is scheduled, and is the node where the first task finally runs.

[0091] In the embodiments of the present application, the implementation process of S112 can include:

[0092] Step A1, determining at least two target first tasks from the plurality of first tasks based on a feature parameter of the first tasks.

[0093] The feature parameter includes priority, interruptability, and recoverability. The interruptability is used to indicate whether the first task can be interrupted, and the recoverability is used to indicate whether the first task can be recovered, such as self-recovery (i.e., automatic recovery).

[0094] The control node can determine the first task with a priority less than a priority threshold (i.e., lower priority), interruptability, and recoverability as a target first task from the plurality of first tasks. That is, the control node can filter a batch of first tasks with low priority, interruptability, and support for automatic recovery from the computer cluster as potential to-be-scheduled tasks. The priority threshold can be pre-stored by the control node.

[0095] Optionally, the interruptability and recoverability of each first task can be determined by the control node through simulation of scheduling the first task.

[0096] Step A2, determining a scheduling mode of the target first task based on the second resource capacity occupied by each target first task, the first resource capacity, and the initial resource capacity of each node.

[0097] The condition for the target first task to run on a node includes that the remaining resource capacity of the node is greater than or equal to the second resource capacity occupied by the target first task before the target first task is migrated. The remaining resource capacity of the node is the difference between the initial resource capacity and a sum of the second resource capacities occupied by all currently running first tasks in the node.

[0098] Based on this, for each target first task, the control node can select a destination node from the plurality of nodes based on the size relationship between the remaining resource capacity of each node and the second resource capacity occupied by the target first task. The remaining resource capacity of the destination node is greater than or equal to the second resource capacity, that is, the destination node is a schedulable destination node of the target first task. Then, the control node can determine a scheduling mode of the target first task as: scheduling the target first task from a source node where the target first task currently locates to a destination node. Any scheduling mode of the target first task is used to indicate a destination node of the target first task.

[0099] Step A3, performing permutation and combination on the scheduling modes of all target first tasks to obtain a plurality of candidate resource scheduling paths.

[0100] For the scenario that there are multiple scheduling manners for the target first task, the control node can arrange and combine the scheduling manners of all target first tasks to obtain multiple candidate resource scheduling paths. For example, the control node can arrange and combine the scheduling manners of all target first tasks to obtain multiple initial resource scheduling paths. Then, for each initial resource scheduling path, if the control node determines that the target first tasks included in the initial resource scheduling path do not have resource conflicts, the control node determines the initial resource scheduling path as a candidate resource scheduling path.

[0101] Resource conflicts refer to that, among all target first tasks included in a certain initial resource scheduling path, the sum of the second resource capacities of multiple target first tasks with the same destination node is greater than the remaining resource capacity of the destination node, i.e., the total demand of the multiple target first tasks for resources exceeds the remaining resource capacity that can be currently provided by the destination node. For example, in a certain initial resource scheduling path, target first task A and target first task B need to be scheduled to node 1, and the sum of the second resource capacities of target first task A and target first task B is greater than the remaining resource capacity of node 1. Therefore, target first task A and target first task B have resource conflicts.

[0102] The following takes a specific embodiment as an example to illustrate a scheme in which the control node simulates resource scheduling based on the resource capacities occupied by 2 target first tasks, the first resource capacity, and the remaining resource capacities of nodes to obtain multiple candidate resource scheduling paths.

[0103] For example, in a computer cluster, there are node 1, node 2, and node 3, and the initial resource capacities of the three nodes are all 8 GPUs. Node 1 runs first task A that occupies 2 GPUs and first task B that occupies 1 GPU, and the remaining resource capacity of node 1 is 5 GPUs. Node 2 runs first task C that occupies 2 GPUs, first task D that occupies 1 GPU, and first task E that occupies 1 GPU, and the remaining resource capacity of node 2 is 4 GPUs. Node 3 runs first task F that occupies 1 GPU and first task G that occupies 4 GPUs, and the remaining resource capacity of node 1 is 3 GPUs.

[0104] Suppose the target first task is first task A and first task B, and the first resource capacity is 8 GPUs, the scheduling mode of the first task A can include: scheduling to node 2 or scheduling to node 3, that is, migrating the first task A to node 2 or node 3. The scheduling mode of the first task B can include: scheduling to node 2 or scheduling to node 3. The control node can arrange and combine the first task A and the first task B and the scheduling mode to obtain 4 candidate resource scheduling paths. The 4 candidate resource scheduling paths include: candidate resource scheduling path 1: the first task A is scheduled to node 2, and the first task B is scheduled to node 2; candidate resource scheduling path 2: the first task A is scheduled to node 2, and the first task B is scheduled to node 3; candidate resource scheduling path 3: the first task A is scheduled to node 3, and the first task B is scheduled to node 2; and candidate resource scheduling path 4: the first task A is scheduled to node 3, and the first task B is scheduled to node 3.

[0105] In the embodiment, the second resource capacity occupied by each first task currently being executed in each node in the plurality of nodes is obtained; based on the second resource capacity occupied by the first task, the first resource capacity, and the characteristic parameter of the first task, a plurality of candidate resource scheduling paths are obtained. Since the characteristic parameter of the first task is considered, the running failure of the first task caused by blindly scheduling the non-interruptible first task is avoided.

[0106] In some embodiments, the control node periodically performs resource fragmentation consolidation. For example, the control node can pre-store a consolidation period of resource fragmentation, which can be 12 hours or 24 hours.

[0107] In some embodiments, when the consolidation condition of resource fragmentation is met, the control node simulates resource scheduling according to the first resource capacity and the first task currently being executed to obtain a plurality of candidate resource scheduling paths for resource fragmentation consolidation. The consolidation condition can include that the computer cluster receives a second task to be executed, and the waiting time of the second task is greater than a first time threshold, and / or the dispersion degree of the resources in the computer cluster is greater than a degree threshold. The first time threshold and the degree threshold are pre-stored by the control node. As can be seen, the control node can timely perform resource fragmentation consolidation after accurately identifying the scheduling bottleneck caused by fragmented resources.

[0108] The embodiments of the present application trigger resource fragmentation consolidation in a plurality of ways periodically and / or when the consolidation condition of resource fragmentation is met, which can realize active consolidation of resource fragmentation and ensure timely consolidation of resource fragmentation. Moreover, since a plurality of triggering conditions are supported, the flexibility and intelligence of the consolidation are improved.

[0109] For the case that the arrangement condition comprises: the waiting duration of the second task to be executed is greater than the first duration threshold, the control node can start timing from the receiving time of the second task, and determine the waiting duration of the second task.

[0110] For the case that the arrangement condition comprises: the dispersion degree is greater than the degree threshold, as shown in Figure 3 , the process of the control node obtaining the dispersion degree of the resources in the computer cluster can comprise:

[0111] S31: Obtain the proportion of unavailable resources of the computer cluster.

[0112] The proportion of unavailable resources (which can also be referred to as the resource fragmentation rate) refers to: the proportion of idle resources that cannot be effectively utilized in the total idle resources. The idle resources that cannot be effectively utilized are the sum of the resource fragments in the computer cluster.

[0113] Assuming that the resources are GPUs, the proportion of unavailable resources F can satisfy the following formula (1):

[0114] (1)

[0115] In formula (1), N is the total number of multiple tasks, and the number of GPUs occupied by any two tasks is different. R i denotes the actual utilization rate of various tasks. R i is the actual executable quantity, which is the number of tasks T i that can actually be run after considering the distribution of idle GPUs on each node. That is, R i is the number of tasks T i that can be run by each node of the computer cluster using idle GPUs. T

[0116] E i denotes the theoretical maximum executable quantity, which is the number of tasks T i determined without considering the distribution of idle GPUs on the node. That is, E i is the number of tasks T i that can be run by all idle A GPUs. E i can be obtained by dividing A by T i and taking the integer part, that is, . If E i is 0, then in the calculation of the proportion of unavailable resources, E i and R i are both taken as 1.

[0117] For example, if there are 4 idle GPUs, but they are scattered in 4 different nodes, each node has 1 idle GPU, then the R of the "task T2 requiring 2 GPUs" = 0 (because no node can have 2 idle GPUs), and E = 2. i i = 2.

[0118] S32: If it is determined that the proportion of unavailable resources is greater than the proportion threshold, it is determined that the dispersion degree of resources in the computer cluster is greater than the degree threshold.

[0119] Since the dispersion degree of resources is positively correlated with the proportion of unavailable resources, the control node can determine that the dispersion degree of resources in the computer cluster is greater than the degree threshold when it is determined that the proportion of unavailable resources is greater than the proportion threshold. The proportion threshold is pre-stored by the control node.

[0120] In the embodiments of the present application, the control node determines that the time length used for obtaining the candidate resource scheduling path is greater than the third time length threshold, and ends the current operation. The control node starts timing from the beginning of simulating resource scheduling, and determines that the time length used is greater than the third time length threshold, and the candidate resource scheduling path has not been obtained, then the obtaining of the candidate resource scheduling path is ended, that is, the simulation of resource scheduling is ended. The third time length threshold is pre-stored by the computer cluster.

[0121] In this way, it is avoided that a new task to be executed in the computer cluster, while the cluster spends a long time in obtaining the candidate resource scheduling path, so that the task to be executed cannot be added to the task queue of the simulation scheduling, resulting in the situation that the task to be executed is waiting for a long time.

[0122] Figure 4 Another flowchart of a resource fragmentation consolidation method provided by the embodiments of the present application is shown. As shown in Figure 4 , the method is based on the embodiment shown in Figure 4 , and a possible implementation of S12 is shown as follows: Figure 1

[0123] S121: Obtain the scheduling cost of each candidate resource scheduling path.

[0124] ​​For each candidate resource scheduling path, the control node can obtain the number of the to-be-scheduled first tasks (i.e., the target first task in the foregoing) included in the candidate resource scheduling path, the interruption sensitivity of the to-be-scheduled first tasks, the similarity of the destination nodes of any two to-be-scheduled first tasks with the correlation degree greater than the degree threshold, and the scheduling cost of the to-be-scheduled first tasks. Then, the control node can obtain the scheduling cost of each candidate resource scheduling path based on the number of the to-be-scheduled first tasks, the interruption sensitivity of the to-be-scheduled first tasks, the similarity of the destination nodes of any two to-be-scheduled first tasks with the correlation degree greater than the degree threshold, and the scheduling cost of the to-be-scheduled first tasks.

[0125] The scheduling cost of the candidate resource scheduling path is positively correlated with the number of the to-be-scheduled first tasks, the interruption sensitivity of the to-be-scheduled first tasks, and the scheduling cost of the to-be-scheduled first tasks. In addition, compared with the case where the destination nodes of two to-be-scheduled first tasks with the correlation degree greater than the degree threshold are different, the scheduling cost of the candidate resource scheduling path is lower in the case where the destination nodes of the two to-be-scheduled first tasks are the same.

[0126] That is, assuming that the candidate resource scheduling path includes two to-be-scheduled first tasks with a high correlation degree, the scheduling cost of the candidate resource scheduling path is lower in the case where the destination nodes of the two first tasks are the same, while the number of the to-be-scheduled first tasks included in the candidate resource scheduling path, the interruption sensitivity of each to-be-scheduled first task, and the scheduling cost remain unchanged.

[0127] The interruption sensitivity of a task refers to the impact of interruption of the task on the task itself during the running of the task. For example, a high-sensitivity task refers to a task whose interruption can cause data loss and has a large interruption cost. A low-sensitivity task refers to a task that can be recovered by restarting the task after interruption and has a small interruption cost.

[0128] The correlation degree between tasks refers to the degree of mutual dependence between tasks, which can also be referred to as resource affinity. For example, the correlation degree between two subtasks of a task is high, and the correlation degree between different tasks is low.

[0129] The scheduling cost of a task is positively correlated with the data local dependence degree of the task. That is, the higher the data local dependence degree of a to-be-scheduled first task, the higher the scheduling cost of the to-be-scheduled first task. The data local dependence degree refers to the dependence of the running of a task on local data of a node. For example, if the execution of a to-be-scheduled first task does not depend on local data, such as data used by the first task during running can be obtained through data sharing. Then, the data local dependence degree of the to-be-scheduled first task is low, and the scheduling cost of the to-be-scheduled first task is low.

[0130] Therefore, the method considers comprehensive parameters to determine the migration cost of the candidate resource scheduling path, and thus can ensure high accuracy of the determined migration cost.

[0131] It can be understood that the control node can obtain the interruption sensitivity of the first task and the scheduling cost of the first task by simulating scheduling of the first task. When the computer cluster receives a task, a configuration file of the task is also received synchronously. The static configuration file includes a task that is highly associated with the task.

[0132] Optionally, the control node pre-stores a scheduling cost function, and the control node can determine the scheduling cost of the candidate resource scheduling path based on the scheduling cost function and the number of the first task, the interruption sensitivity, the similarity and difference of the destination nodes, and the scheduling cost of the first task.

[0133] S122: determining the candidate resource scheduling path with the minimum scheduling cost as the target resource scheduling path.

[0134] After the control node obtains the scheduling cost of each candidate resource scheduling path, the control node determines the candidate resource scheduling path with the minimum scheduling cost as the target resource scheduling path.

[0135] Since the candidate resource scheduling path with the minimum migration cost is determined as the target resource scheduling path, it can be ensured that resource scheduling based on the target resource scheduling path not only reduces the impact of migration on the task, but also reduces the impact of resource reallocation on the computer cluster, thereby ensuring stable operation of the computer cluster on the premise of resource fragmentation consolidation.

[0136] Optionally, a virtualized scheduling environment can be constructed in the control node. The virtualized scheduling environment can reproduce the resource usage of the real computer cluster and synchronizes all the first tasks to be scheduled. That is, the virtualized scheduling environment is a virtual scheduling sandbox that is completely consistent with the real cluster environment of the computer cluster, and can be regarded as a virtual "shadow cluster" of the computer cluster. The control node can simulate resource scheduling in the virtualized scheduling environment to obtain multiple candidate resource scheduling paths, and obtain the scheduling cost of the multiple candidate resource scheduling paths by pre-acting the task scheduling, so as to determine the target resource scheduling path (i.e., the optimal scheduling scheme).

[0137] It can be seen that the method provided in the embodiments of the present application can simulate and predict the scheduling behavior in the virtual scheduling environment before migrating the task, and can find the optimal scheduling rearrangement scheme through iterative search and multi-dimensional cost function evaluation. Therefore, on the one hand, it can ensure that the target first task migrated from the source node has a corresponding destination node, avoiding the risk of task interruption caused by blind eviction; on the other hand, it ensures that all scheduling operations have feasibility verification and controllable cost; on the other hand, it can realize the "zero-cost trial and error" of the fragmentation consolidation operation, avoiding disturbing the operation of the computer cluster. In addition, since the entire process is carried out in the "shadow cluster", it will not cause any interference to the actual task.

[0138] Figure 5 The flowchart of another resource fragmentation consolidation method provided in the embodiments of the present application is shown in FIG. 6. As shown in FIG. 6, Figure 5 Figure 5 is based on the embodiment shown in Figure 1 Before S13 is executed, it can further include:

[0139] S51: Locking the resources of the first resource capacity in the first node and the resources of the second resource capacity in the second node based on the target resource scheduling path.

[0140] The target resource scheduling path indicates scheduling the first task to be scheduled in the first node from the first node to the second node in the plurality of nodes, so that the remaining resource capacity of the first node reaches the first resource capacity. The first node is the source node of the first task to be scheduled included in the target resource scheduling path, and the second node is the destination node of the first task.

[0141] In some embodiments, the control node can lock the resources of the first resource capacity in the first node and the resources of the second resource capacity in the second node based on the target resource scheduling path in the case where it is determined that there is a second task to be executed. In this way, the utilization rate of the resources in the computer cluster can be ensured to be high.

[0142] In some embodiments, the control node can create a first placeholder instance of the first resource capacity in the first node to lock the resources of the first resource capacity in the first node based on the target resource scheduling path; and create a second placeholder instance in the second node to lock the resources of the second resource capacity in the second node.

[0143] ​The first (or second) placeholder instance is an empty task, i.e., the first (or second) placeholder instance itself does not carry any workload, thereby not affecting the node load. After the first (or second) placeholder instance is created, it means that the corresponding resource capacity on the first (or second) node is "occupied". That is, the placeholder instance has high priority and exclusivity, which can ensure that other scheduling requests cannot be scheduled on the resources occupied by the placeholder instance.

[0144] The identifier of the first placeholder instance is the same as the identifier of the second task to be executed, and the identifier of the second placeholder instance is the same as the identifier of the first task to be scheduled. The identifier of each of the first and second placeholder instances is used to uniquely identify the placeholder instance in the plurality of placeholder instances. The identifier of the first task is used to uniquely identify the first task in the plurality of first tasks. For example, the identifier of the first task can be a number.

[0145] It can be understood that, in the process of scheduling the first task to the second node, the control node can identify the second placeholder instance with the same identifier as the first task based on the identifier, and allocate the resources occupied by the second placeholder instance to the first task. In this way, the first task can be successfully scheduled to run on the second node. After the first task is scheduled to the second node, the control node can clear the second placeholder instance.

[0146] In this embodiment, after the expected resources (i.e., the resources of the first resource capacity in the first node and the resources of the second resource capacity in the second node) are obtained through the simulation scheduling, the expected resources are atomically locked by the placeholder instances, which can effectively ensure that the expected resources are not preempted by other tasks during task scheduling. That is, the resources of the first resource capacity can only be used by the second task, and the resources of the second resource capacity can only be used by the scheduled first task. In this way, on the one hand, the consistency between the scheduling result after the resource scheduling according to the target resource scheduling path and the target resource scheduling path is ensured, the determinism, reproducibility and controllability of the scheduling result are guaranteed, and the reliability of the resource fragmentation consolidation is improved; on the other hand, the scheduling race and resource backfill failure are avoided, so that the situation that the second task and the scheduled first task cannot be executed can be avoided, i.e., the situation of invalid scheduling can be avoided.

[0147] After the resources of the first resource capacity and the resources of the second resource capacity are locked, one possible implementation of S13 is as follows:

[0148] The control node can release the resource occupied by the first task in the first node. Meanwhile, the control node can create a copy task corresponding to the first task to be scheduled in the second node. The copy task is completely consistent with the first task, which can be understood as the copy task is obtained by copying the first task. The copy task can automatically identify the resource locked on the second node for running the first task, and run based on the resource. Thus, the consistency of resource utilization and scheduling path can be ensured, thereby realizing accurate scheduling of the first task from the first node to the second node.

[0149] Figure 6 A flowchart of another resource fragmentation arrangement method provided by an embodiment of the present application is shown in FIG. 6. As shown in FIG. 6, Figure 6 Figure 6 Figure 1 Based on the embodiment shown in FIG. 6, after performing S13, the method can further include:

[0150] S61: If at least one first task is not successfully scheduled, the first task successfully scheduled is scheduled to the source node of the first task, and the resource occupied by the first task in the destination node of the first task is released.

[0151] The control node can detect whether there is a first task not successfully scheduled in the first tasks to be scheduled. In the case that at least one first task is not successfully scheduled, the control node can schedule the first task successfully scheduled to the source node of the first task, and release the resource occupied by the first task in the destination node of the first task.

[0152] That is, once a first task to be scheduled fails to be successfully scheduled due to abnormal reasons, the control node can automatically cancel the scheduling of other first tasks successfully scheduled, so as to restore the system state of the computer cluster to the system state before the resource fragmentation arrangement. In this way, on the one hand, the frequency of manual intervention can be reduced, and unattended scheduling optimization can be supported; on the other hand, closed-loop control and self-healing of the resource fragmentation arrangement can be realized, so that the computer cluster resources can be prevented from being in a locked state for a long time due to partial task blocking, and the smooth running of the computer cluster can be affected, thereby the continuity of the task can be effectively guaranteed. This cancellation mechanism is particularly important in a multi-tenant complex cluster environment.

[0153] In some embodiments, for each first task to be scheduled in the target resource scheduling path, the control node can obtain a scheduling result of the first task. The scheduling result is scheduling success or scheduling failure.

[0154] In some embodiments, after the control node schedules the first task successfully scheduled (i.e., the first task) to the node before scheduling (i.e., the source node), the control node can also clear the second placeholder instance corresponding to the first task. ​​

[0155] Optionally, if the control node determines that the duration of satisfaction of the arrangement condition reaches a second duration threshold and the resource arrangement is not completed, the resource fragmentation arrangement can be ended. In this way, it can be avoided that new tasks cannot be scheduled due to excessive occupation of computing resources in the resource fragmentation arrangement process, thereby ensuring the continuity of task execution.

[0156] Optionally, the method provided by the embodiments of the present application has high pluggability and configurability. Specifically, the user can specify which queues in the cluster support fragmentation arrangement, which task types (such as recoverable task types) are allowed to be migrated, and the simulation scheduling period through the plug-in parameters of the scheduler in the control node. In addition, the scheduling cost function, the second duration threshold (which can also be referred to as the pre-play window size), and the maximum number of evictable tasks can be flexibly adjusted according to the task scenario, and have good scalability. Different task scenarios include different types of tasks.

[0157] Resource fragmentation refers to scattered and underutilized small-capacity resources provided by a computer cluster. The small-capacity resources are difficult to meet the resource requirements of large new tasks, resulting in the large new tasks being unable to be executed. The large new task refers to a task that has rigid requirements for continuity and concentration of resources and cannot be executed by using scattered resources through cross-node splitting.

[0158] Currently, when allocating resources, new tasks are preferentially allocated to nodes with less remaining resource capacity for execution, so that the idle resources in the computer cluster are concentrated on a small number of underutilized nodes, thereby slowing down the generation of new resource fragmentation. However, as the number of tasks increases, the generated resource fragmentation will gradually increase, resulting in a situation that the scheduling requirements of large new tasks cannot be met due to excessive resource fragmentation. As can be seen, the reliability of this method of slowing down the generation of new resource fragmentation from the source to meet the resource requirements of large new tasks is low.

[0159] In order to improve the reliability, in the related art, the descheduler mechanism can be used to identify nodes with low resource utilization in the computer cluster, and actively schedule tasks on the nodes to nodes with high resource utilization, that is, actively evict one or more tasks on the nodes with low resource utilization. In this way, the node with low resource utilization is emptied, so that the node with low resource utilization has continuous available large-capacity idle resources for execution of new tasks. The evicted tasks are found new suitable nodes by the control node.

[0160] However, the method in the related art has the following problems: 1. Blindness and uncertainty. Specifically, the descheduler mechanism is "blind" when performing the eviction operation. It is only responsible for "kicking out" the task, but cannot predict or guarantee the result after the eviction operation. As a result, it may lead to the failure of the arrangement and the invalid arrangement. The arrangement failure refers to that after the descheduler evicts the instance, the overall resource status of the computer cluster may not be improved, and even because there is no other suitable node to run the evicted task, the evicted task itself cannot be rescheduled, and the running state changes to the waiting state, causing service interruption. The invalid arrangement refers to that the capacity of the resources occupied by the task evicted by the descheduler is much smaller than the capacity of the resources required by the new task to be executed. For example, the descheduler may successfully migrate a task occupying 1 GPU from node A to node B, but if the execution of the new task to be executed requires complete 8 GPUs. Then, this arrangement operation is invalid arrangement, and causes a task disturbance.

[0161] 2. Non-atomicity and scheduling race. Specifically, the eviction and scheduling in the descheduler mechanism are two independent and asynchronous processes. Non-atomicity refers to that between the eviction and the scheduling, the scheduling of the current task may be interrupted by other higher priority tasks. Scheduling race refers to that multiple tasks compete for the same resource. In the descheduler mechanism, from the eviction of the task, the release of the resources of the evicted task, to the scheduling of the new task on the released resources, there is a time window. During this window period, the released resources may be "preempted" by other unrelated tasks in the computer cluster with higher priority. Moreover, there may be multiple evicted tasks competing for the same resource, resulting in the situation that the evicted task that does not preempt the resource cannot be re-run due to scheduling failure. As a result, it may lead to the increase of a large number of tasks that fail to preempt the resources in the computer cluster, resulting in the instability of the state of the computer cluster.

[0162] 3. Lack of global optimality and cost awareness. Specifically, the decision logic of the descheduler mechanism is relatively simple. For example, resource scheduling is only based on the resource utilization threshold of a single node, that is, the task on the node with low resource utilization is scheduled to the node with high resource utilization. For example, it may evict a high-priority task while retaining a low-priority task that can be started and stopped at any time. As a result, it leads to a high cost of resource fragmentation arrangement, and a large impact on the task.

[0163] The method provided in the embodiment of the application comprises the following steps: obtaining a plurality of candidate resource scheduling paths according to the first resource capacity and the first task currently being executed; determining a target resource scheduling path from the plurality of candidate resource scheduling paths; and scheduling the first task based on the target resource scheduling path to perform resource fragmentation consolidation.

[0164] By simulating resource scheduling to obtain the target resource scheduling path, a suitable destination node can be found for the evicted task, and the problems of blindness and uncertainty in the related art are solved. The task interruption caused by blind task eviction is avoided, and it is ensured that all evicted tasks can be scheduled to other nodes to be re-executed, so that the situations of consolidation failure and invalid consolidation can be avoided.

[0165] When the candidate resource scheduling path is determined, the first resource capacity, the second resource capacity of all migratable first tasks in the computer cluster, and the initial resource capacity of each node are considered, so that the path of resource scheduling can be planned from a global perspective. Moreover, by calculating the cost of the scheduling path and determining the candidate resource scheduling path with the minimum scheduling cost as the target resource scheduling path, the consolidation scheme with the minimum task interference, i.e., the globally optimal target resource scheduling path, can be determined. That is, an optimal scheduling scheme that can complete the consolidation target and minimize the interference to the task can be found. Thus, the consolidation cost of the resource fragmentation this time and the influence on the task are effectively reduced.

[0166] In the process of scheduling the first task based on the target resource scheduling path, the resource is atomically locked by setting the placeholder instance, so that the consolidated resource can be prevented from being used by other tasks, and the situation that the first tasks after migration occupy the same resource and cause the scheduling failure of the first task is avoided, thereby solving the problems of non-atomicity and scheduling race.

[0167] In summary, after the resource scheduling method provided in the embodiment of the application is used to perform resource fragmentation consolidation, on the one hand, the average resource utilization rate of each node in the computer cluster can be improved; on the other hand, the waiting time of the task can be shortened, and the efficiency of the task execution can be improved; and on the other hand, the service capacity boundary of the computer cluster can be expanded.

[0168] Figure 7A structural schematic diagram of a resource fragmentation arrangement device is provided for an embodiment of the present application. The device comprises: a simulation module 701, a determination module 702 and a scheduling module 703. The simulation module 701 is configured to simulate resource scheduling according to a first resource capacity and a first task currently being executed by each node to obtain a plurality of candidate resource scheduling paths, wherein the remaining resource capacity of at least one node can reach the first resource capacity after resource fragmentation arrangement is performed by using each candidate resource scheduling path. The determination module 702 is configured to determine a target resource scheduling path from the plurality of candidate resource scheduling paths. The scheduling module 703 is configured to schedule the first task based on the target resource scheduling path to perform resource fragmentation arrangement.

[0169] Optionally, the simulation module 701 is specifically configured to obtain a second resource capacity occupied by the first task currently being executed by each node in the plurality of nodes; simulate resource scheduling based on the second resource capacity, the first resource capacity and a characteristic parameter of the first task to obtain the plurality of candidate resource scheduling paths, wherein the characteristic parameter comprises: priority, interruptability and recoverability of the first task.

[0170] Optionally, the simulation module 701 is specifically configured to determine at least two target first tasks from the plurality of first tasks based on the characteristic parameter of the first task, wherein the priority of each target first task is lower than a priority threshold, the target first task is interruptable and recoverable; determine a scheduling mode of the target first task based on the second resource capacity occupied by each target first task, the first resource capacity and an initial resource capacity of each node; and arrange and combine the scheduling modes of all target first tasks to obtain the plurality of candidate resource scheduling paths.

[0171] Optionally, the simulation module 701 is further configured to take a snapshot of the plurality of nodes to obtain the second resource capacity occupied by the first task currently being executed by each node in the plurality of nodes.

[0172] Optionally, the determination module 702 is specifically configured to obtain a scheduling cost of each candidate resource scheduling path; and determine the candidate resource scheduling path with the minimum scheduling cost as the target resource scheduling path.

[0173] Optionally, the determining module 702 is specifically configured to: for each candidate resource scheduling path, acquire the number of the first tasks to be scheduled, the interruption sensitivity of the first tasks to be scheduled, the difference of the destination nodes of any two first tasks to be scheduled with the correlation degree greater than the degree threshold, and the scheduling cost of the first tasks to be scheduled; and based on the number of the first tasks to be scheduled, the interruption sensitivity of the first tasks to be scheduled, the difference of the destination nodes of any two first tasks to be scheduled with the correlation degree greater than the degree threshold, and the scheduling cost of the first tasks to be scheduled, acquire the scheduling cost of the candidate resource scheduling path; wherein the scheduling cost of the candidate resource scheduling path is positively correlated with the number of the first tasks to be scheduled, the interruption sensitivity of the first tasks to be scheduled, and the scheduling cost of the first tasks, and the scheduling cost of the candidate resource scheduling path is lower in the case that the destination nodes of the two first tasks to be scheduled are different than in the case that the destination nodes of the two first tasks to be scheduled are the same.

[0174] Optionally, the target resource scheduling path indicates that the first task in the first node is scheduled from the first node to a second node in the plurality of nodes, so that the remaining resource capacity of the first node reaches the first resource capacity; and the scheduling module 703 is further configured to: based on the target resource scheduling path, lock the resources of the first resource capacity in the first node, and lock the resources of the second resource capacity in the second node; wherein the second resource capacity is the resource capacity occupied by the first task in the first node.

[0175] Optionally, the scheduling module 703 is specifically configured to: based on the target resource scheduling path, create a first placeholder instance of the first resource capacity in the first node, so as to lock the resources of the first resource capacity in the first node; and create a second placeholder instance in the second node, so as to lock the resources of the second resource capacity in the second node.

[0176] Optionally, the scheduling module 703 is further configured to, in a case that it is determined that there is a second task to be executed, based on the target resource scheduling path, lock the resources of the first resource capacity in the first node, and lock the resources of the second resource capacity in the second node; wherein the resource capacity required by the second task is less than or equal to the first resource capacity.

[0177] Optionally, the number of the first tasks is a plurality, and each first task runs in a node; and the scheduling module 703 is further configured to, if at least one first task fails to be scheduled successfully, schedule the first tasks that have been successfully scheduled to the source node of the first tasks, and release the resources occupied by the first tasks in the destination nodes of the first tasks.

[0178] Optionally, the scheduling module 703 is further configured to simulate resource scheduling according to the first resource capacity and the first task currently being executed, if the resource fragment consolidation condition is met; wherein the consolidation condition comprises: a waiting time length of the second task to be executed is greater than a first time threshold, and / or a dispersion degree of resources in the computer cluster is greater than a degree threshold.

[0179] Optionally, the consolidation condition comprises: the dispersion degree of resources in the computer cluster is greater than the degree threshold; the scheduling module 703 is specifically configured to obtain a proportion of unavailable resources of the computer cluster; if it is determined that the proportion of unavailable resources is greater than a proportion threshold, it is determined that the dispersion degree of resources in the computer cluster is greater than the degree threshold.

[0180] Optionally, the scheduling module 703 is specifically configured to: if the consolidation condition is met for a second time threshold, the resource consolidation has not been completed, and / or if the candidate resource scheduling path is obtained for a third time threshold, the resource fragment consolidation is ended.

[0181] The device of the embodiment corresponds to the technical solutions of the above-mentioned method embodiments, and has similar implementation principles and technical effects, which will not be described here.

[0182] The embodiment of the application further provides a controller, which comprises a processor and a memory. The memory stores programs or instructions that can be run on the processor, and the programs or instructions are executed by the processor to implement the steps of the resource fragment consolidation method as shown in Figures 1 to 6 .

[0183] The embodiment of the application further provides a control node, which comprises the controller provided by the embodiment of the application.

[0184] The embodiment of the application further provides a computer cluster, which comprises the control node provided by the embodiment of the application.

[0185] The embodiment of the application further provides a computer readable storage medium, which stores programs or instructions, and the programs or instructions are executed by a processor to implement the steps of the resource fragment consolidation method as shown in Figures 1 to 6 .

[0186] The embodiment of the application further provides a computer program product, which is executed by a processor of a vehicle or a cloud server to implement the steps of the resource fragment consolidation method as shown in Figures 1 to 6 .

[0187] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned example methods can be realized by means of a computer software product and a general hardware platform as necessary, and of course can also be realized by hardware. The computer software product is stored in a storage medium (such as a ROM, a RAM, a magnetic disc, an optical disc, etc.), and includes a plurality of instructions for enabling a terminal or a network side device to execute the method described in each embodiment of the present application.

[0188] The embodiments of the present application are described above in combination with the drawings, but the present application is not limited to the specific embodiments described above, and the specific embodiments described above are merely illustrative rather than limiting. Those skilled in the art can make many forms of embodiments under the inspiration of the present application without departing from the scope of the present application and the scope protected by the claims, and these embodiments all belong to the protection of the present application.

Claims

1. A method of resource fragmentation consolidation, the method comprising: identifying a plurality of resource fragments; and consolidating the plurality of resource fragments into a single resource. 1 The method is applied to a computer cluster, the computer cluster comprising: obtaining second resource capacity occupied by a first task currently executed by each node in the plurality of nodes; determining at least two target first tasks from the plurality of first tasks based on characteristic parameters of the first tasks, each target first task having a priority lower than a priority threshold, being interruptible and being recoverable, the characteristic parameters comprising: priority, interruptibility and recoverability of the first task; determining a scheduling mode of the target first tasks based on the first resource capacity, second resource capacity occupied by each target first task, and initial resource capacity of each node; performing permutation and combination on the scheduling modes of all the target first tasks to obtain a plurality of candidate resource scheduling paths, wherein after resource fragmentation consolidation is performed using each candidate resource scheduling path, the remaining resource capacity of at least one node reaches the first resource capacity; determining a target resource scheduling path from the plurality of candidate resource scheduling paths; scheduling the first tasks based on the target resource scheduling path to perform resource fragmentation consolidation.

2. The method of claim 1, wherein, The obtaining of the second resource capacity occupied by the first task currently executed by each node in the plurality of nodes comprises: performing snapshot on the plurality of nodes to obtain the second resource capacity occupied by the first task currently executed by each node in the plurality of nodes.

3. The method according to claim 1 or 2, characterized in that, The determining of the target resource scheduling path from the plurality of candidate resource scheduling paths comprises: obtaining a scheduling cost of each candidate resource scheduling path; determining the candidate resource scheduling path with the minimum scheduling cost as the target resource scheduling path.

4. The method of claim 3, wherein, Each candidate resource scheduling path comprises: a destination node of the first task to be scheduled; and the obtaining of the scheduling cost of each candidate resource scheduling path comprises: for each candidate resource scheduling path, obtaining: a number of the first tasks to be scheduled, interruptibility of the first tasks to be scheduled, similarity of destination nodes of any two first tasks to be scheduled with a correlation degree greater than a degree threshold, and a scheduling cost of the first tasks to be scheduled; obtaining the scheduling cost of the candidate resource scheduling path based on the number of the first tasks to be scheduled, the interruptibility of the first tasks to be scheduled, the similarity of the destination nodes of any two first tasks to be scheduled with the correlation degree greater than the degree threshold, and the scheduling cost of the first tasks to be scheduled; wherein the scheduling cost of the candidate resource scheduling path is positively correlated with the number of the first tasks to be scheduled, the interruptibility of the first tasks to be scheduled, and the scheduling cost of the first tasks, and the scheduling cost of the candidate resource scheduling path is lower in the case where the destination nodes of the two first tasks to be scheduled are the same than in the case where the destination nodes of the two first tasks to be scheduled are different.

5. The method according to claim 1 or 2, characterized in that, The target resource scheduling path indicates to schedule a first task in the first node from the first node to a second node in the plurality of nodes to reach the first resource capacity for the remaining resource capacity of the first node; Before the scheduling the first task based on the target resource scheduling path, the method further comprises: locking resources of the first resource capacity in the first node and resources of a second resource capacity in the second node based on the target resource scheduling path; wherein the second resource capacity is the resource capacity occupied by the first task in the first node.

6. The method of claim 5, wherein, The locking resources of the first resource capacity in the first node and resources of a second resource capacity in the second node based on the target resource scheduling path comprises: creating a first placeholder instance of the first resource capacity in the first node to lock the resources of the first resource capacity in the first node based on the target resource scheduling path; and creating a second placeholder instance in the second node to lock the resources of the second resource capacity in the second node.

7. The method of claim 5, wherein, The locking resources of the first resource capacity in the first node and resources of a second resource capacity in the second node based on the target resource scheduling path comprises: locking resources of the first resource capacity in the first node and resources of a second resource capacity in the second node based on the target resource scheduling path in a case that it is determined that there is a second task to be executed; wherein the second task requires a resource capacity less than or equal to the first resource capacity.

8. The method of claim 1 or 2, wherein, The number of the first tasks is a plurality, each of the first tasks is running in a node; after the scheduling the first task based on the target resource scheduling path, the method further comprises: if at least one of the first tasks is not successfully scheduled, scheduling the successfully scheduled first task to a source node of the first task and releasing resources occupied by the first task in a destination node of the first task.

9. The method of claim 1 or 2, wherein, The simulating resource scheduling according to the first resource capacity and the first task currently being executed comprises: simulating resource scheduling according to the first resource capacity and the first task currently being executed in a case that a resource fragmentation consolidation condition is met; wherein the consolidation condition comprises that a waiting time length of a second task to be executed is greater than a first time threshold, and / or a dispersion degree of resources in the computer cluster is greater than a degree threshold.

10. The method of claim 9, wherein, The consolidation condition comprises that the dispersion degree of resources in the computer cluster is greater than the degree threshold; before the simulating resource scheduling according to the first resource capacity and the first task currently being executed, the method further comprises: obtaining a proportion of unavailable resources of the computer cluster; if it is determined that the proportion of the unavailable resources is greater than a proportion threshold, it is determined that the dispersion degree of resources in the computer cluster is greater than the degree threshold.

11. The method of claim 9, wherein, The method further comprises: If the duration of the satisfaction of the arrangement condition reaches a second duration threshold, and the resource arrangement is not completed, and / or if the duration of the acquisition of the candidate resource scheduling path is greater than a third duration threshold, the resource fragmentation arrangement is ended.

12. A controller characterized by comprising: The controller comprises a processor and a memory, the memory stores programs or instructions executable on the processor, and the programs or instructions are executed by the processor to implement the steps of the resource fragmentation arrangement method according to any one of claims 1 to 11.

13. A control node, characterized by The control node comprises the controller according to claim 12.

14. A computer cluster, characterized by The computer cluster comprises the control node according to claim 13.

15. A computer-readable storage medium, characterized in that, The readable storage medium stores programs or instructions, and the programs or instructions are executed by the processor to implement the steps of the resource fragmentation arrangement method according to any one of claims 1 to 11.

Images