Resource defragmentation method, controller, control node and computer cluster

Abstract

The invention provides a resource defragmentation method, a controller, a control node and a computer cluster, and relates to the technical field of data processing. The method comprises the following steps: simulating resource scheduling according to a first resource capacity and a first task which is being executed at present to obtain a plurality of candidate resource scheduling paths; determining a target resource scheduling path from the plurality of candidate resource scheduling paths; and based on the target resource scheduling path, scheduling the first task to perform resource defragmentation. After each candidate resource scheduling path is adopted for resource defragmentation, the residual resource capacity of at least one node can reach the first resource capacity. According to the method, the resource fragments in the computer cluster can be actively eliminated by scheduling the first task to arrange the resource fragments. Compared with a mode of slowing down generation of new resource fragments from the source, the scheduling reliability of tasks with relatively high resource requirements can be effectively improved.

Description

Technical Field

[0001] This application belongs to the field of data processing technology, specifically relating to a resource fragmentation method, a controller, a control node, and a computer cluster. Background Technology

[0002] Resource fragmentation (i.e., fragmented resources) refers to small, underutilized resources provided by a computer cluster. This fragmentation makes it difficult to meet the resource requirements of large new tasks, preventing these tasks from being scheduled. Large new tasks are those with rigid requirements for resource continuity and centralization, which cannot be executed by splitting resources across nodes. For example, a task requiring eight graphics processing units (GPUs) cannot run on multiple nodes, each with two to three idle GPUs.

[0003] Currently, resource allocation typically prioritizes assigning new tasks to nodes with limited remaining resource capacity. This concentrates idle resources across a few underutilized nodes, thus mitigating the generation of new resource fragmentation. In other words, the current approach aims to meet the resource demands of large new tasks by reducing the generation of new resource fragmentation at its source.

[0004] However, this approach of meeting the resource needs of large new missions by mitigating the generation of new resource fragments at the source is less reliable. Summary of the Invention

[0005] This application provides a resource fragmentation defragmentation method, controller, control node, and computer cluster, which can solve the problem of low reliability in related technologies that meet the resource requirements of large new tasks by slowing down the generation of new resource fragments from the source.

[0006] Firstly, this application provides a resource defragmentation method, which is applied to a computer cluster, the computer cluster comprising: multiple nodes; including: Based on the first resource capacity and the first task currently being executed, simulated resource scheduling is obtained to obtain multiple candidate resource scheduling paths. After resource fragmentation is performed using each of the candidate resource scheduling paths, the remaining resource capacity of at least one of the nodes can reach the first resource capacity. The target resource scheduling path is determined from the multiple candidate resource scheduling paths; Based on the target resource scheduling path, the first task is scheduled to perform resource fragmentation.

[0007] Optionally, the step of simulating resource scheduling based on the first resource capacity and the currently executing first task to obtain multiple candidate resource scheduling paths includes: Obtain the second resource capacity occupied by the first task currently being executed by each of the plurality of nodes; Based on the second resource capacity, the first resource capacity, and the characteristic parameters of the first task, resource scheduling is simulated to obtain multiple candidate resource scheduling paths, wherein the characteristic parameters include: the priority, interruptibility, and recoverability of the first task.

[0008] Optionally, the step of simulating resource scheduling based on the second resource capacity, the first resource capacity, and the characteristic parameters of the first task to obtain multiple candidate resource scheduling paths includes: Based on the characteristic parameters of the first task, at least two target first tasks are determined from a plurality of first tasks, each of the target first tasks having a priority lower than a priority threshold, being interruptible and recoverable; Based on the second resource capacity occupied by each of the target first tasks, the first resource capacity, and the initial resource capacity of each of the nodes, the scheduling method of the target first tasks is determined; The scheduling methods for all the aforementioned target first tasks are arranged and combined to obtain multiple candidate resource scheduling paths.

[0009] Optionally, obtaining the second resource capacity occupied by the first task currently being executed by each of the plurality of nodes includes: Take snapshots of the plurality of nodes to obtain the second resource capacity occupied by the first task currently being executed by each of the plurality of nodes.

[0010] Optionally, determining the target resource scheduling path from the plurality of candidate resource scheduling paths includes: Obtain the scheduling cost of each of the candidate resource scheduling paths; The candidate resource scheduling path with the lowest scheduling cost is determined as the target resource scheduling path.

[0011] Optionally, each of the candidate resource scheduling paths includes: the destination node of the first task to be scheduled; obtaining the scheduling cost of each of the candidate resource scheduling paths includes: For each of the candidate resource scheduling paths, obtain the number of first tasks to be scheduled included in the candidate resource scheduling path, the interruption sensitivity of the first tasks to be scheduled, the similarities and differences of the destination nodes of any two first tasks to be scheduled whose correlation degree is greater than the degree threshold, and the scheduling cost of the first tasks to be scheduled. Based on the number of the first tasks to be scheduled, the interrupt sensitivity of the first tasks to be scheduled, the similarities and differences of the destination nodes of any two first tasks whose correlation is greater than the degree threshold, and the scheduling cost of the first tasks to be scheduled, the scheduling cost of the candidate resource scheduling path is obtained. The scheduling cost of the candidate resource scheduling path is positively correlated with the number of the first tasks to be scheduled, the interruption sensitivity of the first tasks to be scheduled, and the scheduling cost of the first tasks. Moreover, compared with the case where the destination nodes of the two first tasks to be scheduled are different and the degree of correlation is greater than the degree threshold, the scheduling cost of the candidate resource scheduling path is lower when the destination nodes are the same.

[0012] Optionally, the target resource scheduling path indicates that the first task in the first node is scheduled from the first node to the second node among the plurality of nodes, so that the remaining resource capacity of the first node reaches the first resource capacity; before scheduling the first task based on the target resource scheduling path, the method further includes: Based on the target resource scheduling path, resources of the first resource capacity are locked in the first node, and resources of the second resource capacity are locked in the second node; The second resource capacity is the resource capacity occupied by the first task in the first node.

[0013] Optionally, locking resources of the first resource capacity in the first node and locking resources of the second resource capacity in the second node based on the target resource scheduling path includes: Based on the target resource scheduling path, a first placeholder instance of the first resource capacity is created in the first node to lock the resource of the first resource capacity in the first node; And a second placeholder instance created in the second node to lock the resource of the second resource capacity in the second node.

[0014] Optionally, locking resources of the first resource capacity in the first node and locking resources of the second resource capacity in the second node based on the target resource scheduling path includes: If it is determined that there is a second task to be executed, based on the target resource scheduling path, the resources of the first resource capacity are locked in the first node, and the resources of the second resource capacity are locked in the second node; The resource capacity required for the second task is less than or equal to the resource capacity required for the first task.

[0015] Optionally, the number of the first tasks is multiple, and each first task runs on a single node; after scheduling the first tasks based on the target resource scheduling path, the method further includes: If at least one of the first tasks fails to schedule, the first tasks that have been successfully scheduled will be scheduled to the source node of the first task, and the resources occupied by the first task in the destination node of the first task will be released.

[0016] Optionally, the step of simulating resource scheduling based on the first resource capacity and the currently executing first task includes: If the conditions for reorganizing resource fragments are met, then resource scheduling is simulated based on the first resource capacity and the first task currently being executed. The sorting conditions include: the waiting time of the second task to be executed is greater than a first time threshold, and / or the degree of resource dispersion in the computer cluster is greater than a degree threshold.

[0017] Optionally, the sorting conditions include: the degree of resource dispersion in the computer cluster is greater than a degree threshold; before simulating resource scheduling based on the first resource capacity and the currently executing first task, the method further includes: Obtain the percentage of unavailable resources in the computer cluster; If it is determined that the proportion of unavailable resources is greater than the proportion threshold, then it is determined that the degree of resource dispersion in the computer cluster is greater than the degree threshold.

[0018] Optionally, the method further includes: If the duration for which the sorting conditions are met reaches the second duration threshold and resource sorting has not yet been completed, and / or if the acquisition duration of the candidate resource scheduling path is greater than the third duration threshold, then resource fragment sorting ends.

[0019] Secondly, this application provides a resource fragmentation sorting device, comprising: The simulation module is used to simulate resource scheduling based on the first resource capacity and the first task currently being executed, and obtain multiple candidate resource scheduling paths. After resource fragmentation is performed using each of the candidate resource scheduling paths, the remaining resource capacity of at least one of the nodes can reach the first resource capacity. The determining module is used to determine the target resource scheduling path from the multiple candidate resource scheduling paths; The scheduling module is used to schedule the first task based on the target resource scheduling path in order to perform resource fragmentation.

[0020] Optionally, the simulation module is specifically used to obtain the second resource capacity occupied by the first task currently being executed by each of the plurality of nodes; based on the second resource capacity, the first resource capacity, and the characteristic parameters of the first task, simulate resource scheduling to obtain a plurality of candidate resource scheduling paths, wherein the characteristic parameters include: the priority, interruptibility, and recoverability of the first task.

[0021] Optionally, the simulation module is specifically used to determine at least two target first tasks from multiple first tasks based on the characteristic parameters of the first task, wherein each target first task has a priority lower than a priority threshold, is interruptible, and is recoverable; determine the scheduling method of the target first tasks based on the second resource capacity occupied by each target first task, the first resource capacity, and the initial resource capacity of each node; and arrange and combine the scheduling methods of all target first tasks to obtain multiple candidate resource scheduling paths.

[0022] Optionally, the simulation module is further configured to take snapshots of the plurality of nodes to obtain the second resource capacity occupied by the first task currently being executed by each of the plurality of nodes.

[0023] Optionally, the determining module is specifically used to obtain the scheduling cost of each of the candidate resource scheduling paths; and to determine the candidate resource scheduling path with the lowest scheduling cost as the target resource scheduling path.

[0024] Optionally, the determining module is specifically used to, for each candidate resource scheduling path, obtain the number of first tasks to be scheduled, the interrupt sensitivity of the first tasks to be scheduled, the similarities and differences of the destination nodes of any two first tasks to be scheduled whose correlation is greater than a threshold, and the scheduling cost of the first tasks to be scheduled; based on the number of first tasks to be scheduled, the interrupt sensitivity of the first tasks to be scheduled, the similarities and differences of the destination nodes of any two first tasks to be scheduled whose correlation is greater than a threshold, and the scheduling cost of the first tasks to be scheduled, obtain the scheduling cost of the candidate resource scheduling path; wherein, the scheduling cost of the candidate resource scheduling path is positively correlated with the number of first tasks to be scheduled, the interrupt sensitivity of the first tasks to be scheduled, and the scheduling cost of the first tasks, and compared with the case where the destination nodes of the two first tasks to be scheduled whose correlation is greater than a threshold are different, the scheduling cost of the candidate resource scheduling path is lower when the destination nodes are the same.

[0025] Optionally, the target resource scheduling path indicates that the first task in the first node is scheduled from the first node to the second node among the plurality of nodes, so that the remaining resource capacity of the first node reaches the first resource capacity; the scheduling module is further configured to: based on the target resource scheduling path, lock the resources of the first resource capacity in the first node, and lock the resources of the second resource capacity in the second node; wherein, the second resource capacity is the resource capacity occupied by the first task in the first node.

[0026] Optionally, the scheduling module is specifically configured to, based on the target resource scheduling path, create a first placeholder instance of the first resource capacity in the first node to lock the resource of the first resource capacity in the first node; and create a second placeholder instance in the second node to lock the resource of the second resource capacity in the second node.

[0027] Optionally, the scheduling module is further configured to, upon determining that there is a second task to be executed, lock resources of the first resource capacity in the first node and lock resources of the second resource capacity in the second node based on the target resource scheduling path; wherein the resource capacity required by the second task is less than or equal to the first resource capacity.

[0028] Optionally, the number of the first tasks is multiple, and each first task runs on a node; the scheduling module is further configured to, if at least one first task fails to be scheduled, schedule the successfully scheduled first task to the source node of the first task, and release the resources occupied by the first task in the destination node of the first task.

[0029] Optionally, the scheduling module is further configured to simulate resource scheduling based on the first resource capacity and the currently executing first task when the resource fragmentation conditions are met; wherein the fragmentation conditions include: the waiting time of the second task to be executed is greater than a first duration threshold, and / or, the degree of resource dispersion in the computer cluster is greater than a degree threshold.

[0030] Optionally, the sorting conditions include: the degree of resource dispersion in the computer cluster is greater than a degree threshold; the scheduling module is specifically used to obtain the proportion of unavailable resources in the computer cluster; if it is determined that the proportion of unavailable resources is greater than a proportion threshold, then it is determined that the degree of resource dispersion in the computer cluster is greater than a degree threshold.

[0031] Optionally, the scheduling module is specifically used to end resource fragmentation if the duration for which the sorting conditions are met reaches a second duration threshold and resource sorting has not yet been completed, and / or if the acquisition duration of the candidate resource scheduling path is greater than a third duration threshold.

[0032] Thirdly, this application provides a controller, including: a processor and a memory, the memory storing a program or instructions executable on the processor, the program or instructions, when executed by the processor, implementing the steps of the resource defragmentation method as described in any of the first aspects.

[0033] Fourthly, this application provides a control node, which includes the controller as described in the third aspect.

[0034] Fifthly, this application provides a computer cluster, the computer cluster including the control node as described in the fourth aspect.

[0035] In a sixth aspect, this application provides a computer-readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the resource defragmentation method as described in any of the first aspects.

[0036] In a seventh aspect, this application provides a computer program product that, when executed by a processor of a vehicle or a cloud server, implements the steps of the resource defragmentation method as described in any of the first aspects.

[0037] This application provides a resource fragmentation defragmentation method, controller, control node, and computer cluster. The method simulates resource scheduling based on a first resource capacity and a currently executing first task, obtaining multiple candidate resource scheduling paths. From these candidate paths, a target resource scheduling path is determined. Based on the target resource scheduling path, the first task is scheduled for resource fragmentation defragmentation. After defragmentation using each candidate resource scheduling path, at least one node's remaining resource capacity reaches the first resource capacity. In other words, this method can proactively eliminate resource fragmentation in the computer cluster by scheduling the first task for resource fragmentation defragmentation, effectively consolidating scattered resource fragments into contiguous large blocks of resources. Compared to mitigating the generation of new resource fragments at the source, the method provided in this application effectively avoids situations where excessive resource fragmentation prevents task scheduling, thereby significantly improving the scheduling reliability of tasks with high resource demands. Attached Figure Description

[0038] Figure 1 A flowchart illustrating a resource fragmentation defragmentation method provided in an embodiment of this application; Figure 2 A flowchart illustrating another resource fragmentation defragmentation method provided in this application embodiment; Figure 3 A flowchart illustrating a method for obtaining dispersion degree provided in an embodiment of this application; Figure 4 A flowchart illustrating another resource fragmentation defragmentation method provided in this application embodiment; Figure 5 A flowchart illustrating yet another resource fragmentation defragmentation method provided in this application embodiment; Figure 6 A flowchart illustrating yet another resource fragmentation defragmentation method provided in this application embodiment; Figure 7 This is a schematic diagram of the structure of a resource fragmentation sorting device provided in an embodiment of this application. Detailed Implementation

[0039] The technical solutions in the embodiments of this application will be clearly described below with reference to the accompanying drawings. All other embodiments obtained by those skilled in the art based on the embodiments in this application are within the scope of protection of this application.

[0040] The terms "first," "second," etc., used in this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and that the objects distinguished by "first" and "second" are generally of the same class and are not limited in number.

[0041] The following embodiments of this application can be applied to computer clusters, such as control nodes in a computer cluster. For example, they can be executed by a controller in the control node. Specifically, the controller can be equipped with a scheduler, and the following embodiments can be executed by the scheduler. The following describes embodiments of the resource fragmentation method with several specific examples. Figure 1 This is a flowchart illustrating a resource fragmentation defragmentation method provided in an embodiment of this application. Figure 1 As shown: S11: Simulate resource scheduling based on the first resource capacity and the first task currently being executed to obtain multiple candidate resource scheduling paths.

[0042] Specifically, by performing resource defragmentation using each candidate resource scheduling path, at least one node's remaining resource capacity can reach, or be greater than or equal to, the first resource capacity. This first resource capacity can be pre-stored by the control node. Alternatively, the first resource capacity can be the capacity of resources required by the second task to be executed received by the computer cluster. Resource defragmentation refers to releasing the resources occupied by the first task currently being executed on a node by scheduling (i.e., migrating) it to another node, thereby allowing the released fragmented resources to be aggregated into large, contiguous blocks of resources.

[0043] Candidate resource scheduling paths refer to resource reallocation methods, aiming to relocate the first task currently running on one node to another node, so that the remaining resource capacity of the first node reaches the first resource capacity. Therefore, the method provided in this application embodiment can enable at least one node to execute a task with the first resource capacity without adding new resources, through resource reallocation.

[0044] S12: Determine the target resource scheduling path from multiple candidate resource scheduling paths.

[0045] The control node can calculate the scheduling cost of multiple candidate resource scheduling paths and determine the candidate resource scheduling path with the lowest scheduling cost as the target resource scheduling path.

[0046] S13: Based on the target resource scheduling path, schedule the first task to perform resource fragmentation.

[0047] The control node schedules the first task to consolidate the originally scattered resource fragments into a continuous and usable first resource capacity, thereby reorganizing the resource fragments.

[0048] In this embodiment, the method simulates resource scheduling based on a first resource capacity and a currently executing first task to obtain multiple candidate resource scheduling paths; from these candidate paths, a target resource scheduling path is determined; and based on the target resource scheduling path, the first task is scheduled to perform resource fragmentation. After performing resource fragmentation using each candidate resource scheduling path, at least one node's remaining resource capacity can reach the first resource capacity. That is, this method can proactively eliminate resource fragmentation in a computer cluster by scheduling the first task to perform resource fragmentation, thereby effectively integrating scattered resource fragments into contiguous large blocks of resources. Compared to mitigating the generation of new resource fragments at the source, the method provided in this application can effectively avoid situations where excessive resource fragmentation leads to task scheduling failures, thus effectively improving the scheduling reliability of tasks with high resource demands.

[0049] Figure 2 This is a flowchart illustrating another resource fragmentation defragmentation method provided in an embodiment of this application. Figure 2 As shown, Figure 2 Is Figure 1 Based on the illustrated embodiment, one possible implementation of S11 is as follows: S111: Get the second resource capacity occupied by the first task currently being executed by each of the multiple nodes.

[0050] Specifically, the control node takes snapshots of multiple nodes to obtain the second resource capacity occupied by the first task currently being executed by each node. A snapshot is a momentary record of the current state of a node in a computer cluster. This current state includes: the first task running on the node, the resource capacity occupied by each first task, and the node's initial resource capacity. This initial resource capacity is the node's maximum resource capacity.

[0051] Optionally, there can be multiple first tasks, with each first task running on a single node.

[0052] S112: Based on the second resource capacity, the first resource capacity, and the characteristic parameters of the first task, simulate resource scheduling to obtain multiple candidate resource scheduling paths.

[0053] Each candidate resource scheduling path includes: a first task to be scheduled, and a destination node for that first task. The destination node is the node where the first task will be executed after scheduling, and the node where the first task will ultimately run.

[0054] In this embodiment of the application, the implementation process of S112 may include: Step A1: Based on the feature parameters of the first task, determine at least two target first tasks from multiple first tasks.

[0055] The characteristic parameters include: priority, interruptibility, and recoverability. Interruptibility indicates whether the first task can be interrupted, and recoverability indicates whether the first task can be resumed, such as automatically.

[0056] The control node can select the target first task from multiple first tasks, choosing those with a priority lower than a priority threshold (i.e., lower priority), which are interruptible and recoverable. In other words, the control node can filter a batch of low-priority, interruptible, and automatically recoverable first tasks from the computer cluster as potential tasks to be scheduled. This priority threshold can be pre-stored by the control node.

[0057] Optionally, the interruptibility and recoverability of each first task can be determined by the control node after simulating the scheduling of the first task.

[0058] Step A2: Based on the second resource capacity, the first resource capacity, and the initial resource capacity of each target first task, determine the scheduling method of the target first task.

[0059] The conditions under which the target first task can run on a node include: before the target first task is migrated, the node's remaining resource capacity is greater than or equal to the second resource capacity occupied by the target first task. The node's remaining resource capacity is the difference between the initial resource capacity and the sum of the values, which is the sum of the second resource capacities occupied by all currently running first tasks on that node.

[0060] Based on this, for each target first task, the control node can filter out the destination node from multiple nodes based on the relationship between the remaining resource capacity of each node and the second resource capacity occupied by the target first task. If the remaining resource capacity of the destination node is greater than or equal to the second resource capacity, then the destination node is a schedulable destination node for the target first task. Then, the control node can determine one scheduling method for the target first task: scheduling the target first task from its current source node to a destination node. Any scheduling method for the target first task is used to indicate a destination node for the target first task.

[0061] Step A3: Arrange and combine the scheduling methods of all target primary tasks to obtain multiple candidate resource scheduling paths.

[0062] In scenarios where there are multiple scheduling methods for the primary objective task, the control node can permutate and combine all the scheduling methods of the primary objective task to obtain multiple candidate resource scheduling paths. For example, the control node can permutate and combine all the scheduling methods of the primary objective task to obtain multiple initial resource scheduling paths. Then, for each initial resource scheduling path, if the control node determines that there are no resource conflicts among the primary objective tasks included in the initial resource scheduling path, it will designate the initial resource scheduling path as a candidate resource scheduling path.

[0063] A resource conflict occurs when, in a given initial resource scheduling path, the sum of the second resource capacities of multiple target tasks sharing the same destination node exceeds the remaining resource capacity of that destination node. In other words, the total resource demand of these multiple target tasks exceeds the currently available remaining resource capacity of that destination node. For example, in a given initial resource scheduling path, both target task A and target task B need to be scheduled to node 1, and the sum of their second resource capacities exceeds the remaining resource capacity of node 1. Therefore, target task A and target task B have a resource conflict.

[0064] The following is a specific embodiment illustrating a scheme for simulating resource scheduling based on the resource capacity occupied by the first objective task, the first resource capacity, and the remaining resource capacity of each node, to obtain multiple candidate resource scheduling paths: For example, a computer cluster contains nodes 1, 2, and 3, each with an initial resource capacity of 8 GPUs. Node 1 runs task A (using 2 GPUs) and task B (using 1 GPU), leaving 5 GPUs available. Node 2 runs task C (using 2 GPUs), task D (using 1 GPU), and task E (using 1 GPU), leaving 4 GPUs available. Node 3 runs task F (using 1 GPU) and task G (using 4 GPUs), leaving 3 GPUs available.

[0065] Assuming the primary tasks are Task A and Task B, and the primary resource capacity is 8 GPUs, the scheduling methods for Task A can include scheduling it to Node 2 or Node 3, i.e., migrating Task A to Node 2 or Node 3. Similarly, the scheduling methods for Task B can also include scheduling it to Node 2 or Node 3. The control node can arrange and combine Task A and Task B with their respective scheduling methods to obtain four candidate resource scheduling paths. These four candidate resource scheduling paths are: Candidate Resource Scheduling Path 1: Task A is scheduled to Node 2, and Task B is scheduled to Node 2; Candidate Resource Scheduling Path 2: Task A is scheduled to Node 2, and Task B is scheduled to Node 3; Candidate Resource Scheduling Path 3: Task A is scheduled to Node 3, and Task B is scheduled to Node 2; Candidate Resource Scheduling Path 4: Task A is scheduled to Node 3, and Task B is scheduled to Node 3.

[0066] In this embodiment, the second resource capacity occupied by the first task currently being executed by each of multiple nodes is obtained. Based on the second resource capacity occupied by the first task, the first resource capacity, and the characteristic parameters of the first task, multiple candidate resource scheduling paths are obtained. By considering the characteristic parameters of the first task, the failure of the first task due to blindly scheduling an uninterruptible first task is avoided.

[0067] In some embodiments, the control node periodically defragments resources. For example, the control node may pre-store the defragmentation cycle for resource fragments, such as a defragmentation cycle of 12 hours or 24 hours.

[0068] In some embodiments, when the resource fragmentation conditions are met, the control node simulates resource scheduling based on the first resource capacity and the currently executing first task to obtain multiple candidate resource scheduling paths for resource fragmentation. The fragmentation conditions may include: the computer cluster receives a second task to be executed, and the waiting time of the second task is greater than a first duration threshold; and / or, the degree of resource dispersion in the computer cluster is greater than a degree threshold. The first duration threshold and the degree threshold are pre-stored by the control node. Therefore, the control node can promptly perform resource fragmentation after accurately identifying the scheduling bottleneck caused by fragmented resources.

[0069] This application's embodiments trigger resource fragment defragmentation through periodicity and / or by meeting various defragmentation conditions, enabling proactive defragmentation and ensuring timely defragmentation. Furthermore, the support for multiple triggering conditions enhances the flexibility and intelligence of the defragmentation process.

[0070] For situations where the waiting time for the second task to be executed exceeds the first time threshold, the control node can start timing from the moment the second task is received to determine the waiting time of the second task.

[0071] The conditions for sorting include: when the degree of dispersion is greater than a certain threshold, such as... Figure 3 As shown, the process by which the control node obtains the degree of resource dispersion in the computer cluster may include: S31: Obtain the percentage of unavailable resources in the computer cluster.

[0072] The percentage of unavailable resources (also known as resource fragmentation rate) refers to the proportion of idle resources that cannot be effectively utilized out of the total idle resources. These idle resources that cannot be effectively utilized are the sum of resource fragments in the computer cluster.

[0073] Assuming the resource is a GPU, the proportion F of unavailable resources can satisfy the following formula (1): (1) In formula (1), N is the total number of various tasks, and the number of GPUs occupied by any two tasks is different. This represents the actual utilization rate of various tasks. R i The actual runnable amount is the actual number of tasks T that can be run after considering the distribution of idle GPUs across various nodes. i The quantity. That is, R. i To utilize the available GPUs on each node of a computer cluster to run tasks T i The quantity. T i This represents a task that uses i GPUs. i can be an integer greater than or equal to 1.

[0074] E i The theoretical maximum number of executable tasks is T, which is determined without considering the distribution of idle GPUs across nodes. i The quantity. That is, E i The number of tasks T that can be run by all A currently idle GPUs is... i The quantity. The E i You can divide A by T i And then round down to get the result, i.e. If E i If E is 0, then when calculating the percentage of unavailable resources, E i and R i All values ​​are 1.

[0075] For example, if there are 4 idle GPUs, but they are distributed across 4 different nodes, with 1 idle GPU on each node, then the R of "task T2 that requires 2 GPUs" is... i =0 (because there are no nodes that can be combined to create 2 free GPUs), while E i =2.

[0076] S32: If it is determined that the proportion of unavailable resources is greater than the proportion threshold, then it is determined that the degree of resource dispersion in the computer cluster is greater than the degree threshold.

[0077] Since the degree of resource dispersion is positively correlated with the proportion of unavailable resources, the control node can determine that the degree of resource dispersion in the computer cluster exceeds a certain threshold when it determines that the proportion of unavailable resources is greater than a certain threshold. This certain threshold is pre-stored by the control node.

[0078] In this embodiment, the control node determines that the time taken to obtain a candidate resource scheduling path exceeds a third time threshold, and then terminates the current operation. The control node starts timing from the beginning of the simulated resource scheduling. If, when the time taken exceeds the third time threshold, no candidate resource scheduling path has been obtained, then the acquisition of candidate resource scheduling paths ends, which means the simulated resource scheduling ends. This third time threshold is pre-stored by the computer cluster.

[0079] This avoids a situation where a new task is waiting to be executed in the computer cluster, but the cluster takes a long time to obtain candidate resource scheduling paths, preventing the task from being added to the simulated scheduling task queue and causing the task to wait for a long time.

[0080] Figure 4 This is a flowchart illustrating another resource fragmentation defragmentation method provided in an embodiment of this application. Figure 4 As shown, Figure 4 Is Figure 1Based on the illustrated embodiment, one possible implementation of S12 is as follows: S121: Obtain the scheduling cost of each candidate resource scheduling path.

[0081] For each candidate resource scheduling path, the control node can obtain the number of first tasks to be scheduled (i.e., the target first task mentioned above) included in the candidate resource scheduling path, the interruption sensitivity of the first tasks to be scheduled, the similarities and differences between the destination nodes of any two first tasks to be scheduled whose correlation is greater than a threshold, and the scheduling cost of the first tasks to be scheduled. Then, based on the number of first tasks to be scheduled, the interruption sensitivity of the first tasks to be scheduled, the similarities and differences between the destination nodes of any two first tasks to be scheduled whose correlation is greater than a threshold, and the scheduling cost of the first tasks to be scheduled, the control node can obtain the scheduling cost of each candidate resource scheduling path.

[0082] Among them, the scheduling cost of the candidate resource scheduling path is positively correlated with the number of first tasks to be scheduled, the interrupt sensitivity of the first tasks to be scheduled, and the scheduling cost of the first tasks to be scheduled. Moreover, compared with the case where the destination nodes of the two first tasks to be scheduled are different and the correlation degree is greater than the degree threshold, the scheduling cost of the candidate resource scheduling path is lower when the destination nodes of the two first tasks are the same.

[0083] In other words, assuming that the candidate resource scheduling path includes two highly correlated first tasks to be scheduled, then if the number of first tasks to be scheduled in the candidate resource scheduling path, the interrupt sensitivity of each first task to be scheduled, and the scheduling cost remain unchanged, the scheduling cost of the candidate resource scheduling path is lower when the destination nodes of the two highly correlated first tasks are the same.

[0084] The interrupt sensitivity of a task refers to the extent to which an interruption during its execution would affect the task itself. For example, a high-sensitivity task is one where interruption could lead to data loss, resulting in a significant cost. A low-sensitivity task, on the other hand, can be resumed by restarting the task after an interruption, with a relatively low cost.

[0085] The degree of correlation between tasks refers to the degree of interdependence between tasks, also known as resource affinity. For example, the correlation between two subtasks of a task is high, while the correlation between different tasks is low.

[0086] The scheduling cost of a task is positively correlated with its data locality dependency. That is, the higher the data locality dependency of the first task to be scheduled, the higher its scheduling cost. Data locality dependency refers to the degree to which a task's execution depends on the local data of a node. For example, if the execution of a first task to be scheduled does not depend on local data—for instance, if the data used by the first task can be obtained through data sharing—then the scheduling cost of the first task to be scheduled is low due to its low data locality dependency.

[0087] Therefore, this method considers a relatively comprehensive range of parameters to determine the migration cost of candidate resource scheduling paths, thus ensuring a high degree of accuracy in determining the migration cost.

[0088] Understandably, the control node can simulate the scheduling of the first task to obtain its interrupt sensitivity and scheduling cost. When the computer cluster receives a task, it simultaneously receives the task's configuration file. This static configuration file includes tasks closely related to the current task.

[0089] Optionally, the control node pre-stores a scheduling cost function. The control node can determine the scheduling cost of the candidate resource scheduling path based on the scheduling cost function, the number of the first task, the interrupt sensitivity, the similarities and differences of the destination node, and the scheduling cost of the first task.

[0090] S122: Determine the candidate resource scheduling path with the minimum scheduling cost as the target resource scheduling path.

[0091] After obtaining the scheduling costs of each candidate resource scheduling path, the control node determines the candidate resource scheduling path with the lowest scheduling cost as the target resource scheduling path.

[0092] Since the candidate resource scheduling path with the lowest migration cost can be determined as the target resource scheduling path, it can be ensured that resource scheduling based on the target resource scheduling path can not only reduce the impact of migration on tasks, but also reduce the impact of resource reallocation on the computer cluster. Thus, the stable operation of the computer cluster is guaranteed while realizing resource defragmentation.

[0093] Optionally, a virtualized scheduling environment can be built in the control node. This virtualized scheduling environment can reproduce the resource usage of the real computer cluster and synchronize all the first tasks to be scheduled. That is, this virtualized scheduling environment is a virtual scheduling sandbox that is completely consistent with the real cluster environment of the computer cluster, and can be regarded as a virtual "shadow cluster" of the computer cluster. The control node can simulate resource scheduling in this virtualized scheduling environment to obtain multiple candidate resource scheduling paths, and obtain the scheduling costs of multiple candidate resource scheduling paths by rehearsing task scheduling, thereby determining the target resource scheduling path (i.e., the optimal scheduling scheme).

[0094] Therefore, the method provided in this application can simulate and predict scheduling behavior in a virtualized scheduling environment before migrating tasks; and can find the optimal scheduling rearrangement scheme through iterative search and multidimensional cost function evaluation. Thus, on the one hand, it ensures that each target task migrated from the source node has a corresponding destination node, avoiding the risk of task interruption caused by blind eviction; on the other hand, it guarantees that all scheduling operations have feasibility verification and cost controllability; furthermore, it enables "zero-cost trial and error" of defragmentation operations, avoiding disruption to the operation of the computer cluster. Moreover, since the entire process is carried out entirely in a "shadow cluster," it will not cause any interference to the actual tasks.

[0095] Figure 5 This is a flowchart illustrating another resource fragmentation defragmentation method provided in an embodiment of this application. Figure 5 As shown, Figure 5 Is Figure 1 Based on the illustrated embodiment, before executing S13, the following may also be included: S51: Based on the target resource scheduling path, lock resources of the first resource capacity in the first node and resources of the second resource capacity in the second node.

[0096] The target resource scheduling path indicates that the first task to be scheduled in the first node will be scheduled from the first node to a second node among multiple nodes, so that the remaining resource capacity of the first node reaches the first resource capacity. The first node is the source node of the first task to be scheduled included in the target resource scheduling path, and the second node is the destination node of the first task.

[0097] In some embodiments, if a control node determines that a second task exists to be executed, it can lock resources of a first resource capacity in a first node and resources of a second resource capacity in a second node, based on a target resource scheduling path. This ensures high resource utilization within the computer cluster.

[0098] In some embodiments, the control node may, based on the target resource scheduling path, create a first placeholder instance of a first resource capacity in a first node to lock the resources of the first resource capacity in the first node; and create a second placeholder instance in a second node to lock the resources of the second resource capacity in the second node.

[0099] The first (or second) placeholder instance is an empty task, meaning it does not carry any workload and therefore does not affect the node load. Once created, the first (or second) placeholder instance signifies that the corresponding resource capacity on the first (or second) node is "occupied." In other words, the placeholder instance has high priority and exclusivity, ensuring that other scheduling requests cannot be scheduled on the resources it occupies.

[0100] The identifier of the first placeholder instance is the same as the identifier of the second task to be executed, and the identifier of the second placeholder instance is the same as the identifier of the first task to be scheduled. The identifier of each placeholder instance, both the first and second, is used to uniquely identify that placeholder instance among multiple placeholder instances. The identifier of the first task is used to uniquely identify that first task among multiple first tasks. For example, the identifier of the first task can be a number.

[0101] Understandably, during the process of scheduling the first task to the second node, the control node can identify a second placeholder instance with the same identifier as the first task based on the identifier, and allocate the resources occupied by the second placeholder instance to the first task. This allows the first task to be successfully scheduled to run on the second node. Furthermore, after scheduling the first task to the second node, the control node can clear the second placeholder instance.

[0102] In this embodiment, after obtaining the expected resources (i.e., resources of the first resource capacity in the first node and resources of the second resource capacity in the second node) through simulated scheduling, atomic locking of these expected resources using placeholder instances effectively ensures that the expected resources will not be preempted by other tasks during task scheduling. That is, resources of the first resource capacity can only be used by the second task, and resources of the second resource capacity can only be used by the scheduled first task. Therefore, on the one hand, this ensures the consistency between the scheduling result after scheduling according to the target resource scheduling path and the target resource scheduling path, guaranteeing the determinism, reproducibility, and controllability of the scheduling result, thereby improving the reliability of resource fragmentation. On the other hand, it avoids scheduling race conditions and resource backfilling failures, thus preventing situations where the second task and the scheduled first task cannot be executed, and thus avoiding invalid scheduling.

[0103] After locking the resources of the first resource capacity and the resources of the second resource capacity, one possible implementation of S13 is as follows: The control node can release the resources occupied by the first task on the first node. Simultaneously, the control node can create a replica task on the second node corresponding to the first task to be scheduled. This replica task is completely identical to the first task; it can be understood as a copy of the first task. Furthermore, this replica task can automatically identify the resources locked on the second node for the first task to run and run based on those resources. This ensures consistency between resource utilization and the scheduling path, thereby achieving accurate scheduling of the first task from the first node to the second node.

[0104] Figure 6 This is a flowchart illustrating another resource fragmentation defragmentation method provided in an embodiment of this application. Figure 6 As shown, Figure 6 Is Figure 1 Based on the illustrated embodiment, after executing S13, the following may also be included: S61: If at least one first task fails to be scheduled, the first task that has been successfully scheduled will be scheduled to the source node of the first task, and the resources occupied by the first task in the destination node of the first task will be released.

[0105] The control node can detect whether there is a first task that failed to be scheduled among the first tasks to be scheduled. If at least one first task fails to be scheduled, the control node can reschedule the first task that has been successfully scheduled to the source node of the first task and release the resources occupied by the first task in the destination node of the first task.

[0106] In other words, if a first task awaiting scheduling fails to be scheduled due to an anomaly, the control node can automatically cancel the scheduling of other successfully scheduled first tasks, restoring the computer cluster's system state to its state before resource defragmentation. This reduces the frequency of manual intervention, supporting unattended scheduling optimization. Furthermore, it enables closed-loop control and self-healing of resource defragmentation, preventing the computer cluster's resources from being locked for extended periods due to partial task blocking, thus ensuring the smooth operation of the computer cluster and guaranteeing task continuity. This cancellation mechanism is particularly important in complex multi-tenant cluster environments.

[0107] In some embodiments, for each first task to be scheduled in the target resource scheduling path, the control node can obtain the scheduling result of the first task. The scheduling result is either scheduling success or scheduling failure.

[0108] In some embodiments, after the control node schedules the first task that has been scheduled (i.e., the first task that has been successfully scheduled) to the node before scheduling (i.e. the source node), it can also clear the second placeholder instance corresponding to the first task.

[0109] Optionally, if the control node determines that resource defragmentation has not been completed within the second time threshold after the defragmentation conditions have been met, then the resource defragmentation can be terminated. This avoids excessive consumption of computing resources during the defragmentation process, which could prevent new tasks from being scheduled, thus ensuring the continuity of task execution.

[0110] Optionally, the method provided in this application embodiment has extremely high pluggability and configurability. Specifically, users can specify which queues in the cluster support defragmentation, which task types (e.g., recoverable tasks) are allowed to be migrated, and simulate scheduling cycles by controlling the plugin parameters of the scheduler in the control node. Furthermore, the scheduling cost function, the second duration threshold (also known as the rehearsal window size), and the maximum number of tasks that can be evicted can all be flexibly adjusted according to the task scenario, exhibiting good scalability. Different task scenarios include different types of tasks.

[0111] Resource fragmentation refers to the dispersed and underutilized small-capacity resources provided by a computer cluster. These small-capacity resources are insufficient to meet the resource requirements of large new tasks, preventing their execution. Large new tasks are those with rigid requirements for resource continuity and centralization, which cannot be executed by splitting resources across nodes.

[0112] Currently, resource allocation prioritizes assigning new tasks to nodes with limited remaining resource capacity. This concentrates idle resources across a few underutilized nodes, mitigating the generation of new resource fragmentation. However, as the number of tasks increases, resource fragmentation also increases, potentially leading to situations where excessive fragmentation fails to meet the scheduling demands of large new tasks. Therefore, this method of mitigating new resource fragmentation at its source to meet the resource requirements of large new tasks has relatively low reliability.

[0113] To improve reliability, related technologies utilize a descheduler mechanism to identify nodes with low resource utilization in a computer cluster and proactively reschedule tasks from these nodes to nodes with higher resource utilization. This involves actively evicting one or more tasks from the low-utilization node. Consequently, the low-utilization node gains access to a large, continuously available pool of idle resources for new task execution. The control node then finds a suitable new node for the evicted tasks.

[0114] However, the methods in related technologies have the following problems: First, blindness and uncertainty. Specifically, the descheduler mechanism is "blind" when performing eviction operations; it only "kicks out" tasks but cannot predict or guarantee the outcome after the eviction operation. This can lead to compaction failures and ineffective compaction. Compaction failure means that after the descheduler evicts an instance, the overall resource status of the computer cluster may not improve. In some cases, because there are no other suitable nodes to run the evicted task, the evicted task itself cannot be rescheduled, changing from a running state to a waiting state, causing service interruption. Ineffective compaction means that the resource capacity occupied by the task evicted by the descheduler is far less than the resource capacity required by the new task to be executed. For example, the descheduler may successfully migrate a task occupying 1 GPU from node A to node B, but if the new task to be executed requires a full 8 GPUs, then this compaction operation is ineffective and causes task interference.

[0115] II. Non-atomicity and Scheduling Race Conditions. Specifically, in the descheduler mechanism, eviction and scheduling are two independent and asynchronous processes. Non-atomicity means that between eviction and scheduling, the current task's scheduling may be interrupted by other higher-priority tasks. Scheduling race conditions refer to multiple tasks competing for the same resource. In the descheduler mechanism, there is a time window from the eviction of a task and the release of its resources to the scheduling of a new task and the release of its resources. During this window, the released resources may be "preempted" by other unrelated, higher-priority tasks in the computer cluster. Furthermore, multiple evictioned tasks may compete for the same resource, causing evictioned tasks that fail to preempt the resource to be unable to restart due to scheduling failure. This leads to a large number of tasks failing to preempt resources in the computer cluster, resulting in instability in the cluster's state.

[0116] Third, it lacks global optimization and cost awareness. Specifically, the decision-making logic of the scheduler mechanism is relatively simple. For example, it only bases resource scheduling on the resource utilization threshold of a single node, that is, it schedules tasks on nodes with low resource utilization to nodes with high resource utilization. For example, it may evict a high-priority task while retaining a low-priority task that can be started and stopped at any time. As a result, the cost of reorganizing resource fragmentation is high, and the impact on tasks is significant.

[0117] The method provided in this application embodiment simulates resource scheduling based on a first resource capacity and a currently executing first task to obtain multiple candidate resource scheduling paths; determines a target resource scheduling path from the multiple candidate resource scheduling paths; and schedules the first task based on the target resource scheduling path to perform resource fragmentation.

[0118] By simulating resource scheduling to obtain the target resource scheduling path, a suitable destination node can be found for the evicted tasks, thus solving the problems of blindness and uncertainty in related technologies. This avoids task interruptions caused by blindly evicting tasks and ensures that all evicted tasks can be scheduled to other nodes for re-running, thereby avoiding failed or ineffective cleanup.

[0119] When determining candidate resource scheduling paths, the initial resource capacity, the second resource capacity of all migrated first tasks in the computer cluster, and the initial resource capacity of each node are considered. This allows for planning resource scheduling paths from a global perspective. Furthermore, by calculating the cost of each scheduling path and identifying the candidate resource scheduling path with the lowest cost as the target resource scheduling path, the optimal resource scheduling scheme with minimal interference to tasks can be determined—the globally optimal target resource scheduling path. In other words, an optimal scheduling scheme can be found that achieves the resource fragmentation goal while minimizing interference with tasks. This effectively reduces the cost of resource fragmentation reorganization and minimizes the impact on tasks.

[0120] In the process of scheduling the first task based on the target resource scheduling path, by setting placeholder instances to atomically lock the resources, we can avoid the situation where the sorted resources are used by other tasks, and at the same time avoid the situation where multiple migrated first tasks compete for the same resource, causing the scheduling of the first task to fail. This solves the problems of non-atomicity and scheduling race conditions.

[0121] In summary, by using the resource scheduling method provided in this application to defragment resources, on the one hand, the average resource utilization of each node in the computer cluster can be improved; on the other hand, the waiting time of tasks can be shortened and the efficiency of task execution can be improved; and on the other hand, the service capacity boundary of the computer cluster can be expanded.

[0122] Figure 7This application provides a schematic diagram of a resource defragmentation device. The device includes a simulation module 701, a determination module 702, and a scheduling module 703. The simulation module 701 simulates resource scheduling based on a first resource capacity and a currently executing first task, obtaining multiple candidate resource scheduling paths. After defragmenting resource fragments using each candidate resource scheduling path, at least one node's remaining resource capacity reaches the first resource capacity. The determination module 702 determines a target resource scheduling path from the multiple candidate resource scheduling paths. The scheduling module 703 schedules the first task based on the target resource scheduling path to perform resource defragmentation.

[0123] Optionally, the simulation module 701 is specifically used to obtain the second resource capacity occupied by the first task currently being executed by each of the multiple nodes; based on the second resource capacity, the first resource capacity, and the characteristic parameters of the first task, simulate resource scheduling to obtain multiple candidate resource scheduling paths, wherein the characteristic parameters include: the priority, interruptibility, and recoverability of the first task.

[0124] Optionally, the simulation module 701 is specifically used to determine at least two target first tasks from multiple first tasks based on the characteristic parameters of the first task, wherein each target first task has a priority lower than the priority threshold, is interruptible and recoverable; determine the scheduling method of the target first tasks based on the second resource capacity, the first resource capacity, and the initial resource capacity of each node occupied by each target first task; and arrange and combine the scheduling methods of all target first tasks to obtain multiple candidate resource scheduling paths.

[0125] Optionally, the simulation module 701 is also used to take snapshots of multiple nodes to obtain the second resource capacity occupied by the first task currently being executed by each of the multiple nodes.

[0126] Optionally, the determining module 702 is specifically used to obtain the scheduling cost of each candidate resource scheduling path; and to determine the candidate resource scheduling path with the lowest scheduling cost as the target resource scheduling path.

[0127] Optionally, the determining module 702 is specifically used to, for each candidate resource scheduling path, obtain the number of first tasks to be scheduled in the candidate resource scheduling path, the interruption sensitivity of the first tasks to be scheduled, the similarities and differences of the destination nodes of any two first tasks to be scheduled whose correlation is greater than the degree threshold, and the scheduling cost of the first tasks to be scheduled; based on the number of first tasks to be scheduled, the interruption sensitivity of the first tasks to be scheduled, the similarities and differences of the destination nodes of any two first tasks to be scheduled whose correlation is greater than the degree threshold, and the scheduling cost of the first tasks to be scheduled, obtain the scheduling cost of the candidate resource scheduling path; wherein, the scheduling cost of the candidate resource scheduling path is positively correlated with the number of first tasks to be scheduled, the interruption sensitivity of the first tasks to be scheduled, and the scheduling cost of the first tasks, and compared with the case where the destination nodes of the two first tasks to be scheduled whose correlation is greater than the degree threshold are different, the scheduling cost of the candidate resource scheduling path is lower when the destination nodes are the same.

[0128] Optionally, the target resource scheduling path indicates that the first task in the first node will be scheduled from the first node to the second node among multiple nodes, so that the remaining resource capacity of the first node reaches the first resource capacity; the scheduling module 703 is further configured to: based on the target resource scheduling path, lock the resources of the first resource capacity in the first node, and lock the resources of the second resource capacity in the second node; wherein, the second resource capacity is the resource capacity occupied by the first task in the first node.

[0129] Optionally, the scheduling module 703 is specifically used to create a first placeholder instance of a first resource capacity in a first node based on the target resource scheduling path, so as to lock the resource of the first resource capacity in the first node; and to create a second placeholder instance in a second node, so as to lock the resource of the second resource capacity in the second node.

[0130] Optionally, the scheduling module 703 is further configured to, upon determining that there is a second task to be executed, lock resources of a first resource capacity in the first node and lock resources of a second resource capacity in the second node based on the target resource scheduling path; wherein the resource capacity required by the second task is less than or equal to the first resource capacity.

[0131] Optionally, there may be multiple first tasks, each of which runs on a single node. The scheduling module 703 is also configured to, if at least one first task fails to be scheduled, schedule the successfully scheduled first task to the source node of the first task and release the resources occupied by the first task in the destination node of the first task.

[0132] Optionally, the scheduling module 703 is further configured to simulate resource scheduling based on the first resource capacity and the currently executing first task, provided that the resource fragmentation conditions are met; wherein the fragmentation conditions include: the waiting time of the second task to be executed is greater than a first duration threshold, and / or, the degree of resource dispersion in the computer cluster is greater than a degree threshold.

[0133] Optionally, the sorting conditions include: the degree of resource dispersion in the computer cluster is greater than the degree threshold; the scheduling module 703 is specifically used to obtain the proportion of unavailable resources in the computer cluster; if it is determined that the proportion of unavailable resources is greater than the proportion threshold, then it is determined that the degree of resource dispersion in the computer cluster is greater than the degree threshold.

[0134] Optionally, the scheduling module 703 is specifically used to end resource fragmentation if the duration for which the sorting conditions are met reaches the second duration threshold and resource sorting has not yet been completed, and / or if the acquisition duration of the candidate resource scheduling path is greater than the third duration threshold.

[0135] The apparatus in this embodiment can be used to execute the technical solutions of the above method embodiments. Its implementation principle and technical effect are similar, and will not be repeated here.

[0136] This application embodiment also provides a controller, which includes a processor and a memory. The memory stores programs or instructions that can run on the processor, and when the program or instructions are executed by the processor, they implement... Figures 1 to 6 The steps of the resource fragmentation method shown are as follows.

[0137] This application also provides a control node, which includes the controller provided in this application embodiment.

[0138] This application also provides a computer cluster, which includes the control node as described in the embodiments of this application.

[0139] This application embodiment also provides a computer-readable storage medium storing a program or instructions, which, when executed by a processor, implement... Figures 1 to 6 The steps of the resource fragmentation method shown are as follows.

[0140] This application also provides a computer program product, which, when executed by a processor of a vehicle or a cloud server, implements the following: Figures 1 to 6 The steps of the resource fragmentation method shown are as follows.

[0141] From the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of computer software products plus necessary general-purpose hardware platforms, and of course, they can also be implemented by hardware. The computer software product is stored in a storage medium (such as ROM, RAM, magnetic disk, optical disk, etc.) and includes several instructions to cause the terminal or network-side device to execute the methods described in the various embodiments of this application.

[0142] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other implementations under the guidance of this application without departing from the spirit and scope of the claims. All of these implementations are within the protection scope of this application.

Claims

1. A method for organizing resource fragments, characterized in that, The method is applied to a computer cluster, the computer cluster comprising: multiple nodes; including: Based on the first resource capacity and the first task currently being executed, simulated resource scheduling is obtained to obtain multiple candidate resource scheduling paths. After resource fragmentation is performed using each of the candidate resource scheduling paths, the remaining resource capacity of at least one of the nodes can reach the first resource capacity. The target resource scheduling path is determined from the multiple candidate resource scheduling paths; Based on the target resource scheduling path, the first task is scheduled to perform resource fragmentation.

2. The method according to claim 1, characterized in that, The process of simulating resource scheduling based on the first resource capacity and the currently executing first task yields multiple candidate resource scheduling paths, including: Obtain the second resource capacity occupied by the first task currently being executed by each of the plurality of nodes; Based on the second resource capacity, the first resource capacity, and the characteristic parameters of the first task, resource scheduling is simulated to obtain multiple candidate resource scheduling paths, wherein the characteristic parameters include: the priority, interruptibility, and recoverability of the first task.

3. The method according to claim 2, characterized in that, The simulation of resource scheduling based on the second resource capacity, the first resource capacity, and the characteristic parameters of the first task yields multiple candidate resource scheduling paths, including: Based on the characteristic parameters of the first task, at least two target first tasks are determined from a plurality of first tasks, each of the target first tasks having a priority lower than a priority threshold, being interruptible and recoverable; Based on the second resource capacity occupied by each of the target first tasks, the first resource capacity, and the initial resource capacity of each of the nodes, the scheduling method of the target first tasks is determined; The scheduling methods for all the aforementioned target first tasks are arranged and combined to obtain multiple candidate resource scheduling paths.

4. The method according to claim 2, characterized in that, The step of obtaining the second resource capacity occupied by the first task currently being executed by each of the plurality of nodes includes: Take snapshots of the plurality of nodes to obtain the second resource capacity occupied by the first task currently being executed by each of the plurality of nodes.

5. The method according to any one of claims 1 to 4, characterized in that, The step of determining the target resource scheduling path from the plurality of candidate resource scheduling paths includes: Obtain the scheduling cost of each of the candidate resource scheduling paths; The candidate resource scheduling path with the lowest scheduling cost is determined as the target resource scheduling path.

6. The method according to claim 5, characterized in that, Each candidate resource scheduling path includes: the destination node of the first task to be scheduled; obtaining the scheduling cost of each candidate resource scheduling path includes: For each of the candidate resource scheduling paths, obtain the number of first tasks to be scheduled included in the candidate resource scheduling path, the interruption sensitivity of the first tasks to be scheduled, the similarities and differences of the destination nodes of any two first tasks to be scheduled whose correlation degree is greater than the degree threshold, and the scheduling cost of the first tasks to be scheduled. Based on the number of the first tasks to be scheduled, the interrupt sensitivity of the first tasks to be scheduled, the similarities and differences of the destination nodes of any two first tasks whose correlation is greater than the degree threshold, and the scheduling cost of the first tasks to be scheduled, the scheduling cost of the candidate resource scheduling path is obtained. The scheduling cost of the candidate resource scheduling path is positively correlated with the number of the first tasks to be scheduled, the interruption sensitivity of the first tasks to be scheduled, and the scheduling cost of the first tasks. Moreover, compared with the case where the destination nodes of the two first tasks to be scheduled are different and the degree of correlation is greater than the degree threshold, the scheduling cost of the candidate resource scheduling path is lower when the destination nodes are the same.

7. The method according to any one of claims 1 to 4, characterized in that, The target resource scheduling path indicates that the first task in the first node will be scheduled from the first node to the second node among the plurality of nodes, so that the remaining resource capacity of the first node reaches the first resource capacity. Before scheduling the first task based on the target resource scheduling path, the method further includes: Based on the target resource scheduling path, resources of the first resource capacity are locked in the first node, and resources of the second resource capacity are locked in the second node; The second resource capacity is the resource capacity occupied by the first task in the first node.

8. The method according to claim 7, characterized in that, The step of locking resources of the first resource capacity in the first node and locking resources of the second resource capacity in the second node based on the target resource scheduling path includes: Based on the target resource scheduling path, a first placeholder instance of the first resource capacity is created in the first node to lock the resource of the first resource capacity in the first node; And a second placeholder instance created in the second node to lock the resource of the second resource capacity in the second node.

9. The method according to claim 7, characterized in that, The step of locking resources of the first resource capacity in the first node and locking resources of the second resource capacity in the second node based on the target resource scheduling path includes: If it is determined that there is a second task to be executed, based on the target resource scheduling path, the resources of the first resource capacity are locked in the first node, and the resources of the second resource capacity are locked in the second node; The resource capacity required for the second task is less than or equal to the resource capacity required for the first task.

10. The method according to any one of claims 1 to 4, characterized in that, The number of the first tasks is multiple, and each first task runs on a node; after scheduling the first tasks based on the target resource scheduling path, the method further includes: If at least one of the first tasks fails to schedule, the first tasks that have been successfully scheduled will be scheduled to the source node of the first task, and the resources occupied by the first task in the destination node of the first task will be released.

11. The method according to any one of claims 1 to 4, characterized in that, The simulated resource scheduling based on the first resource capacity and the currently executing first task includes: If the conditions for reorganizing resource fragments are met, then resource scheduling is simulated based on the first resource capacity and the first task currently being executed. The sorting conditions include: the waiting time of the second task to be executed is greater than a first time threshold, and / or the degree of resource dispersion in the computer cluster is greater than a degree threshold.

12. The method according to claim 11, characterized in that, The sorting conditions include: the degree of resource dispersion in the computer cluster is greater than a degree threshold; before simulating resource scheduling based on the first resource capacity and the currently executing first task, the method further includes: Obtain the percentage of unavailable resources in the computer cluster; If it is determined that the proportion of unavailable resources is greater than the proportion threshold, then it is determined that the degree of resource dispersion in the computer cluster is greater than the degree threshold.

13. The method according to claim 11, characterized in that, The method further includes: If the duration for which the sorting conditions are met reaches the second duration threshold and resource sorting has not yet been completed, and / or if the acquisition duration of the candidate resource scheduling path is greater than the third duration threshold, then resource fragment sorting ends.

14. A controller, characterized in that, The controller includes a processor and a memory, the memory storing a program or instructions executable on the processor, the program or instructions being executed by the processor to implement the steps of the resource defragmentation method as described in any one of claims 1 to 13.

15. A control node, characterized in that, The control node includes the controller as described in claim 14.

16. A computer cluster, characterized in that, The computer cluster includes the control node as described in claim 15.

17. A computer-readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the resource defragmentation method as described in any one of claims 1 to 13.

Images