Task migration method of heterogeneous unmanned cluster based on joint optimization

Abstract

The invention discloses a task migration method of a heterogeneous unmanned cluster based on joint optimization. The method comprises the following steps: firstly fitting the residual available rate of each resource in each node into a distribution probability density function conforming to beta distribution, then calculating the residual available value distribution of each resource in each type of nodes in combination with the total resource amount of different types of nodes, and determining the residual available value distribution vector of all resources during cluster operation. And setting an availability rate threshold value for each resource, when the residual availability rate of the resources in the node is lower than the set threshold value, indicating that the node has resource overload, and calculating the migration value of the task k executed by the node. Sequencing all the tasks in the overload node according to the migration values from high to low, searching a target node in sequence, and migrating the tasks from the overload node to the target node until the overload state is quitted. According to the method, the related information of the resource margin of the cluster is jointly considered, so that a matched target node can be found more easily when the task is migrated.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of scheduling management, and relates to task management and migration of an unmanned cluster, in particular to a task migration method for a heterogeneous unmanned cluster based on joint optimization. BACKGROUND

[0002] In a large unmanned cluster, each node has some available resources, and task scheduling is performed through the resources of the nodes. However, the scheduling algorithm may not be balanced when allocating resources, which may cause some nodes to be overloaded and some nodes to be significantly underloaded, resulting in resource waste. The task migration algorithm is to reschedule the allocated tasks during the operation of the cluster, so as to maintain the balanced state of the resource utilization of each node and ensure the normal operation of the tasks.

[0003] Prior art 1 (CN106844051A) proposes a task migration algorithm suitable for an edge computing scenario, which performs simple sorting according to the CPU utilization and temperature of each node to obtain a CPU overload node queue and a migration target node queue. Then, based on the principle of the least number of migrated tasks, tasks are selected on each overload node, and for each task to be migrated, the corresponding target node is selected based on the principle of the shortest migration time to generate a corresponding migration scheme. This method indeed improves the availability of services to some extent, but only considers the load of the CPU and does not consider the impact of the migrated tasks on the target nodes during migration.

[0004] Prior art 2 (CN112087509A) proposes a task migration method for an edge computing platform, which divides the nodes of a server cluster into different types based on the CPU utilization, converts the physical network topology into a complete graph structure, colors the links between nodes based on graph coloring processing, and finally predicts the CPU resource utilization of the server based on a linear regression model to perform task migration operations between different types of nodes. This method discusses the evaluation of the load of the nodes, but does not consider which tasks should be migrated. At the same time, the node load prediction method based on the linear regression model cannot be used in a cluster mainly composed of unmanned nodes.

[0005] Therefore, the existing technical solutions mostly focus on cloud environments composed of server clusters, and the migration idea is relatively simple, which has low industrial applicability. In addition, in order to complete different types of work, different types of nodes may exist in an unmanned cluster, and how to reasonably evaluate the load of these nodes and design a migration scheme is a problem that needs to be solved. SUMMARY

[0006] In view of the deficiencies of the prior art, the present application provides a task migration method for a heterogeneous unmanned cluster based on joint optimization, which considers the value of the task to be migrated and the matching degree after migration on the basis of evaluating the node load, and ensures that the overloaded node can exit the overloaded state and the resource utilization of the target node after migration is more balanced based on joint optimization of the selection of the task and the target node.

[0007] A task migration method for a heterogeneous unmanned cluster based on joint optimization, specifically comprising the following steps: Step 1, determining the configuration information of the unmanned cluster Determine the types of nodes contained in the unmanned cluster and the number of nodes of each type. Determine the total resource number of each type of node.

[0008] Step 2, predicting the residual available rate of node resources Use the scheduling algorithm to run, obtain the residual available rate of each node, then fit it as a beta distribution, obtain the value of parameters α and β, as the distribution probability density function of the residual available rate x of the jth resource j : Where, the residual available rate x of the resource j ∈[0,1], t is the integral variable, α j , β j are the parameters to be fitted, which determine the mean and variance of the beta distribution : Step 3, determining the distribution of the residual available value of node resources Based on the residual available rate distribution probability density function of the resources, combined with the total amount of resources of different types of nodes, the residual available value distribution of various resources in each type of node is calculated, and the specific steps are as follows: s3.1, in the ith type of node, the distribution probability density function of the residual available value x of the jth resource j : Where, ij Indicates the total number of the jth resource in the ith type of node.

[0009] ​s3.2, calculate the residual available numerical value distribution vector of all resources in the ith type of node : Gi() is the joint probability density function of the residual available numerical values x1, x2,... x n of all resources in the ith type of node, and n represents the number of resource types in the ith type of node.

[0010] s3.3, calculate the residual available numerical value distribution vector of all resources in the cluster runtime according to the residual available numerical value distribution vector of the resources of a single type of node : where typeSum represents the number of node types in the cluster, and p(i) represents the frequency of occurrence of the ith type of node.

[0011] Step 4, calculate the task migration value Set an available rate threshold t ij for each resource in the ith type of node to obtain the threshold vector of the ith type of node .

[0012] When the residual available rate of the resources in the node is lower than the set threshold, it indicates that the node is overloaded, and the migration value val(k) of the task k executed by the node is calculated: where γ1 and γ2 are custom constant coefficients. represents the numerical demand of task k for resource j, represents the residual available rate of resource j in the overloaded node. represents the demand numerical value vector of task k for various resources, and the coefficient is used to ensure that the target node for task k migration is not overloaded.

[0013] Step 5, match the target node Sort all tasks in the overloaded node from high to low according to the migration value, and sequentially find the target node according to the order, migrate the task from the overloaded node to the target node, and until the overloaded node exits the overloaded state position.

[0014] where the target node is found by traversing all nodes in the cluster, and the matching degree of the node and the task to be migrated is calculated on the basis that the node migration does not cause the node to be overloaded; for the task k to be migrated, after traversing nodes, the node with the optimal matching degree is selected as the target node for migration, where C represents the total number of nodes in the unmanned cluster.

[0015] As preferred, the calculation method of the matching degree is: wherein, nodeResourceUsage j is the current usage value of the jth type of resource of the node node i’ , nodeResource j is the total value of the jth type of resource of the node node i’ . The node with the lowest matching degree Match(task, node) is selected as the target node, and the task is migrated from the overloaded node to the target node.

[0016] The present application has the following beneficial effects: 1. The present application takes into account the diversity of node types in the cluster, considers the allocation of heterogeneous resources, and can be applied to task migration in a heterogeneous cluster.

[0017] 2. In the process of screening tasks to be migrated, the relevant information of the resource margin of the cluster itself is considered jointly, so that it is easier to find a suitable target node when migrating tasks.

[0018] 3. The task migration process is simple and has low computational complexity, and does not require the use of a neural network model to estimate node resources, has low computing power requirements, and is easy to deploy in an unmanned system cluster.

[0019] 4. For the task to be migrated k, if the value of the item corresponding to γ1 in the migration value val(k) is higher, the probability of finding a node with a high matching degree is higher, so in the process of finding the target node, the task k with a higher migration value val(k) needs to traverse fewer nodes, and can find a suitable target node faster. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 is a schematic diagram of a heterogeneous cluster; Figure 2 is a task migration flowchart. DETAILED DESCRIPTION

[0021] The present application will be further explained and described below in conjunction with the accompanying drawings; A task migration method for a heterogeneous unmanned cluster based on joint optimization, specifically comprising the following steps: Step 1, determining the configuration information of the unmanned cluster determining the types and number of nodes included in the unmanned cluster, as shown in Figure 1 , a unmanned cluster includes three different types of unmanned nodes.

[0022] Determine the total resource number of each type of node, including the number of CPU cores, the total amount of memory (MB), network bandwidth, GPU memory capacity, etc., and abstract it into a vector , where type_i represents the node type.

[0023] Step 2, Estimate the remaining available rate of node resources When all tasks are allocated to nodes in the cluster through the scheduling algorithm, the resource utilization rate of each node is between 0% and 100%. For any kind of resource, a resource utilization distribution law can be formed with utilization rate as the horizontal coordinate and probability density as the vertical coordinate, and the resource remaining available rate opposite to the utilization rate.

[0024] Select the beta distribution to describe the remaining available rate x j of the jth resource during the operation of the cluster : where the remaining available rate x j of the resource ∈ [0, 1], t is the integral variable. α j , β j are parameters to be fitted, and the parameters α and β determine the mean and variance of the beta distribution : For a well-matched task set and cluster, the mean of the remaining available rate of resources should be close to 0, and if the scheduling algorithm is good, the variance should be low. Therefore, the values of parameters α and β will be different due to the matching between the task set and the cluster executing the task set, as well as the selection of the scheduling algorithm. In engineering practice, in order to make the final scheduling result more consistent with the beta distribution, when directly predicting, for an excellent scheduling algorithm and a task set that is well matched with the cluster, a larger α+β value should be selected, and α should be much smaller than β. For a fixed task set, the resource remaining available rate of each idle node can be obtained by trial running using the scheduling algorithm, and then fitted as a beta distribution, so as to directly calculate the true parameters α and β.

[0025] Step 3, Determine the distribution of the remaining available rate of node resources The decision whether to migrate a task between two nodes is not based on the resource occupancy rate of the task on the overloaded node and the resource remaining availability rate of the target node, but on the specific resource usage value of the task and the resource remaining availability value of the target node. Therefore, the resource remaining availability rate distribution probability density function is needed to calculate the resource remaining availability value distribution of each type of node in combination with the total amount of resources of different types of nodes, so as to obtain the resource remaining availability value distribution of the nodes in the cluster operation, and the specific steps are as follows: s3.1, based on the resource remaining availability rate distribution probability density function It can be seen that in the ith type of node, the distribution probability density function of the resource j remaining availability value x j is: wherein, ij represents the total number of resources j in the ith type of node.

[0026] s3.2, calculate the resource remaining availability value distribution vector of all resources in the ith type of node : G i is the joint probability density function of all resource remaining availability values x1, x2,...x n in the ith type of node, and n represents the number of resource types in the ith type of node. The parameters x1, x2,...x n of Gi are the parameters input to the joint probability density function. As long as the resource remaining availability values in the ith type of node are given, the corresponding probability density distribution can be obtained as the size of the possibility of finding similar resource availability value vectors in the ith type of node.

[0027] s3.3, according to the resource remaining availability value distribution vector of a single type of node, calculate the resource remaining availability value distribution vector of all resources in the cluster operation : wherein typeSum represents the number of node types in the cluster, and p(i) represents the frequency of the ith type of node.

[0028] According to the resource remaining availability value distribution vector of all resources , the sparsity of the distribution of the remaining available resources in each node in the cluster can be obtained, so that the probability of matching the tasks selected for migration on the overloaded node to the target nodes with better similarity is greater.

[0029] Step 4, calculate the task migration value ​Set an availability threshold t for each resource in the i-th type of node. ij The threshold vector of the i-th type of node is obtained. The availability threshold t ij It should be given in terms of availability and designed according to actual needs. For example, when there is a strict requirement that each node has a large amount of resources available, the availability threshold t can be set. ij Set it higher, such as 20% to 30%. When the task set puts a lot of pressure on the cluster, it can be set lower, such as 5% to 10%, to prevent tasks from constantly migrating between nodes without making progress, or even wasting available resources and increasing migration costs.

[0030] like Figure 2 As shown, when the remaining availability of resources in a node falls below a set threshold, it indicates that the node is overloaded and a task needs to be migrated. During task selection, it's necessary to consider whether a suitable node can perfectly accommodate the task, and the contribution of migrating the task to the node's exit from or removal from the overload state. This is determined using the following migration value function: Where val(k) represents the migration value of task k. This represents the numerical requirement of resource j for task k. This represents the remaining availability of resource j within the overloaded node. This represents a numerical vector representing the resource requirements of task k. It is a constant greater than the availability threshold, used to ensure that the migration of task k will not cause the target node to become overloaded.

[0031] In the migration value function, Used to estimate the probability that task k matches the migration target node. The parameters used to evaluate the impact of the migration of task k on the current overloaded node are γ1 and γ2, which are custom constant coefficients. In this embodiment, γ1=0.7 and γ2=0.3.

[0032] Step 5: Match target nodes Because tasks with higher value are more likely to find target nodes and are more likely to find perfectly matching target nodes, all tasks in the overloaded node are sorted from high to low migration value, and target nodes are searched in order. Tasks are migrated from the overloaded node to the target node until the overloaded node exits the overloaded state.

[0033] Because the term corresponding to γ1 in the migration value function val(k) ensures a high enough probability density, there is a high probability of finding a target node corresponding to the "remaining resource available value" to perform matching, so all resource available nodes in the cluster can be traversed to select the target node, and the traversal depth will not be too large.

[0034] For the task k to be migrated, traverse nodes, and calculate the matching degree between the task k to be migrated on the overloaded node i and the resource available node node i’ , select the node with the lowest matching degree Match(task, node) as the target node, and migrate the task k from the overloaded node i to the target node: where C represents the total number of nodes in the unmanned cluster, nodeResourceUsage j is the current usage value of the jth type of resource on the node node i’ , and nodeResource j is the total value of the jth type of resource on the node node i’ .

Claims

1. A task migration method for heterogeneous unmanned clusters based on joint optimization, characterized in that: Specifically, the following steps are included: Step 1: Determine the types of nodes in the unmanned cluster and the number of nodes of each type; determine the total number of resources for each type of node. Step 2: Run a trial using a scheduling algorithm to obtain the remaining resource availability rate of each node, and then fit it to a beta distribution as the prediction of the remaining availability rate x of the j-th resource. j probability density function α j β j The fitting parameters are used to determine the mean and variance of the beta distribution; Step 3: Based on the probability density function of the remaining availability distribution of resources By combining the total resources of different types of nodes, the remaining available numerical distribution vector of all resources during cluster runtime is calculated. : Where typeSum represents the number of node types in the cluster, and p(i) represents the frequency of occurrence of the i-th type of node; It is the distribution vector of the remaining available values ​​of all resources in the i-th type of node: n represents the number of resource types in the i-th type of node. ij This represents the total number of resources j in the i-th type of node; Step 4: Set an availability threshold t for each resource in the i-th type of node. ij When the remaining availability of resources in a node is lower than a set threshold, it indicates that the node is overloaded. Calculate the migration value val(k) of task k executed by that node: Where γ1 and γ2 are user-defined constant coefficients; This represents the numerical requirement of resource j for task k. This represents the remaining availability of resource j within the overloaded node; This represents a numerical vector of the resource requirements of task k, with coefficients... This is used to ensure that the target node for task k migration is not overloaded; Step 5: Sort all tasks in the overloaded node according to their migration value from high to low, find the target node in order, and migrate the tasks from the overloaded node to the target node until the overloaded node exits the overloaded state. The process of finding the target node involves traversing all nodes in the cluster and calculating the matching degree between the node and the task to be migrated, ensuring that migrating the task will not cause node overload. For the task k to be migrated, the process involves traversing... After a certain number of nodes, the node with the best matching degree is selected as the target node for migration, where C represents the total number of nodes in the unmanned cluster.

2. The task migration method for heterogeneous unmanned clusters based on joint optimization as described in claim 1, characterized in that: Determine the total number of resources for each type of node, including the number of CPU cores, total memory, network bandwidth, GPU memory capacity, etc., and abstract it into a vector. , where type_i represents the node type.

3. The task migration method for heterogeneous unmanned clusters based on joint optimization as described in claim 1, characterized in that: The availability threshold t ij Provided in terms of availability.

4. The task migration method for heterogeneous unmanned clusters based on joint optimization as described in claim 3, characterized in that: When the task set puts significant pressure on the cluster, the availability threshold should be set between 5% and 10%; when each node needs a large amount of available resources, the availability threshold t should be set. ij The range is 20% to 30%.

5. The task migration method for heterogeneous unmanned clusters based on joint optimization as described in claim 1, characterized in that: Set γ1=0.7 and γ2=0.

3.

6. The task migration method for heterogeneous unmanned clusters based on joint optimization as described in claim 1, characterized in that: The matching degree is calculated as follows: Among them, nodeResourceUsage j For node i’ The current usage value of the j-th type of resource, nodeResource j For node i’ The total value of the j-th type of resources; select the node with the lowest match degree Match(task, node) as the target node for task k migration.

7. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method of any one of claims 1 to 6.

Images