Reinforcement learning scheduling method and system based on resource prediction and action mutation

Abstract

The invention provides a reinforcement learning scheduling method and system based on resource prediction and action mutation in a heterogeneous cloud environment. The method comprises the steps of obtaining the number of tasks in a task waiting queue; if the number reaches the preset number threshold value, obtaining current state information of the node, and performing prediction based on a pre-trained prediction network to obtain state information of an execution moment; performing feature extraction and coding to obtain a node state vector and a task feature vector; merging to obtain a merged vector and inputting the merged vector into a reinforcement learning model to generate a decision strategy; and assigning tasks to the node mutation of the target type for the decision strategy based on a preset probability to obtain a final decision strategy. According to the embodiment of the invention, the node state can be predicted through the prediction network, and then the node state and the task demand are input into the reinforcement learning framework to obtain the scheduling decision, so that the task completion rate and the resource utilization rate of the heterogeneous cloud computing system are improved, and the energy consumption is reduced.

Description

Technical Field

[0001] The present invention relates to the field of cloud computing technology, and in particular to a reinforcement learning scheduling method and system based on resource prediction and action mutation in a heterogeneous cloud environment. Background Art

[0002] Heterogeneous cloud computing is a computing method that combines computing units with different types of instruction sets and architectures, such as CPU (Central Processing Unit), GPU (Graphics Processing Unit), etc. The application of this computing method on cloud computing platforms can improve performance and energy efficiency. The system faces multiple challenges such as node heterogeneity, unique task requirements and device diversity. Existing scheduling methods cannot achieve satisfactory performance and service quality. Summary of the invention

[0003] To solve the above problems, an embodiment of the present invention provides a reinforcement learning scheduling method based on resource prediction and action mutation in a heterogeneous cloud environment, including: obtaining the number of tasks in a task waiting queue, and judging whether the number reaches a preset number threshold; if the preset number threshold is reached, obtaining the current state information of the node, and predicting the current state information of the node based on a pre-trained prediction network to obtain the state information of the node at the execution time; extracting and encoding the state information of the node at the execution time and the task feature information of the tasks in the task waiting queue, respectively, to obtain a node state vector and a task feature vector; merging the node state vector and the task feature vector to obtain a merged vector, and inputting the merged vector into a reinforcement learning model to generate a decision strategy; assigning tasks to node mutations of the target type based on the preset probability of the decision strategy to obtain a final decision strategy.

[0004] The reinforcement learning scheduling method based on resource prediction and action mutation in a heterogeneous cloud environment provided by the embodiment of the present invention can predict the node status by predicting the network, and then input the node status and task requirements into the reinforcement learning framework to obtain a scheduling decision, thereby improving the task completion rate and resource utilization rate of the heterogeneous cloud computing system and reducing energy consumption.

[0005] Optionally, the method further includes: if the waiting time of the task waiting queue reaches a preset time threshold, obtaining the current state information and task characteristic information of the node; and generating a scheduling decision plan using a preset scheduling strategy based on the current state information and task characteristic information.

[0006] In the embodiment of the present invention, the preset scheduling strategy is used for scheduling at regular intervals, which can maintain the activity of the system, prevent resources from being idle, and ensure that the system can maintain a certain stability and responsiveness when facing unstable external task requirements.

[0007] Optionally, the prediction network is a long short-term memory (LSTM) neural network; the method also includes: obtaining historical status information of the node at fixed time intervals, and normalizing the historical status information; constructing a time series with a time window for the normalized historical status information, and then using seasonal and trend decomposition (STL) to obtain three components of season, trend and residual; inputting the three components into the long short-term neural network to obtain the predicted value of each component, and determining the sum of the predicted values ​​of each component to obtain the predicted value of the status information at the target moment; comparing the predicted value of the status information at the target moment with the true value at the target moment to obtain the mean square error, and updating the long short-term neural network based on minimizing the mean square error.

[0008] The embodiment of the present invention provides a specific process for training a long-term and short-term neural network. The state information of the node at the execution time can be predicted based on the trained long-term and short-term neural network to avoid the adverse effect on scheduling caused by the change of the node state due to communication delays, so that the node state used by the method at the decision time is more in line with the execution time.

[0009] Optionally, the state information of the node at the time of execution and the task characteristic information of the tasks in the task waiting queue are respectively feature extracted and encoded to obtain a node state vector and a task characteristic vector, including: extracting the state information of the node at the time of execution to obtain a node state vector of length N, and extracting the task characteristic information of the tasks in the task waiting queue to obtain an M-dimensional task characteristic vector; encoding the node state vector of length N and the M-dimensional task characteristic vector to obtain a node state vector and a task characteristic vector of the same length.

[0010] In the embodiment of the present invention, feature extraction and encoding are performed on the node state and the task respectively, and the two vectors can be encoded into vectors with the same length.

[0011] Optionally, the reinforcement learning model is an Actor-Critic algorithm model, and the step of inputting the merged vector into the reinforcement learning model to generate a decision strategy includes: inputting the merged vector into an Actor network to output a probability distribution matrix A; performing maximum probability sampling on the probability distribution matrix A, determining the node corresponding to the maximum value of each row of the probability distribution matrix A as the scheduling node of the task, and obtaining a decision strategy.

[0012] In the embodiment of the present invention, the Actor-Critic network is used to generate initial scheduling decisions, and excellent scheduling decisions are made within an acceptable time, which helps to improve the performance of heterogeneous cloud computing systems.

[0013] Optionally, the method further includes: using a Critic network to evaluate the merged vector to obtain an evaluation score V.

[0014] In the embodiment of the present invention, evaluation is performed based on the Critic network, so as to modify the weight parameters of the network so that it can adapt to different node states and task requirements.

[0015] Optionally, the method further includes: evaluating the final decision strategy to obtain a decision reward R, and punishing load imbalance and low decision efficiency.

[0016] The embodiments of the present invention provide a specific method for punishing the decision-making strategy, including balancing between nodes and balancing the resource utilization rate within a node, so as to improve computing performance.

[0017] Optionally, the method further includes: recording the probability distribution matrix A, the evaluation score V, the decision reward R at the first moment, and the probability distribution matrix A′, the evaluation score V′, and the decision reward R′ at the next moment; updating the training Critic network based on minimizing the mean square error between V and V′; and updating the training Actor network based on the following loss function:

[0018] actor_loss = -logA × (γV' + RV)

[0019] Among them, γ is the discount factor.

[0020] In the embodiment of the present invention, the Actor-Critic network can be updated to adapt to different node states and task requirements.

[0021] Optionally, the method also includes: obtaining the state information of the node when all the tasks are transmitted to the node; comparing the state information of the node at the time of execution with the state information of the node at completion to obtain a mean square error, and updating the prediction network based on minimizing the mean square error.

[0022] In the embodiment of the present invention, the prediction network is updated after a certain number of the above decisions are made to improve the accuracy of the prediction network.

[0023] An embodiment of the present invention provides a reinforcement learning scheduling system based on resource prediction and action mutation in a heterogeneous cloud environment, which is used to execute any of the methods described above.

[0024] The reinforcement learning scheduling system based on resource prediction and action mutation in a heterogeneous cloud environment provided by an embodiment of the present invention can achieve the same technical effect as the reinforcement learning scheduling method based on resource prediction and action mutation in the above-mentioned heterogeneous cloud environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0026] Figure 1 A flow chart of a reinforcement learning scheduling method based on resource prediction and action mutation in a heterogeneous cloud environment provided by an embodiment of the present invention;

[0027] Figure 2 It is a flow chart of a reinforcement learning scheduling method based on resource prediction and action mutation in an embodiment of the present invention;

[0028] Figure 3 This is a structural diagram of the feature coding part in an embodiment of the present invention;

[0029] Figure 4 This is a diagram of the Actor network structure in an embodiment of the present invention;

[0030] Figure 5 This is a diagram of the Critic network structure in an embodiment of the present invention;

[0031] Figure 6 It is a simplified structural diagram of the reinforcement learning scheduling method based on resource prediction and action mutation in an embodiment of the present invention. DETAILED DESCRIPTION

[0032] In order to make the above-mentioned objects, features and advantages of the present invention more clearly understood, the specific embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0033] The embodiment of the present invention provides a reinforcement learning scheduling method based on resource prediction and action mutation in a heterogeneous cloud environment. The node status is predicted by the prediction network, and then the node status and task requirements are input into the reinforcement learning framework to obtain a scheduling decision, thereby improving the task completion rate and resource utilization rate of the heterogeneous cloud computing system and reducing energy consumption.

[0034] The heterogeneous cloud computing system to which the above method is applicable should have a series of key features that ensure that the system can efficiently and flexibly handle diverse tasks. Specifically, the system should contain multiple types of nodes, which may have the same or different computing resources to meet the resource requirements of different tasks. When tasks are submitted to the system, these tasks may also have the same or different computing resource requirements, and these requirements information is open to the scheduler, so that the scheduler can make the best scheduling decision based on comprehensive information.

[0035] The tasks processed by the above method are regarded as the smallest indivisible work units at the computer level, that is, these tasks are logically independent and there is no dependency between tasks. Each task has clear computing resource requirements, and the task will only be executed when the system can provide enough resources to meet these requirements. After the task is submitted to the heterogeneous cloud computing system, it will not be executed immediately, but will first enter a waiting queue. This design helps the system manage tasks in an orderly manner and avoid resource conflicts and overloads. At the same time, it also reduces the communication overhead and computing burden in the scheduling process.

[0036] In summary, the above method of this embodiment aims to improve the overall performance and efficiency of heterogeneous cloud computing systems in processing diversified tasks through intelligent scheduling strategies. However, it is worth noting that due to the use of a unified scheduling mechanism, the waiting time of some tasks may increase accordingly. Therefore, in practical applications, it is necessary to weigh the relationship between performance improvement and waiting time to find the best balance point.

[0037] This embodiment provides a reinforcement learning scheduling method based on resource prediction and action mutation in a heterogeneous cloud environment. Figure 1 The flowchart of the reinforcement learning scheduling method based on resource prediction and action mutation in a heterogeneous cloud environment is shown. The method can be executed by a scheduler and mainly includes the following steps:

[0038] S102, obtaining the number of tasks in the task waiting queue, and determining whether the number reaches a preset number threshold.

[0039] The number of tasks in the task waiting queue, that is, the size of the waiting queue, if the cumulative number reaches the preset threshold, the subsequent reinforcement learning scheduling process is triggered. On the other hand, in order to avoid a task waiting time too long, a decision will be made when no decision is made for a certain period of time.

[0040] Optionally, in order to balance the waiting time and the computing load caused by scheduling, the preset quantity threshold (ie, the decision length L) is set to 10, that is, scheduling is performed once every 10 tasks.

[0041] S104, if the preset number threshold is reached, the current state information of the node is obtained, and the current state information of the node is predicted based on the pre-trained prediction network to obtain the state information of the node at the execution time.

[0042] The status information may include: the number of remaining CPU cores, memory size, GPU size, hard disk size, bandwidth size, average waiting time of historical tasks, node waiting queue length, etc. The average waiting time of historical tasks does not need to consider too many historical tasks. For example, the average waiting time of the five most recently completed tasks can be calculated.

[0043] For each decision, the current state information must first be obtained from the node, and then a decision strategy is generated, and then tasks are assigned to nodes according to the decision strategy. In a practical environment, the data size of a task may reach hundreds of megabytes, and the communication delay of transmitting all task data to the node is large. During this period of time, the state of the node will change. Therefore, in this embodiment, a prediction network is used to predict the state of the node at the time of execution.

[0044] The prediction network uses a long-term and short-term neural network. The prediction network is trained based on the historical state information of the node. The historical state information can be the state information of the real historical moment, or the state information recorded by the simulation experiment using the existing scheduling strategy. For example, random scheduling is used: a node is randomly selected from the nodes that can meet the task requirements for allocation, and the node information is recorded at fixed time intervals.

[0045] Optionally, a long-term and short-term neural network based on seasonal and trend decomposition is trained, and seasonal and trend decomposition is used to obtain three components of season, trend and residual. Finally, the sum of the components is compared with the true value, and the mean square error between the two is used as the loss. The long-term and short-term neural network is trained by minimizing the mean square error.

[0046] When the system is just starting, there is no state information data from the past, so the current state can be regarded as the state at the decision execution time.

[0047] S106, extracting and encoding the state information of the node at the execution time and the task feature information of the task in the task waiting queue to obtain a node state vector and a task feature vector.

[0048] Specifically, the state information of the node at the time of execution is extracted to obtain a node state vector of length N, and the task feature information of the tasks in the task waiting queue is extracted to obtain an M-dimensional task feature vector; then, the node state vector of length N and the M-dimensional task feature vector are encoded to obtain a node state vector and a task feature vector of the same length.

[0049] Considering the difference in the length of N and M, directly inputting them into the reinforcement learning network will cause an imbalance between the node state vector and the task feature vector, so they need to be encoded into two vectors of the same length.

[0050] S108, merging the node state vector and the task feature vector to obtain a merged vector, and inputting the merged vector into a reinforcement learning model to generate a decision strategy.

[0051] For example, the reinforcement learning model may adopt an Actor-Critic algorithm model, and input the merged vector into the Actor-Critic algorithm model to generate a decision strategy.

[0052] Optionally, after merging the above-mentioned node state vector and task feature vector, a merged vector is obtained, and the merged vector is input into the Actor network to output a probability distribution matrix A; the probability distribution matrix A is sampled with maximum probability, and the node corresponding to the maximum value of each row of the probability distribution matrix A is determined as the scheduling node of the task to obtain a decision strategy.

[0053] Perform maximum probability sampling on the probability distribution matrix. Specifically, one row represents the probability of a task being assigned to each node. The index with the highest probability in a row is selected as the scheduling node for the task. Perform the above operation on each row to get a preliminary scheduling decision.

[0054] S110, assign tasks to target type nodes based on preset probabilities, and obtain the final decision strategy. The preliminary scheduling decision in the above steps for tasks that do not require GPU resources still tends to be scheduled to nodes that have both CPU (Graphics Processing Unit) and GPU (Central Processing Unit) resources, rather than directly scheduling to nodes that only have CPU resources. In simple terms, the network tends to assign tasks directly to nodes that have both CPU and GPU resources. Therefore, further guidance is needed to make its scheduling decisions more reasonable.

[0055] In this embodiment, the task with zero GPU resource demand but assigned to a node with GPU resources is specifically implemented, and is suddenly assigned to a node with only CPU resources with a probability of ε. For example, the value of ε is set to 0.5. The final decision strategy is obtained after the sudden assignment. Specifically: for the above system, the decision strategy D is a vector of length L (L is the decision length), and the i-th component in D represents the scheduling of the i-th task to the D(i) node. In the preliminary scheduling decision, if the k-th task does not require GPU resources, but D(k) nodes have GPU resources, let D(k) be equal to the serial number of the node without GPU resources with a probability of ε. (If nodes 1 and 2 do not have GPU resources, let D(k) be equal to 1 or 2).

[0056] Furthermore, the prediction network can be updated after a certain number of the above decisions are made to improve the accuracy of the prediction network. Based on this, the above method also includes: obtaining the state information of the node when all tasks are transmitted to the node; then, comparing the state information of the node at the time of execution with the state information of the node at the time of completion to obtain the mean square error, and updating the prediction network based on minimizing the mean square error.

[0057] The reinforcement learning scheduling method based on resource prediction and action mutation in a heterogeneous cloud environment provided by the embodiment of the present invention can predict the node status by predicting the network, and then input the node status and task requirements into the reinforcement learning framework to obtain a scheduling decision, thereby improving the task completion rate and resource utilization rate of the heterogeneous cloud computing system and reducing energy consumption.

[0058] In this embodiment, if there is no new task demand for a long time, that is, the above-mentioned preset number threshold condition is not met for a long time, in order to avoid infinite waiting of tasks in the task waiting queue, the existing scheduling strategy will be used for scheduling. Based on this, the above method may also include the following:

[0059] If the waiting time of the task waiting queue reaches the preset time threshold, the current state information of the node and the task characteristic information are obtained; then, according to the current state information and the task characteristic information, a scheduling decision plan is generated using a preset scheduling strategy. Exemplarily, the preset scheduling strategy is a random scheduling strategy, which aims to maintain the activity of the system, prevent idle resources, and ensure that the system can maintain a certain stability and responsiveness when facing unstable external task requirements.

[0060] Taking the prediction network as a long-term and short-term neural network as an example, the training process is as follows:

[0061] First, obtain the historical status information of the node at fixed time intervals and normalize the historical status information. The node status information used in pre-training can use some simple, existing scheduling strategies to simulate data and record the corresponding node information. Since the order of magnitude of each indicator is too different, the data obtained needs to be normalized.

[0062] Secondly, the normalized historical state information is used to construct a time series using a time window, and then seasonal and trend decomposition is used to obtain three components: seasonal, trend, and residual.

[0063] Then, the above three components are input into the long-term and short-term neural network to obtain the predicted value of each component, and the sum of the predicted values ​​of each component is determined to obtain the predicted value of the state information at the target moment.

[0064] Finally, the predicted value of the state information at the target moment is compared with the true value at the target moment to obtain the mean square error, and the long-term and short-term neural network is updated based on minimizing the mean square error.

[0065] In this embodiment, a Critic network may be used for evaluation. Based on this, the method further includes: using the Critic network to evaluate the merged vector to obtain an evaluation score V.

[0066] In this embodiment, the above decision can be evaluated to obtain a decision reward R, and a penalty is imposed on load imbalance and low decision efficiency. Based on this, the above method also includes: evaluating the final decision plan to obtain a decision reward R, and a penalty is imposed on load imbalance and low decision efficiency.

[0067] Furthermore, the Actor-Critic network is updated based on the evaluation scores and decision rewards. Based on this, the above method may also include:

[0068] Record the probability distribution matrix A, evaluation score V, and decision reward R at the first moment, as well as the probability distribution matrix A′, evaluation score V′, and decision reward R′ at the next moment;

[0069] Update the training Critic network based on minimizing the mean square error between V and V′;

[0070] Update the training Actor network based on the following loss function:

[0071] actor_loss = -logA × (γV' + RV)

[0072] Among them, γ is the discount factor.

[0073] The embodiment of the present invention aims at the task scheduling problem of heterogeneous cloud computing system, constructs a long-term and short-term neural network based on season and trend decomposition to predict the state of nodes, and then uses the Actor-Critic algorithm in reinforcement learning to generate a scheduling method.

[0074] Specifically, the Actor part generates a probability distribution matrix through the node status, task requirements, and the prior relationship between all resources and tasks owned by the node, and then selects the scheduling method based on the probability distribution; the Critic generates an evaluation score based on the Actor's choice, and the Actor will also modify the probability distribution of the scheduling method based on the evaluation score. At the same time, in order to make the scheduling result more reasonable, the results generated by the Actor are mutated, which further improves the performance of the system. The effectiveness of this method has been verified in simulation experiments.

[0075] Figure 2 The flowchart of the reinforcement learning scheduling method based on resource prediction and action mutation is shown, which includes the following steps:

[0076] S201, determine whether the length of the waiting queue reaches L. If yes, execute S203, if not, execute S202.

[0077] S202, determine whether the waiting time reaches T. If not, return to execute S201, and if so, execute S209.

[0078] S203, obtaining node status information and task feature information.

[0079] S204: Use the prediction module to generate a prediction value of the execution time.

[0080] S205, using the Actor-Critic network to generate a scheduling strategy and evaluation score.

[0081] S206, sudden change of action.

[0082] S207, executing scheduling decision.

[0083] S208, update the prediction module and the Actor-Critic network.

[0084] S209, obtaining node status information and task feature information.

[0085] S210, generating a scheduling decision using random scheduling.

[0086] The specific steps of the method provided by the embodiment of the present invention are as follows: Steps 1 to 2 are pre-training steps of the prediction module of the method, and steps 3 to 12 are real-time decision-making and network updating steps of the method in a heterogeneous cloud computing system.

[0087] Step 1: Obtain node status information for training the prediction network. The data should include multiple pieces of information with the same dimension, each of which corresponds to the status of a node. Optionally, the node status includes the number of remaining CPU cores, memory size, GPU size, hard disk size, bandwidth size, average waiting time of historical tasks, and node waiting queue length. When considering the average waiting time of historical tasks, in order to improve the timeliness and relevance of the calculation, it is not necessary to trace back too much historical data. Optionally, the average time of the five most recently completed tasks can be calculated.

[0088] The node status information used in pre-training can be simulated using some simple, existing scheduling strategies and record the corresponding node information. For example, random scheduling is used: randomly selecting a node from the nodes that can meet the task requirements to allocate, and recording the node information at fixed time intervals.

[0089] Step 2: Train the long-term and short-term neural network based on seasonal and trend decomposition. Specifically:

[0090] For the node status information at fixed time intervals obtained in step 1, the data obtained needs to be normalized because the order of magnitude difference of each indicator is too large. The common MAX-MIN normalization will cause the node status information and task information to not match. To avoid this problem, each attribute is normalized by dividing it by the maximum value of the corresponding attribute possessed by the node in the system.

[0091] Next, we construct a time series with time-windows as the time window, and then use the seasonal and trend decomposition (STL) to obtain the three components of season, trend and residual. In order to reduce the computational complexity of the decomposition process, the period parameter in the STL decomposition is set to 3. The three components are input into the long short-term neural network (LSTM) to obtain the predicted values ​​of the components. Finally, the sum of the components is compared with the true value (i.e., the node information at the next moment of the time-window continuous time series), and the mean square error between the two is used as the loss. The network is updated by minimizing the mean square error.

[0092] Optionally, the LSTM network structure is as follows:

[0093] The sizes of the three input components are all time-windows×N, where N is the state length of all nodes in the system (N=Node-Num*Node-State-Len, Node-Num is the number of nodes, and Node-State-Len is the state information of each node, i.e., the 7 feature information mentioned in step 1).

[0094] Each component is input into the LSTM layer with the same structure. The feature dimension of the hidden layer is 32, the number of hidden layers is 16, and the dropout is set to 0.5 to prevent overfitting.

[0095] Dimension mapping layer, a fully connected neural network with N neurons, with an input size equal to the feature dimension of the hidden layer of the LSTM layer, 32, that is, remapping the predicted size to the required size.

[0096] The sum of the results of the three dimensional mapping layers is the predicted node state.

[0097] Step 3: Monitor the size of the waiting queue of the system. If it reaches a certain size L, use the Actor-Critic reinforcement learning framework to start scheduling (steps 5 to 11), otherwise wait for the task; at the same time, in order to avoid a task waiting too long, a decision will be made when no decision is made for a certain period of time (step 4). Optionally, in order to balance the waiting time and the computing load brought by scheduling, the decision length L is set to 10, that is, scheduling is performed once every 10 tasks.

[0098] Step 4: If there is no new task demand for a long time, random scheduling will be used to generate a scheduling strategy to avoid infinite waiting in the waiting queue. For example, the data of alibaba-trace-2018-2V is used for simulation. There is no situation where there is no new task for a long time in the data, but considering the actual situation and to ensure the stability of the system, this embodiment takes this situation into consideration.

[0099] Step 5: Because for each decision, we first need to obtain the state information from the node, then generate the decision strategy, and then assign it to the node according to the decision strategy. The data size of the task reaches hundreds of megabytes (such as the alibaba-trace-2018-2V dataset mentioned above), and the communication delay of transmitting all the task data to the node is large, which will cause the state of the node to change during this period of time. Therefore, we need to use the prediction network to predict the state of the node at the time of execution.

[0100] When the waiting length reaches L, the node status information is obtained, and then the node status (node ​​status vector of length N) is predicted using the prediction network shown in step 2. Step 2 is to use the node status information of the past time-windows to predict the node status information at the decision execution time. When the system is just starting, there is no data from the past time-windows, so the current state is used as the state at the decision execution time.

[0101] For each task, there are CPU cores, memory, video memory (GPU), hard disk, service quality level, running time, waiting time, etc. Ltask information, the task information is L×T L Optionally, T L Select the above 7 parameters. In order to consider the constraint relationship between the L tasks, it is necessary to perform convolution operations on them to extract their feature relationships. Optionally, the two-dimensional matrix is ​​pulled into a one-dimensional vector, and then two layers of one-dimensional convolution are used.

[0102] The input of the first convolution layer is L*T L , the output size is set to 16, the convolution kernel size is set to 16, the step size is set to 1, and the relu activation function is used.

[0103] The input of the second convolution layer is 16, the output size is set to 8, the convolution kernel size is set to 6, and the stride is set to 2. The relu activation function is used.

[0104] Finally, we get an M-dimensional task feature vector. The size of M in this method is 224.

[0105] Step 6: There is a difference in the length of N and M. Directly inputting them into the network will cause an imbalance between the node state vector and the task feature vector, so they need to be encoded into two vectors of the same length. The node state encoding layer takes a vector of size N as input and sets the output size to 128; similarly, the task feature encoding layer takes a vector of size M as input and sets the output size to 128. That is, a node state encoding vector and a task feature encoding vector of length 128 are obtained.

[0106] Figure 3 The structure diagram of the feature coding part is shown, in which it shows that the task feature information passes through convolution layer 1 and convolution layer 2, and is output to the task feature coding layer to obtain the task feature coding; and it shows that the STL decomposition obtains three components of season, trend and residual, which are respectively input into the corresponding LSTM layers, and then added to obtain the node state prediction value, which is output to the node state coding layer to obtain the node state coding.

[0107] Step 7: Merge the two vectors obtained in step 6, input the merged vector into the final Actor network, and output a probability distribution matrix of size L×Node_Num (decision length×number of nodes). Optionally, Figure 4 The Actor network structure diagram is shown, and the constructed network structure is as follows:

[0108] A fully connected layer with an input size of 256 and an output size of 128.

[0109] Followed by a relu activation layer and a dropout layer to avoid overfitting, with a dropout-rate of 0.5.

[0110] Probability distribution mapping layer, the input size is 128, and the output size is L×Node_Num.

[0111] Finally, a softmax operation is performed on each row to obtain a probability distribution matrix of size L×Node_Num.

[0112] Finally, the maximum probability sampling is performed according to the probability matrix. Specifically, one row represents the probability of a task being assigned to each node, and the index with the highest probability in a row is selected as the scheduling node for the task. The above operation is performed on each row to obtain a preliminary scheduling decision.

[0113] Step 8: The preliminary scheduling decision made in step 7 for tasks that do not require GPU resources still tends to be scheduled to nodes that have both CPU and GPU resources, rather than directly scheduling to nodes that only have CPU resources.

[0114] The network tends to assign tasks directly to nodes that have both CPU and GPU resources, so further guidance is needed to make its scheduling decisions more reasonable. For example, for tasks that have zero GPU resource requirements but are assigned to nodes with GPU resources, the probability of ε is suddenly assigned to nodes that only have CPU resources, and the value of ε is set to 0.5. After the sudden assignment, the final decision strategy is obtained.

[0115] Step 9: Use the Critic network to evaluate the current system status.

[0116] Figure 5 The structure diagram of the Critic network is shown. The constructed Critic network structure is as follows: The current system state directly uses the two vectors of length 128 obtained in step 6. The node state encoding vector and the task feature encoding vector are combined and input to the fully connected layer, the input size is 128*2, the output size is 128, and relu is used as the activation function; then the result is input to the dropout layer and the fully connected layer, and the output size is 1. The evaluation score -V is obtained.

[0117] Step 10: Execute the scheduling decision strategy and assign tasks to corresponding nodes.

[0118] Step 11: Because the prediction network has been pre-trained, this part is updated every five scheduling decisions. When all tasks are transmitted, the system node status is obtained again. The mean square error between the predicted value and the true value of the prediction network is used as the loss of the prediction network, and the network is updated to minimize the loss value.

[0119] Step 12: Evaluate the decision made in step 8 and obtain the decision reward - R. Penalize load imbalance and low decision efficiency. Optionally,

[0120] (1) Load imbalance penalty

[0121] Imbalance penalty between nodes. A good system state hopes that the CPU, memory, and GPU usage of all nodes are similar. In order to reduce calculations, the negative of the maximum difference in CPU, memory, and GPU usage of all nodes is regarded as an imbalance penalty between nodes.

[0122] Unbalanced penalty within a node: within the same node, the CPU, memory, and GPU usage rates are similar to avoid the situation where one value is very high and the others are low. The negative difference between the three is also used as a penalty, and there is a certain tolerance. If the difference is less than 0.2, no penalty is given.

[0123] (2) Rewards and penalties for decision-making efficiency

[0124] If all decision strategies can be implemented, that is, the node can meet the requirements of the task, and the waiting time of each task is short, a decision reward of 1 is given. For tasks that cannot meet the requirements, each one is given a penalty of -0.5.

[0125] Step 13: Record the probability distribution matrix A generated in step 7, the evaluation score V generated by the critic network, and the decision reward R. At the next decision moment, (A′, V′, R′) will also be generated. For two adjacent moments, V ≈ γV ′ +R, where γ is the discount factor, set to 0.98. Therefore, V and γV ′ +R is used as the loss of the Critic network, and the mean square error is minimized to update the training Critic network. For the Actor network, the loss is defined as:

[0126] actor_loss = -logA×(γV ′ +RV)

[0127] Update the actor network by minimizing actor_loss.

[0128] Step 14: Go to step 3 and wait for the next scheduling.

[0129] Figure 6 The schematic diagram of the structure of the reinforcement learning scheduling method based on resource prediction and action mutation is shown. The heterogeneous cloud computing system outputs feature encoding to the Actor network, the Actor network outputs the decision strategy, and the Critic network outputs the evaluation error, which can update the Actor network and the Critic network.

[0130] The embodiment of the present invention provides a reinforcement learning scheduling method based on resource prediction and action mutation in a heterogeneous cloud environment. It adopts a long-term and short-term neural network prediction model based on season and trend decomposition, so that the node state used by the method at the decision time is more consistent with the execution time; in terms of scheduling decisions, Actor-Critic is used to generate the initial scheduling decision. Since the initial scheduling decision is not effective for tasks with a GPU requirement of 0, some mutations are made to these decision components, and the weight parameters of the network can be modified in real time to adapt to different node states and task requirements. These characteristics enable the method provided by the embodiment of the present invention to make excellent scheduling decisions within an acceptable time, which helps to improve the performance of heterogeneous cloud computing systems and has broad application prospects in the field of heterogeneous cloud computing task scheduling.

[0131] An embodiment of the present invention provides a reinforcement learning scheduling system based on resource prediction and action mutation in a heterogeneous cloud environment, which is used to execute the steps of the method provided in the above embodiment.

[0132] The reinforcement learning scheduling system based on resource prediction and action mutation in the above-mentioned heterogeneous cloud environment provided by the embodiment of the present invention has the same implementation principle and technical effects as the above-mentioned embodiments. For the sake of brief description, for matters not mentioned in the system embodiment, reference can be made to the corresponding contents in the above-mentioned method embodiment.

[0133] An embodiment of the present invention provides an electronic device, which includes a processor and a storage device. The storage device stores a computer program that can be run on the processor. When the processor executes the computer program, the steps of the method provided in the above embodiment are implemented.

[0134] An embodiment of the present invention provides a computer-readable medium, wherein the computer-readable medium stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the method described in the above embodiment.

[0135] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the control device through a computer program, and the program can be stored in a computer-readable storage medium. When the program is executed, it may include the processes of the above-mentioned method embodiments, wherein the storage medium may be a memory, a disk, an optical disk, etc.

[0136] In this article, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0137] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0138] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A reinforcement learning scheduling method based on resource prediction and action mutation in a heterogeneous cloud environment, characterized in that: include: Obtaining the number of tasks in the task waiting queue, and determining whether the number reaches a preset number threshold; If the preset number threshold is reached, the current state information of the node is obtained, and the current state information of the node is predicted based on the pre-trained prediction network to obtain the state information of the node at the execution time; Extracting and encoding the state information of the node at the time of execution and the task feature information of the tasks in the task waiting queue to obtain a node state vector and a task feature vector; Merging the node state vector and the task feature vector to obtain a merged vector, and inputting the merged vector into a reinforcement learning model to generate a decision strategy; The decision strategy is used to assign tasks to node mutations of the target type based on a preset probability to obtain a final decision strategy.

2. The method according to claim 1, characterized in that The method further comprises: If the waiting time of the task waiting queue reaches a preset time threshold, the current state information of the node and the task feature information are obtained; According to the current status information and task feature information, a scheduling decision plan is generated using a preset scheduling strategy.

3. The method according to claim 1, characterized in that The prediction network is a long-term and short-term neural network; the method further comprises: Acquire historical status information of the node at fixed time intervals, and normalize the historical status information; The normalized historical state information is used to construct a time series using a time window, and then seasonal and trend decomposition is used to obtain three components: seasonal, trend, and residual. Inputting the three components into the long-short term neural network to obtain the predicted value of each component, and determining the sum of the predicted values ​​of each component to obtain the predicted value of the state information at the target time; The predicted value of the state information at the target moment is compared with the true value at the target moment to obtain a mean square error, and the long short-term neural network is updated based on minimizing the mean square error.

4. The method according to claim 1, characterized in that: The feature extraction and encoding of the state information of the node at the execution time and the task feature information of the task in the task waiting queue are performed to obtain the node state vector and the task feature vector, including: The state information of the node at the time of execution is extracted to obtain a node state vector of length N, and the task feature information of the tasks in the task waiting queue is extracted to obtain an M-dimensional task feature vector; The node state vector of length N and the task feature vector of dimension M are encoded to obtain a node state vector and a task feature vector of the same length.

5. The method according to claim 1, characterized in that The reinforcement learning model is an Actor-Critic algorithm model, and the step of inputting the merged vector into the reinforcement learning model to generate a decision strategy includes: Input the merged vector into the Actor network and output a probability distribution matrix A; The probability distribution matrix A is sampled with maximum probability, and the node corresponding to the maximum value of each row of the probability distribution matrix A is determined as the scheduling node of the task to obtain a decision strategy.

6. The method according to claim 5, characterized in that The method further comprises: The merged vector is evaluated using a Critic network to obtain an evaluation score V.

7. The method according to claim 5, characterized in that The method further comprises: The final decision strategy is evaluated to obtain a decision reward R, and penalties are imposed on load imbalance and low decision efficiency.

8. The method according to claim 7, characterized in that The method further comprises: Record the probability distribution matrix A, the evaluation score V, and the decision reward R at the first moment, and the probability distribution matrix A′, the evaluation score V′, and the decision reward R′ at the next moment; Update the training Critic network based on minimizing the mean square error between V and V′; Update the training Actor network based on the following loss function: actor_loss = -logA × (γV' + RV) Among them, γ is the discount factor.

9. The method according to claim 1, characterized in that: The method further comprises: Acquire the state information of the node when all the tasks are transmitted to the node; The state information of the node at the time of execution is compared with the state information of the node at the time of completion to obtain a mean square error, and the prediction network is updated based on minimizing the mean square error.

10. A reinforcement learning scheduling system based on resource prediction and action mutation in a heterogeneous cloud environment, characterized in that: Used to perform the method according to any one of claims 1 to 9.

Images