A cloud task scheduling method and system based on reinforcement learning and evaluation model

Abstract

The application relates to the technical field of task scheduling, and provides a cloud task scheduling method and system based on reinforcement learning and an evaluation model, which comprises the following steps: obtaining cloud task and resource information, generating a cloud task scheduling scheme through a policy network of a reinforcement learning model; for the cloud task scheduling scheme, combining cloud task scheduling rules, obtaining a score and improvement suggestions through a task scheduling evaluation large model, and based on the improvement suggestions, adjusting the cloud task and resource information, then updating the cloud task scheduling scheme through the policy network of the trained reinforcement learning model; continuously collecting resource conditions, cloud tasks, the generated cloud task scheduling scheme, the score and the improvement suggestions, after a period of time, adjusting the state space, the action space and the reward function of the reinforcement learning model according to the improvement suggestions, and performing incremental learning on the reinforcement learning model and the task scheduling evaluation large model. The cloud task execution efficiency and quality are effectively improved, and the smooth completion of work is ensured.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of task scheduling, and particularly relates to a cloud task scheduling method and system based on reinforcement learning and evaluation model. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute prior art.

[0003] In the current digital era, cloud environments exhibit unprecedented complexity. Private clouds provide enterprises with highly customized and secure internal computing resource environments, enabling enterprises to finely adjust resources according to their business needs and ensure the security and privacy of core data. Public clouds, with their powerful resource pools and convenient service models, attract a large number of users who are sensitive to costs and seek rapid deployment. Users can obtain computing, storage, and network resources on demand without investing a large amount of hardware facility costs. Hybrid clouds skillfully combine the characteristics of private clouds and public clouds, with key business of enterprises running stably in private cloud environments and some non-core, flexible-demand business migrating flexibly to public clouds, achieving the best balance of resource utilization and cost control. Edge clouds further push computing power to the network edge, greatly reducing data transmission delay and providing strong support for application scenarios with extremely high real-time requirements, such as autonomous driving and industrial Internet of Things.

[0004] Workflow tasks have also become extremely complex. These cloud tasks no longer exist in isolation but are intertwined to form complex workflows with various structures. Some workflows exhibit sequential execution structures, in which subsequent tasks are started one by one after the completion of the previous cloud task. Some workflows have parallel execution structures, in which multiple cloud tasks can be performed simultaneously to improve overall execution efficiency. Some workflows have cyclic structures, in which specific cloud tasks need to be repeatedly executed until specific conditions are met. Different workflows have different goals, such as pursuing the shortest task completion time, maximizing resource utilization, or ensuring high reliability of cloud task execution.

[0005] Under this background, the efficiency and accuracy of cloud task scheduling have become key factors in ensuring efficient execution of workflows and rational utilization of resources. However, traditional workflow task scheduling methods have obvious drawbacks. They mostly rely on fixed rules or simple heuristic algorithms, such as allocating resources in order of cloud task priority or deciding task allocation based on the idle degree of resources. Since resource states change constantly in cloud environments and cloud task demands vary greatly, fixed rules are difficult to adapt to dynamic situations, and simple heuristic algorithms cannot comprehensively consider numerous complex factors, which often leads to suboptimal scheduling in actual scheduling, resulting in low cloud task execution efficiency, a large amount of idle and wasted resources, or long waiting time for resources due to unreasonable resource allocation, which seriously affects the overall progress of workflows.

[0006] With the rapid development of artificial intelligence technology, reinforcement learning algorithms are emerging in the field of cloud task scheduling and showing certain advantages. Reinforcement learning algorithms learn the optimal scheduling strategy from the rewards feedback by constantly interacting with the environment. They can adaptively adjust the scheduling scheme according to the real-time state of resources in the cloud environment and the dynamic demand of cloud tasks.

[0007] However, when encountering sudden irregular cloud tasks, due to the characteristics of such cloud tasks, which are quite different from the regular tasks encountered by reinforcement learning algorithms during training, reinforcement learning algorithms have difficulty in accurately evaluating their impact on overall scheduling, resulting in reduced accuracy of scheduling schemes and inability to effectively respond to complex and variable cloud task scenarios. SUMMARY

[0008] To solve the technical problems in the background art, the present application provides a cloud task scheduling method and system based on reinforcement learning and evaluation model. Through the powerful feature learning and representation capabilities of large models, potential laws and feature information of cloud tasks can be deeply mined to give modification suggestions to adjust cloud task and resource information, making the reinforcement learning model more accurate and reasonable when generating cloud task scheduling schemes. In incremental learning, the state space, action space and reward function parameters of the reinforcement learning model are adjusted based on the improvement suggestions to timely discover and solve potential problems, thereby effectively improving the efficiency and quality of cloud task execution and ensuring the smooth completion of work.

[0009] To achieve the above purpose, the present application adopts the following technical solutions:

[0010] The first aspect of the present application provides a cloud task scheduling method based on reinforcement learning and evaluation model, which comprises:

[0011] Obtain cloud task and resource information, generate cloud task scheduling scheme through the policy network of the reinforcement learning model;

[0012] For the cloud task scheduling scheme, combine the cloud task scheduling rules, get the score and improvement suggestions through the task scheduling evaluation large model, and based on the improvement suggestions, adjust the cloud task and resource information, then update the cloud task scheduling scheme through the policy network of the trained reinforcement learning model;

[0013] Continuously collect resource conditions, cloud tasks, generated cloud task scheduling schemes, scores and improvement suggestions. After a period of time, adjust the state space, action space and reward function of the reinforcement learning model according to the improvement suggestions, and perform incremental learning on the reinforcement learning model and the task scheduling evaluation large model according to the resource conditions, cloud tasks, generated cloud task scheduling schemes, scores and improvement suggestions.

[0014] Further, the joint training further comprises: training the reinforcement learning model and the task scheduling evaluation large model jointly.

[0015] The joint training comprises: freezing parameters of the task scheduling evaluation large model; receiving cloud tasks and resource states, generating a cloud task scheduling scheme through a policy network of the reinforcement learning model, obtaining a reward after interacting with an environment, and updating the resource states; inputting the generated cloud task scheduling scheme into the task scheduling evaluation large model to obtain a score and improvement suggestions; calculating a weighted sum of the reward and the score of the task scheduling evaluation large model, and optimizing the reinforcement learning model; freezing parameters of the reinforcement learning model; receiving cloud tasks and resource states, generating a cloud task scheduling scheme through the reinforcement learning model, the task scheduling evaluation large model evaluating the generated cloud task scheduling scheme to obtain a score and improvement suggestions, combining artificial evaluation results to calculate a loss function, and updating the task scheduling evaluation large model.

[0016] Further, the weight used in the calculation of the weighted sum of the reward and the score of the task scheduling evaluation large model is adjusted according to a difference between the normalized reward and the score and a training number of the policy network.

[0017] Further, the improvement suggestions for adjusting the cloud tasks and resource information comprise: adjusting an available resource range in the resource information; splitting the cloud tasks into different sub-tasks for parallel processing; and adjusting a cloud task scheduling timing.

[0018] Further, the improvement suggestions for adjusting the reinforcement learning model comprise: adding an energy consumption penalty term in a reward function; adding a label of whether a resource can be preempted in a state space; and limiting an action space.

[0019] A second aspect of the present application provides a cloud task scheduling system based on reinforcement learning and an evaluation model, which comprises:

[0020] a scheduling scheme generation module configured to: obtain cloud tasks and resource information, and generate a cloud task scheduling scheme through a policy network of a reinforcement learning model;

[0021] a scheduling scheme updating module configured to: for the cloud task scheduling scheme, obtain a score and improvement suggestions through a task scheduling evaluation large model in combination with a cloud task scheduling rule, and based on the improvement suggestions, adjust cloud tasks and resource information, and then update the cloud task scheduling scheme through a policy network of a trained reinforcement learning model;

[0022] The incremental learning module is configured to: continuously collect resource conditions, cloud tasks, generated cloud task scheduling schemes, scores and improvement suggestions, and after a period of time, adjust the state space, action space and reward function of the reinforcement learning model according to the improvement suggestions, and perform incremental learning on the reinforcement learning model and the task scheduling evaluation large model according to the resource conditions, cloud tasks, generated cloud task scheduling schemes, scores and improvement suggestions.

[0023] Further, the joint training module is configured to: jointly train the reinforcement learning model and the task scheduling evaluation large model.

[0024] The joint training step includes: freezing the parameters of the task scheduling evaluation large model; receiving cloud tasks and resource states, generating a cloud task scheduling scheme through the policy network of the reinforcement learning model, obtaining a reward after interacting with the environment, and updating the resource states; inputting the generated cloud task scheduling scheme into the task scheduling evaluation large model to obtain a score and an improvement suggestion; calculating the weighted sum of the reward and the score of the task scheduling evaluation large model, and optimizing the reinforcement learning model; freezing the parameters of the reinforcement learning model; receiving cloud tasks and resource states, generating a cloud task scheduling scheme through the reinforcement learning model, and the task scheduling evaluation large model evaluating the generated cloud task scheduling scheme to obtain a score and an improvement suggestion, combining the artificial evaluation result, calculating a loss function, and updating the task scheduling evaluation large model.

[0025] Further, the weight used in the calculation of the weighted sum of the reward and the score of the task scheduling evaluation large model is adjusted according to the difference between the normalized reward and score and the number of training times of the policy network.

[0026] Further, the improvement suggestion for adjusting the cloud tasks and resource information includes: adjusting the available resource range in the resource information; splitting the cloud tasks into different sub-tasks for parallel processing; and adjusting the cloud task scheduling timing.

[0027] Further, the improvement suggestion for adjusting the reinforcement learning model includes: adding an energy consumption penalty term in the reward function; adding a label indicating whether the resource can be preempted in the state space; and limiting the action space.

[0028] The third aspect of the present application provides a computer readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the cloud task scheduling method based on reinforcement learning and evaluation model as described above.

[0029] The fourth aspect of the present application provides a computer device including a computer readable storage medium, a processor and a computer program stored on the computer readable storage medium and executable on the processor, wherein the processor implements the steps of the cloud task scheduling method based on reinforcement learning and evaluation model as described above when executing the program.

[0030] Compared with the prior art, the present application has the beneficial effects that:

[0031] The present application can deeply mine the potential laws and feature information of cloud tasks to give modification suggestions to adjust the cloud tasks and resource information, so that the reinforcement learning model is more accurate and reasonable when generating a cloud task scheduling scheme, and adaptively adjusts and optimizes the scheduling scheme, improves the flexibility and adaptability of the scheduling method; and adjusts the state space, action space and reward function parameters of the reinforcement learning model based on the improvement suggestions during incremental learning, discovers and solves potential problems in a timely manner, thereby effectively improving the cloud task execution efficiency and quality and ensuring the smooth completion of work.

[0032] The present application jointly trains the reinforcement learning model and the task scheduling evaluation large model, realizes the co-evolution of the two models, so that the reinforcement learning model can generate a higher quality scheduling scheme, and at the same time improves the accuracy of the evaluation large model and the effectiveness of the improvement suggestions.

[0033] The present application combines the "environment interaction feedback" of traditional reinforcement learning and the "prior knowledge guidance" of the large model through a dynamic weight mechanism, adaptively balances exploration and utilization during the training phase, and strengthens the strategy correction in divergent scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0034] The accompanying drawings, which form a part of this application, are included to provide a further understanding of the application, and are incorporated in and constitute a part of this application. The embodiments of the application illustrate the

[0035] Figure 1 is a flowchart of a cloud task scheduling method based on reinforcement learning and evaluation model according to the first embodiment of the present application. DETAILED DESCRIPTION

[0036] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below in detail with reference to the accompanying drawings of the embodiments of the present application.

[0037] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as generally understood by those skilled in the art to which the present application belongs.

[0038] Embodiment One

[0039] The present embodiment provides a cloud task scheduling method based on reinforcement learning and evaluation model.

[0040] As described in the background, when encountering sudden irregular cloud tasks, reinforcement learning algorithms have difficulty accurately evaluating their impact on overall scheduling, resulting in reduced accuracy of scheduling solutions. However, large models can deeply explore the complex patterns and potential information in cloud tasks, comprehensively and deeply analyze the attributes of cloud tasks, the characteristics of resources, and the complex relationships between them, and capture subtle features and associations that traditional methods cannot detect. The present embodiment combines large models with reinforcement learning algorithms, uses large models to provide more accurate scheduling solution evaluations for reinforcement learning algorithms, and helps reinforcement learning algorithms make better decisions when faced with complex cloud tasks and sudden situations, thereby building a more intelligent, flexible, and efficient cloud task scheduling method to meet the scheduling needs of diverse cloud tasks in today's complex cloud environment.

[0041] The cloud task scheduling method based on reinforcement learning and evaluation model provided by the present embodiment realizes efficient scheduling of cloud tasks by combining reinforcement learning algorithms for cloud task scheduling strategy calculation and using large models for accurate evaluation of cloud task scheduling solutions, thereby improving resource utilization and task execution efficiency.

[0042] The cloud task scheduling method based on reinforcement learning and evaluation model provided by the present embodiment, as shown in Figure 1 , includes the following steps:

[0043] Step 1: Reinforcement learning model training.

[0044] A large amount of cloud task and resource data is used to train the reinforcement learning model, with cloud task and resource information as input and cloud task scheduling solution as output.

[0045] Specifically, each link and factor of workflow task scheduling is abstracted into a reinforcement learning environment, the state space, action space, and reward function are defined, and through continuous trial and error, the agent gradually learns the optimal action in different states, i.e., the optimal cloud task scheduling strategy.

[0046] The reinforcement learning model is based on the Deep Q Network (DQN) algorithm, and its training mainly includes the following steps:

[0047] Step 101, collect historical cloud task and resource information and perform data cleaning to improve data quality.

[0048] Step 102, set the state space, including information such as the attributes of cloud tasks (e.g., task priority, task composition, subtask completion status, quality of service requirements, etc.) and the available state of resources (e.g., resource quantity, link state, etc.), represented by .

[0049] Step 103: Set the action space, representing the available scheduling operations (cloud task scheduling scheme), that is, allocating cloud tasks to specific resources, using... express.

[0050] Step 104: Set the reward function to evaluate the performance of scheduling operations. express.

[0051] The reward function is designed based on the objectives of cloud task scheduling, such as cloud task execution cost, cloud task completion time, and resource utilization. For multiple objectives, a comprehensive evaluation is performed through weighted summation.

[0052] For example, considering the execution cost and completion time of a cloud task, the reward function is: ;in, The cost of the cloud instance required to execute cloud tasks, For cloud task completion time This is a tradeoff coefficient used for control. and The proportion, if If cloud task scheduling only considers the cost of cloud instances; if In this case, cloud task scheduling only considers the completion time.

[0053] Step 105, Utilize data ( The policy Q-network is trained using gradient descent; where, Indicates time , Indicates time The next moment; ) indicates the state Actions are selected and executed through an action selection algorithm. , reached state .

[0054] Specifically, at each moment Based on the current state Actions are selected using an action selection algorithm. And execute it, the policy Q network outputs the predicted Q value. Rewards are given after interacting with the environment. At the same time, it enters the next state. Generate empirical samples ( And store it in the experience replay buffer; calculate the target Q-network in Target Q value under the condition ; Computational strategy Q-network in Theoretical value under the condition ,in, is a discount factor, indicating that the rewards generated in future steps are considered to some extent; the loss function of the policy Q network is calculated ; the parameters of the policy Q network are updated using the gradient descent method to make the predicted Q value approach the theoretical value.

[0055] wherein the action selection algorithm uses the algorithm, and the main process is: with a probability of , select the action that maximizes the predicted Q value ; with a probability of , randomly select an action ; wherein , is the adjustment step size, which gradually decreases as the number of iterations increases.

[0056] Step 106, repeat the training of the policy Q network times, and then update the parameters of the target Q network using the parameters of the policy Q network, wherein is the target Q network parameter update interval, the target Q network and the policy Q network have the same structure and share parameters.

[0057] Step 2: Task scheduling evaluation large model training.

[0058] Summarize the cloud task scheduling rules, including cloud task splitting rules, cloud task objectives, resource usage constraints, priorities, etc.; collect a large number of different cloud task scheduling schemes, and manually evaluate the cloud task scheduling schemes according to the cloud task scheduling rules, label the scores (such as numerical scores of 0-10) and improvement suggestions (such as adjusting the cloud task execution order, replacing resource allocation, etc.).

[0059] Task scheduling evaluation large model selects suitable large model architectures such as qwen, and uses cloud task scheduling rules and historical data to fine-tune the large language model using the Low-Rank Adaptation of Large Language Models (LoRA) method. The input is the feature representation of the cloud task scheduling scheme and the cloud task scheduling rule, and the output is the score result and improvement suggestion of the cloud task scheduling scheme. By continuously adjusting the parameters of the large model, the large model can accurately evaluate the cloud task scheduling scheme.

[0060] Step 3: Reinforcement learning model and task scheduling evaluation large model joint training.

[0061] ​The task scheduling evaluation large model score is incorporated into the reinforcement learning reward system, and the cloud task scheduling scheme and the artificial evaluation result are used to train the large model, so as to realize the co-evolution of the two models. The final goal is to enable the reinforcement learning model to generate a cloud task scheduling scheme with higher quality, and to improve the accuracy of the task scheduling evaluation large model and the effectiveness of the improvement suggestions.

[0062] Step 301, freeze the parameters of the task scheduling evaluation large model, and train the reinforcement learning model.

[0063] In each training step, the reinforcement learning model receives the current cloud task and resource state as input, generates a cloud task scheduling scheme according to its current policy network (policy Q network) and in combination with an action selection algorithm , obtains a reward after interacting with the environment , and enters the next resource state ; the generated cloud task scheduling scheme is input into the task scheduling evaluation large model to obtain the score and improvement suggestions given by the task scheduling evaluation large model; the reward value is calculated according to the reward and the score of the task scheduling evaluation large model, and a new reward is obtained; based on the new reward , the policy Q network and the target Q network of the reinforcement learning model are optimized (specifically, experience samples are generated and stored in an experience replay buffer; the target Q value of the target Q network in is calculated ; the theoretical value of the policy Q network in is calculated ; the loss function of the policy Q network is calculated ; the policy Q network parameters are updated using the gradient descent method to make the predicted Q value approach the theoretical value; the policy Q network is repeatedly trained times, and then the parameters of the target Q network are updated using the policy Q network parameters).

[0064] The reward of the reinforcement learning model is composed of multiple parts, including the target of the cloud task scheduling (i.e. the reward function of the original reinforcement learning model) and the score of the task scheduling evaluation large model, so as to iteratively adjust the parameters of the reinforcement learning model.

[0065] The new reward function is: ; wherein, is the reward function of the original reinforcement learning, is the score of the task scheduling evaluation large model; if the two parts differ by orders of magnitude, the data can be normalized to eliminate the influence of the dimension on the final result, so that different variables, especially variables with large orders of magnitude, are comparable.

[0066] As another implementation, the new reward function is: ; wherein, is a weight, adjusted according to the difference between the normalized and the training frequency n of the policy Q network, i.e., RC Through the dynamic weight mechanism, the "environment interaction feedback" of traditional reinforcement learning and the "prior knowledge guidance" of large models are organically combined, and the exploration and utilization are adaptively balanced during the training phase, and the policy correction in the divergent scenario is strengthened. In the early stage of training or when RC is large, is small, the weight of the large model score S is high, and the intelligent agent is forced to learn the global knowledge of the large model first, speeding up the exploration of the effective policy space; in the later stage of training, with the increase of n or the decrease of RC, the weight of the traditional reward is increased, and the model gradually focuses on utilizing the learned policy to optimize the local details, improving the convergence efficiency. Avoid the problem of "insufficient exploration in the early stage" or "over-reliance on prior knowledge in the later stage" caused by fixed weight, and adaptively balance the learning goals at different stages.

[0067] Step 302, freeze the parameters of the reinforcement learning model, and train the task scheduling evaluation big model.

[0068] The cloud task scheduling scheme generated by the reinforcement learning model and the corresponding artificial evaluation results (scores and improvement suggestions) are used as input data; for the score part, the cloud task scheduling scheme is input into the task scheduling evaluation big model, and the task scheduling evaluation big model parameters are adjusted by minimizing the mean square error (MSE) between the output score of the task scheduling evaluation big model and the artificial score; for the improvement suggestion part, the improvement suggestion is encoded (such as using text embedding), and then the task scheduling evaluation big model is trained through a similar loss function (such as cross-entropy loss function or mean square error loss function) to make it output more accurate and effective improvement suggestions.

[0069] Step 4: Cloud task scheduling scheme generation.

[0070] When a new cloud task arrives, the cloud task and the current resource information are obtained, and the cloud task and the current resource information are input into the policy Q network of the trained reinforcement learning model.

[0071] The policy Q network outputs a cloud task scheduling scheme for the current cloud task and resource information according to the learned optimal policy, i.e., determines which resources each cloud task is executed on and the execution order.

[0072] Step 5: Cloud task scheduling scheme evaluation.

[0073] ​​Based on the trained task scheduling evaluation large model, the cloud task scheduling scheme is evaluated, and according to the improvement suggestions of the task scheduling evaluation large model, the input parameters of the policy Q network are adjusted, and the cloud task scheduling scheme is generated again using the policy Q network.

[0074] The process of cloud task scheduling scheme generation-cloud task scheduling scheme evaluation is repeated until the final cloud task scheduling scheme is obtained.

[0075] Among them, adjusting the input parameters of the policy Q network means: according to the improvement suggestions, adjusting the available resource range in the resource information, excluding resources that are not suitable for executing the task; splitting long-term cloud tasks into different sub-tasks for parallel processing to improve task execution efficiency; adjusting the cloud task scheduling timing, advancing / retarding the start time of the cloud task to avoid resource conflicts, etc.

[0076] This process is not simply repeated, but based on continuous optimization and adjustment, gradually approaching the optimal solution. The task scheduling evaluation large model will continuously repeat the above evaluation, improvement, update and rescheduling process, and each round of cycle is based on the optimization results of the last round, until an optimal task scheduling scheme is finally obtained. The final scheme will become the core basis for guiding the actual task execution, ensuring that the cloud task can be efficiently and orderly promoted.

[0077] Step 6: Cloud task scheduling scheme execution.

[0078] When the final cloud task scheduling scheme is obtained, the cloud task is deployed to the specific resource according to the final cloud task scheduling scheme for execution.

[0079] During the execution of the cloud task, the execution status of each cloud task is monitored in real time. Through monitoring, the actual start time, current execution progress, and estimated completion time of the cloud task can be obtained. If an exception occurs during execution, such as excessive execution time or resource usage exceeding expectations, relevant feedback information can be collected in a timely manner. These feedback information can be used as a reference for subsequent optimization of the cloud task scheduling scheme. At the same time, for the completed cloud task, detailed data such as actual completion time and resource consumption are recorded, which can be used to evaluate the overall effectiveness of this cloud task execution and provide data support for further optimization of the scheduling algorithm and evaluation large model.

[0080] Step 7: Iterative optimization.

[0081] The resource situation, cloud task information, cloud task scheduling scheme, score and improvement suggestion data are continuously collected, after a period of data collection, the reinforcement learning model is adjusted according to the improvement suggestion, and the reinforcement learning model and the task scheduling evaluation large model are incrementally learned according to the resource situation, the task information, the task scheduling scheme, the score and the improvement suggestion, so that the model adapts to the new environment, while avoiding catastrophic forgetting, and realizing the continuous optimization iteration of the model.

[0082] The task scheduling evaluation large model also outputs some improvement suggestions for the reinforcement learning model, such as the cloud task scheduling scheme excessively focuses on cost and completion time, but the energy consumption is too high, it is suggested to add an energy consumption penalty term in the reward function; the cloud task scheduling scheme does not consider whether the resource can be preempted, it is suggested to add a label of whether the resource can be preempted in the state space; some types of tasks are not suitable for using some resources, such as tasks with high accuracy requirements are not suitable for using unstable and interruptible resources, the action space is limited to increase the constraints of action selection. Based on these improvement suggestions, the state space, action space and reward function of the reinforcement learning model are adjusted, which can make the reinforcement learning model more suitable for the existing environment.

[0083] Embodiment two

[0084] The embodiment provides a cloud task scheduling system based on reinforcement learning and evaluation model, which specifically comprises:

[0085] A scheduling scheme generation module configured to: obtain cloud task and resource information, generate a cloud task scheduling scheme through a policy network of a reinforcement learning model;

[0086] A scheduling scheme updating module configured to: for the cloud task scheduling scheme, combine cloud task scheduling rules, obtain a score and an improvement suggestion through a task scheduling evaluation large model, and based on the improvement suggestion, adjust the cloud task and resource information, and then update the cloud task scheduling scheme through the policy network of the trained reinforcement learning model;

[0087] An incremental learning module configured to: continuously collect resource situation, cloud task, generated cloud task scheduling scheme, score and improvement suggestion, after a period of time, adjust the state space, action space and reward function of the reinforcement learning model according to the improvement suggestion, and incrementally learn the reinforcement learning model and the task scheduling evaluation large model according to the resource situation, the cloud task, the generated cloud task scheduling scheme, the score and the improvement suggestion.

[0088] Further, it further comprises a joint training module configured to: jointly train the reinforcement learning model and the task scheduling evaluation large model;

[0089] The step of joint training comprises: freezing parameters of the task scheduling evaluation large model; receiving cloud tasks and resource states, generating a cloud task scheduling scheme through a policy network of the reinforcement learning model, obtaining a reward after interacting with the environment, and updating the resource states; inputting the generated cloud task scheduling scheme into the task scheduling evaluation large model to obtain a score and improvement suggestions; calculating a weighted sum of the reward and the task scheduling evaluation large model score, and optimizing the reinforcement learning model; freezing parameters of the reinforcement learning model; receiving cloud tasks and resource states, generating a cloud task scheduling scheme through the reinforcement learning model, the task scheduling evaluation large model evaluating the generated cloud task scheduling scheme to obtain a score and improvement suggestions, combining artificial evaluation results to calculate a loss function, and updating the task scheduling evaluation large model.

[0090] Further, the weight used for calculating the weighted sum of the reward and the task scheduling evaluation large model score is adjusted according to the difference between the normalized reward and score and the number of times of training of the policy network.

[0091] Further, the improvement suggestions for adjusting the cloud tasks and resource information comprise: adjusting the available resource range in the resource information; splitting the cloud tasks into different sub-tasks for parallel processing; and adjusting the cloud task scheduling timing.

[0092] Further, the improvement suggestions for adjusting the reinforcement learning model comprise: adding an energy consumption penalty term in the reward function; adding a label of whether resources can be preempted in the state space; and limiting the action space.

[0093] It should be noted that each module in the embodiment corresponds to each step in the first embodiment one by one, and the specific implementation process is the same, which will not be repeated here.

[0094] The above only describes the preferred embodiments of the present application and is not used to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A cloud task scheduling method based on reinforcement learning and evaluation model, characterized in that, The method comprises the following steps: obtaining cloud task and resource information, generating a cloud task scheduling scheme through a policy network of a reinforcement learning model; for the cloud task scheduling scheme, combining cloud task scheduling rules, obtaining a score and improvement suggestions through a task scheduling evaluation large model, and based on the improvement suggestions, adjusting the cloud task and resource information, then updating the cloud task scheduling scheme through the policy network of the trained reinforcement learning model; continuously collecting resource conditions, cloud tasks, generated cloud task scheduling schemes, scores and improvement suggestions, and after a period of time, adjusting the state space, action space and reward function of the reinforcement learning model according to the improvement suggestions, and performing incremental learning on the reinforcement learning model and the task scheduling evaluation large model according to the resource conditions, cloud tasks, generated cloud task scheduling schemes, scores and improvement suggestions; jointly training the reinforcement learning model and the task scheduling evaluation large model; the step of jointly training comprises: freezing the parameters of the task scheduling evaluation large model; receiving cloud task and resource states, generating a cloud task scheduling scheme through a policy network of a reinforcement learning model, obtaining a reward after interacting with the environment, and updating the resource states; inputting the generated cloud task scheduling scheme into a task scheduling evaluation large model to obtain a score and improvement suggestions; calculating the weighted sum of the reward and the score of the task scheduling evaluation large model, optimizing the reinforcement learning model; freezing the parameters of the reinforcement learning model; receiving cloud task and resource states, generating a cloud task scheduling scheme through the reinforcement learning model, the task scheduling evaluation large model evaluating the generated cloud task scheduling scheme to obtain a score and improvement suggestions, combining artificial evaluation results to calculate a loss function, and updating the task scheduling evaluation large model. 2.The cloud task scheduling method based on reinforcement learning and evaluation model of claim 1, wherein, The weight used in the calculation of the weighted sum of the reward and the score of the task scheduling evaluation large model is adjusted according to the difference between the normalized reward and score and the number of training times of the policy network. 3.The cloud task scheduling method based on reinforcement learning and evaluation model of claim 1, wherein, The improvement suggestions for adjusting the cloud task and resource information include: adjusting the available resource range in the resource information; splitting the cloud task into different sub-tasks for parallel processing; adjusting the cloud task scheduling timing. 4.The cloud task scheduling method based on reinforcement learning and evaluation model of claim 1, wherein, The improvement suggestions for adjusting the reinforcement learning model include: adding an energy consumption penalty term to the reward function; adding a label indicating whether resources can be preempted in the state space; limiting the action space.

5. A cloud task scheduling system based on reinforcement learning and evaluation model, applying a cloud task scheduling method based on reinforcement learning and evaluation model as claimed in any one of claims 1-4, characterized in that, The method comprises the following steps: a scheduling scheme generation module configured to: obtain cloud task and resource information, and generate a cloud task scheduling scheme through a policy network of a reinforcement learning model; a scheduling scheme updating module configured to: for the cloud task scheduling scheme, combine cloud task scheduling rules, obtain a score and improvement suggestions through a task scheduling evaluation large model, and based on the improvement suggestions, adjust the cloud task and resource information, then update the cloud task scheduling scheme through the policy network of the trained reinforcement learning model; an incremental learning module configured to: continuously collect resource conditions, cloud tasks, generated cloud task scheduling schemes, scores and improvement suggestions, and after a period of time, adjust the state space, action space and reward function of the reinforcement learning model according to the improvement suggestions, and perform incremental learning on the reinforcement learning model and the task scheduling evaluation large model according to the resource conditions, cloud tasks, generated cloud task scheduling schemes, scores and improvement suggestions.

6. The cloud task scheduling system based on reinforcement learning and evaluation model of claim 5, wherein, The joint training module is configured to jointly train the reinforcement learning model and the task scheduling evaluation large model. The joint training comprises: freezing parameters of the task scheduling evaluation large model; receiving cloud tasks and resource states, generating a cloud task scheduling scheme through a policy network of the reinforcement learning model, obtaining a reward after interacting with the environment, and updating the resource states; inputting the generated cloud task scheduling scheme into the task scheduling evaluation large model to obtain a score and improvement suggestions; calculating a weighted sum of the reward and the score of the task scheduling evaluation large model, and optimizing the reinforcement learning model; freezing parameters of the reinforcement learning model; receiving cloud tasks and resource states, generating a cloud task scheduling scheme through the reinforcement learning model, and evaluating the generated cloud task scheduling scheme through the task scheduling evaluation large model to obtain a score and improvement suggestions, combining artificial evaluation results to calculate a loss function, and updating the task scheduling evaluation large model.

7. The cloud task scheduling system based on reinforcement learning and evaluation model of claim 6, wherein, The weight used in the calculation of the weighted sum of the reward and the score of the task scheduling evaluation large model is adjusted according to a difference between the normalized reward and the score and a training number of the policy network.

8. The cloud task scheduling system based on reinforcement learning and evaluation model of claim 5, wherein, The improvement suggestions for adjusting the cloud task and resource information include: adjusting an available resource range in the resource information; splitting the cloud task into different sub-tasks for parallel processing; and adjusting a cloud task scheduling timing.

9. The cloud task scheduling system based on reinforcement learning and evaluation model of claim 5, wherein, The improvement suggestions for adjusting the reinforcement learning model include: adding an energy consumption penalty term in a reward function; adding a label of whether a resource can be preempted in a state space; and limiting an action space.

Images