Cloud task scheduling method and system based on reinforcement learning and evaluation model

Abstract

The invention relates to the technical field of task scheduling, and provides a cloud task scheduling method and system based on reinforcement learning and an evaluation model, and the method comprises the steps: obtaining cloud tasks and resource information, and generating a cloud task scheduling scheme through a strategy network of a reinforcement learning model; for a cloud task scheduling scheme, obtaining scores and improvement suggestions through a task scheduling evaluation large model in combination with a cloud task scheduling rule, and updating the cloud task scheduling scheme through a trained strategy network of a reinforcement learning model after cloud tasks and resource information are adjusted based on the improvement suggestions; and continuously collecting resource conditions, cloud tasks, generated cloud task scheduling schemes, scores and improvement suggestions, after a period of time, adjusting a state space, an action space and a reward function of the reinforcement learning model according to the improvement suggestions, and performing incremental learning on the reinforcement learning model and the task scheduling evaluation large model. The cloud task execution efficiency and quality are effectively improved, and smooth completion of work is guaranteed.

Description

Technical Field

[0001] The present invention belongs to the technical field of task scheduling, and particularly relates to a cloud task scheduling method and system based on reinforcement learning and an evaluation model. Background Art

[0002] The statements in this part merely provide background technical information related to the present invention and do not necessarily constitute prior art.

[0003] In the current digital age, the cloud environment presents unprecedented complexity. Private clouds provide enterprises with a highly customized and secure internal computing resource environment. Enterprises can finely allocate resources according to their own business needs to ensure the security and privacy of core data. Public clouds, with their powerful resource pools and convenient service models, attract a large number of cost-sensitive users who pursue rapid deployment. Users can obtain computing, storage, and network resources on demand without investing a large amount of hardware facility costs. Hybrid clouds cleverly integrate the characteristics of private clouds and public clouds. Key business operations of enterprises run stably in the private cloud environment, while some non-core and elastic-demand services can be flexibly migrated to the public cloud to achieve the best balance between resource utilization and cost control. Edge clouds push computing power to the network edge, greatly reducing data transmission latency and providing strong support for application scenarios with extremely high real-time requirements, such as autonomous driving and industrial Internet of Things.

[0004] Workflow tasks have also become extremely complex. These cloud tasks no longer exist in isolation but are intertwined to form complex workflows with various structures. Some workflows exhibit a sequential execution structure, where subsequent tasks are started in sequence after the previous cloud task is completed. Some have a parallel execution structure, where multiple cloud tasks can be carried out simultaneously to improve the overall execution efficiency. Others have a loop structure, where specific cloud tasks need to be repeatedly executed until specific conditions are met. And different workflows have different goals, aiming for the shortest task completion time, or maximizing resource utilization, or ensuring high reliability in the execution of cloud tasks.

[0005] In this context, the efficiency and accuracy of cloud task scheduling become key factors in ensuring the efficient execution of workflows and the rational utilization of resources. However, traditional workflow task scheduling methods have obvious drawbacks. Most of them rely on fixed rules or simple heuristic algorithms. For example, resources are allocated in sequence according to the priority order of cloud tasks, or task allocation is determined based on the idle degree of resources. Since the resource status in the cloud environment changes constantly and the requirements of cloud tasks vary widely, fixed rules are difficult to adapt to dynamic situations, and simple heuristic algorithms cannot comprehensively consider numerous complex factors. This leads to the inability to achieve optimal scheduling in actual scheduling, resulting in low execution efficiency of cloud tasks, a large amount of resources being idle and wasted, or tasks waiting for resources for a long time due to unreasonable resource allocation, seriously affecting the overall progress speed of the workflow.

[0006] With the booming development of artificial intelligence technology, reinforcement learning algorithms have emerged in the field of cloud task scheduling, demonstrating certain advantages. Reinforcement learning algorithms continuously interact with the environment and learn the optimal scheduling strategy from the rewards feedback by the environment. It can adaptively adjust the scheduling plan according to the real-time state of resources in the cloud environment and the dynamic requirements of cloud tasks.

[0007] However, when encountering sudden irregular cloud tasks, due to the characteristics of such cloud tasks, which are quite different from the regular tasks that the reinforcement learning algorithm encounters during the training process, the reinforcement learning algorithm is difficult to accurately evaluate its impact on the overall scheduling, resulting in a decrease in the accuracy rate of the scheduling plan and being unable to effectively cope with complex and changing cloud task scenarios. Summary of the Invention

[0008] To solve the technical problems existing in the above background technology, the present invention provides a cloud task scheduling method and system based on reinforcement learning and an evaluation model. Through the powerful feature learning and representation ability of the large model, it can deeply mine the potential rules and feature information of cloud tasks and give modification suggestions to adjust cloud task and resource information, making the reinforcement learning model more accurate and reasonable when generating cloud task scheduling plans, and adjusting parameters such as the state space, action space, and reward function of the reinforcement learning model based on the improvement suggestions during incremental learning, discovering and solving potential problems in a timely manner, thereby effectively improving the execution efficiency and quality of cloud tasks and ensuring the smooth completion of work.

[0009] To achieve the above purpose, the present invention adopts the following technical solutions: The first aspect of the present invention provides a cloud task scheduling method based on reinforcement learning and an evaluation model, which includes: Obtain cloud task and resource information, and generate a cloud task scheduling plan through the policy network of the reinforcement learning model; For the cloud task scheduling plan, combine the cloud task scheduling rules, obtain the score and improvement suggestions through the task scheduling evaluation large model, and based on the improvement suggestions, after adjusting the cloud task and resource information, update the cloud task scheduling plan through the policy network of the trained reinforcement learning model; Continuously collect resource conditions, cloud tasks, generated cloud task scheduling plans, scores, and improvement suggestions. After a period of time, adjust the state space, action space, and reward function of the reinforcement learning model according to the improvement suggestions, and perform incremental learning on the reinforcement learning model and the task scheduling evaluation large model according to the resource conditions, cloud tasks, generated cloud task scheduling plans, scores, and improvement suggestions.

[0010] Furthermore, it also includes: jointly training the reinforcement learning model and the task scheduling evaluation large model; The steps of the joint training include: freezing the parameters of the task scheduling evaluation large model; receiving cloud tasks and resource status, generating a cloud task scheduling plan through the policy network of the reinforcement learning model, obtaining rewards after interacting with the environment, and updating the resource status at the same time; inputting the generated cloud task scheduling plan into the task scheduling evaluation large model to obtain scores and improvement suggestions; calculating the weighted sum of the rewards and the scores of the task scheduling evaluation large model to optimize the reinforcement learning model; freezing the parameters of the reinforcement learning model; receiving cloud tasks and resource status, generating a cloud task scheduling plan through the reinforcement learning model, evaluating the generated cloud task scheduling plan by the task scheduling evaluation large model to obtain scores and improvement suggestions, combining the results of manual evaluation, calculating the loss function, and updating the task scheduling evaluation large model.

[0011] Further, the weights used for calculating the weighted sum of the rewards and the scores of the task scheduling evaluation large model are adjusted according to the difference between the normalized rewards and scores and the number of training times of the policy network.

[0012] Further, the improvement suggestions for adjusting cloud task and resource information include: adjusting the available resource range in the resource information; splitting cloud tasks into different subtasks for parallel processing; adjusting the cloud task scheduling time sequence.

[0013] Further, the improvement suggestions for adjusting the reinforcement learning model include: adding an energy consumption penalty term to the reward function; adding a label indicating whether the resource is preemptable to the state space; restricting the action space.

[0014] The second aspect of the present invention provides a cloud task scheduling system based on reinforcement learning and an evaluation model, which includes: A scheduling plan generation module, which is configured to: obtain cloud task and resource information, and generate a cloud task scheduling plan through the policy network of the reinforcement learning model; A scheduling plan update module, which is configured to: for the cloud task scheduling plan, combine the cloud task scheduling rules, obtain scores and improvement suggestions through the task scheduling evaluation large model, and based on the improvement suggestions, adjust the cloud task and resource information, and then update the cloud task scheduling plan through the policy network of the trained reinforcement learning model; An incremental learning module, which is configured to: continuously collect resource conditions, cloud tasks, generated cloud task scheduling plans, scores and improvement suggestions. After a period of time, adjust the state space, action space and reward function of the reinforcement learning model according to the improvement suggestions, and perform incremental learning on the reinforcement learning model and the task scheduling evaluation large model according to the resource conditions, cloud tasks, generated cloud task scheduling plans, scores and improvement suggestions.

[0015] Further, it further includes a joint training module, which is configured to: perform joint training on the reinforcement learning model and the task scheduling evaluation large model; The steps of the joint training include: freezing the parameters of the task scheduling evaluation large model; receiving cloud tasks and resource status, generating a cloud task scheduling plan through the policy network of the reinforcement learning model, obtaining a reward after interacting with the environment, and updating the resource status at the same time; inputting the generated cloud task scheduling plan into the task scheduling evaluation large model to obtain a score and improvement suggestions; calculating the weighted sum of the reward and the score of the task scheduling evaluation large model to optimize the reinforcement learning model; freezing the parameters of the reinforcement learning model; receiving cloud tasks and resource status, generating a cloud task scheduling plan through the reinforcement learning model, and the task scheduling evaluation large model evaluates the generated cloud task scheduling plan to obtain a score and improvement suggestions, combining the artificial evaluation results, calculating the loss function, and updating the task scheduling evaluation large model.

[0016] Further, the weights used for calculating the weighted sum of the reward and the score of the task scheduling evaluation large model are adjusted according to the difference between the normalized reward and score and the number of training times of the policy network.

[0017] Further, the improvement suggestions for adjusting cloud task and resource information include: adjusting the available resource range in the resource information; splitting the cloud task into different subtasks for parallel processing; adjusting the cloud task scheduling time sequence.

[0018] Further, the improvement suggestions for adjusting the reinforcement learning model include: adding an energy consumption penalty term to the reward function; adding a label indicating whether the resource is preemptable in the state space; restricting the action space.

[0019] The third aspect of the present invention provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the steps in a cloud task scheduling method based on reinforcement learning and an evaluation model as described above are implemented.

[0020] The fourth aspect of the present invention provides a computer device, including a computer-readable storage medium, a processor, and a computer program stored on the computer-readable storage medium and executable on the processor. When the processor executes the program, the steps in a cloud task scheduling method based on reinforcement learning and an evaluation model as described above are implemented.

[0021] Compared with the prior art, the beneficial effects of the present invention are: Through the powerful feature learning and representation capabilities of the large model, the present invention can deeply explore the potential rules and feature information of cloud tasks and give modification suggestions to adjust cloud tasks and resource information, making the reinforcement learning model more accurate and reasonable when generating cloud task scheduling schemes, adaptively adjusting and optimizing the scheduling scheme, and improving the flexibility and adaptability of the scheduling method; and when performing incremental learning, adjusting parameters such as the state space, action space, and reward function of the reinforcement learning model based on the improvement suggestions, discovering and solving potential problems in a timely manner, thereby effectively improving the execution efficiency and quality of cloud tasks and ensuring the smooth completion of work.

[0022] The present invention jointly trains the reinforcement learning model and the task scheduling evaluation large model to achieve the co-evolution of the two models, enabling the reinforcement learning model to generate higher-quality scheduling schemes, while improving the accuracy of the evaluation of the evaluation large model and the effectiveness of the improvement suggestions.

[0023] Through a dynamic weight mechanism, the present invention organically combines the "environmental interaction feedback" of traditional reinforcement learning with the "prior knowledge guidance" of the large model, adaptively balancing exploration and exploitation during the training phase, and strengthening policy correction in divergent scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings forming a part of this invention are used to provide a further understanding of the invention. The schematic embodiments and descriptions thereof of the invention are used to explain the invention and do not constitute an improper limitation of the invention.

[0025] Figure 1 It is a flowchart of a cloud task scheduling method based on reinforcement learning and an evaluation model according to Embodiment 1 of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0026] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0027] It should be noted that the following detailed descriptions are all illustrative and are intended to provide further explanations of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which the present invention belongs.

[0028] Embodiment 1 This embodiment provides a cloud task scheduling method based on reinforcement learning and an evaluation model.

[0029] As described in the background art, when encountering sudden irregular cloud tasks, it is difficult for the reinforcement learning algorithm to accurately evaluate its impact on the overall scheduling, resulting in a decrease in the accuracy of the scheduling scheme. The large model can deeply explore the complex rules and potential information in cloud tasks, comprehensively and deeply analyze the attributes of cloud tasks, the characteristics of resources, and the complex relationships between them, and capture the subtle features and associations that are difficult to detect by traditional methods. In this embodiment, the large model is combined with the reinforcement learning algorithm to use the large model to provide a more accurate evaluation of the scheduling scheme for the reinforcement learning algorithm, helping the reinforcement learning algorithm to make better decisions when facing complex cloud tasks and emergencies, so as to construct a more intelligent, flexible and efficient cloud task scheduling method to meet the scheduling needs of diverse cloud tasks in today's complex cloud environment.

[0030] A cloud task scheduling method based on reinforcement learning and an evaluation model provided in this embodiment realizes the efficient scheduling of cloud tasks, improves resource utilization and task execution efficiency by combining the reinforcement learning algorithm to calculate the cloud task scheduling strategy and using the large model to accurately evaluate the cloud task scheduling scheme.

[0031] A cloud task scheduling method based on reinforcement learning and an evaluation model provided in this embodiment, as Figure 1 shown, includes the following steps: Step 1: Training of the reinforcement learning model.

[0032] Use a large amount of cloud task and resource data to train the reinforcement learning model, with the input being cloud task and resource information and the output being the cloud task scheduling scheme.

[0033] Specifically, abstract each link and factor of the workflow task scheduling into a reinforcement learning environment, define the state space, action space and reward function, and through continuous trial and error, enable the agent to gradually learn the optimal actions in different states, that is, the optimal cloud task scheduling strategy.

[0034] Among them, the reinforcement learning model is based on the Deep Q Network (DQN) algorithm, and its training mainly includes the following steps: Step 101: Collect historical cloud task and resource information and perform data cleaning to improve data quality.

[0035] Step 102: Set the state space, including information such as the attributes of cloud tasks (such as task priority, task composition, subtask completion status, quality of service requirements, etc.) and the available status of resources (such as resource quantity, link status, etc.), and use to represent.

[0036] Step 103: Set the action space, representing the schedulable operations (cloud task scheduling scheme), that is, assign the cloud task to a specific resource, and use representation

[0037] Step 104: Set up a reward function for evaluating the quality of scheduling operations, denoted by representation

[0038] The reward function is designed according to the goals of cloud task scheduling, such as: cloud task execution cost, cloud task completion time, resource utilization rate, etc. For multiple goals, comprehensive evaluation is carried out through weighted sum.

[0039] For example, considering the cloud task execution cost and the cloud task completion time, the reward function is:[[]] ; where is the cloud instance cost required to execute the cloud task, is the cloud task completion time, is the trade-off coefficient for controlling and proportion. If , then the cloud task scheduling only considers the cloud instance cost; if , then the cloud task scheduling only considers the completion time.

[0040] Step 105: Use the data ( ) to train the policy Q-network using the gradient descent method; where represents the moment , represents the next moment of the moment ; ( ) represents that for the state , the action is selected and executed through the action selection algorithm, and the state is reached.

[0041] Specifically, at each moment , based on the current state , the action is selected through the action selection algorithm and executed. The policy Q-network outputs the predicted Q value , the reward is obtained after interacting with the environment, and at the same time, the next state is entered, and the experience sample ( ) is generated and stored in the experience replay buffer; calculate the target Q value of the target Q-network in the case of ; calculate the theoretical value of the policy Q-network in the case of , where is the discount factor, indicating that the rewards generated in future steps are considered to a certain extent; calculate the loss function of the policy Q-network ; Update the parameters of the policy Q-network using the gradient descent method to approximate the predicted Q-value to the theoretical value.

[0042] Among them, the action selection algorithm adopts algorithm, and the main process is: with probability, select the that maximizes the predicted Q-value; with probability, randomly select ; among them, , is the adjustment step size, and as the number of iterations increases, gradually decreases.

[0043] Step 106: Repeat training the policy Q-network times, and then update the parameters of the target Q-network with the parameters of the policy Q-network. Among them, is the update interval of the target Q-network parameters. The target Q-network has the same structure as the policy Q-network and shares parameters.

[0044] Step 2: Evaluate the large model training of task scheduling.

[0045] Summarize the cloud task scheduling rules, including cloud task splitting rules, cloud task objectives, resource usage constraints, priorities, etc.; collect a large number of different cloud task scheduling schemes, and manually evaluate the cloud task scheduling schemes according to the cloud task scheduling rules, and label the scores (such as numerical scores from 0 to 10) and improvement suggestions (such as adjusting the cloud task execution order, changing resource allocation, etc.).

[0046] The task scheduling evaluation large model selects a suitable large model architecture such as Tongyi Qianwen (qwen), and uses the cloud task scheduling rules and historical data to fine-tune the large model using the Low-Rank Adaptation of Large Language Models (LoRA). The input is the feature representation of the cloud task scheduling scheme and the cloud task scheduling rules, and the output is the scoring result and improvement suggestions for the cloud task scheduling scheme; by continuously adjusting the large model parameters, the large model can accurately evaluate the cloud task scheduling scheme.

[0047] Step 3: Joint training of the reinforcement learning model and the task scheduling evaluation large model.

[0048] By incorporating the scores of the task scheduling evaluation large model into the reinforcement learning reward system, and using the cloud task scheduling scheme and the manual evaluation results to train the large model, the co-evolution of the two models is achieved; the ultimate goal is to enable the reinforcement learning model to generate higher-quality cloud task scheduling schemes, while improving the accuracy of the task scheduling evaluation large model and the effectiveness of the improvement suggestions.

[0049] Step 301: Freeze the parameters of the task scheduling evaluation model and train the reinforcement learning model.

[0050] In each training step, the reinforcement learning model receives the current cloud task and resource status as input, and generates a cloud task scheduling solution based on its current policy network (policy Q network) combined with the action selection algorithm. , get rewards after interacting with the environment , and enter the next resource state at the same time ; The generated cloud task scheduling plan Input into the task scheduling evaluation model to obtain the score and improvement suggestions given by the task scheduling evaluation model; according to the reward And the task scheduling evaluation model score, calculate the reward value, and get the new reward ; Based on the new rewards , optimize the policy Q network and target Q network of the reinforcement learning model (specifically, generate experience samples ( ) and store it in the experience playback buffer; calculate the target Q network in The target Q value in the case ; Computational strategy Q network in Theoretical value under the condition ; Calculate the loss function of the policy Q network ; Use gradient descent method to update the policy Q network parameters so that the predicted Q value is close to the theoretical value; repeat the training of the policy Q network Afterwards, the policy Q network parameters are used to update the target Q network parameters).

[0051] The reward of the reinforcement learning model consists of multiple parts, including the goal of cloud task scheduling (that is, the reward function of the original reinforcement learning model) and the score of the task scheduling evaluation model, thereby iteratively adjusting the parameters of the reinforcement learning model.

[0052] The new reward function is: ;in, is the reward function of the original reinforcement learning, Scoring of large models for task scheduling evaluation; if the magnitudes of the two parts differ greatly, the data can be normalized to eliminate the impact of the dimension on the final result, making different variables, especially those with large magnitude differences, comparable.

[0053] As another implementation, the new reward function is: ;in, is the weight, according to the normalized and The difference RC And the number of training times n of the strategy Q network is adjusted, that is, 。Through the dynamic weight mechanism, the "environmental interaction feedback" of traditional reinforcement learning is organically combined with the "prior knowledge guidance" of large models, adaptively balancing exploration and exploitation during the training phase, and strengthening policy correction in divergent scenarios. At the initial stage of training or when RC is large, is small, the weight of the large model score S is high, forcing the agent to preferentially learn the global knowledge of the large model and accelerating the exploration of the effective policy space; in the later stage of training, as n increases or RC decreases, the weight of the traditional reward increases, and the model gradually focuses on using the learned policies to optimize local details and improve the convergence efficiency. Avoiding the problems of "insufficient exploration in the early stage" or "over-reliance on prior knowledge in the later stage" caused by fixed weights, and adaptively balancing the learning objectives at different stages.

[0054] Step 302: Freeze the parameters of the reinforcement learning model and train the task scheduling evaluation large model.

[0055] Use the cloud task scheduling scheme generated by the reinforcement learning model and the corresponding manual evaluation results (scores and improvement suggestions) as input data; for the score part, input the cloud task scheduling scheme into the task scheduling evaluation large model, and adjust the parameters of the task scheduling evaluation large model by minimizing the mean square error (MSE) between the output score of the task scheduling evaluation large model and the manual score; for the improvement suggestion part, encode the improvement suggestions (such as using text embedding), and then train the task scheduling evaluation large model through a similar loss function (such as, cross-entropy loss function or mean square error loss function) to enable it to output more accurate and effective improvement suggestions.

[0056] Step 4: Generate the cloud task scheduling scheme.

[0057] When a new cloud task arrives, obtain the cloud task and the current resource information, and input the cloud task and the current resource information into the policy Q-network of the trained reinforcement learning model.

[0058] The policy Q-network outputs a cloud task scheduling scheme for the current cloud task and resource information according to the learned optimal policy, that is, determine on which resources each cloud task is executed and the execution order.

[0059] Step 5: Evaluate the cloud task scheduling scheme.

[0060] Evaluate the cloud task scheduling scheme based on the trained task scheduling evaluation large model, and adjust the input parameters of the policy Q-network according to the improvement suggestions of the task scheduling evaluation large model, and regenerate the cloud task scheduling scheme using the policy Q-network again.

[0061] Repeat the process of cloud task scheduling scheme generation - cloud task scheduling scheme evaluation until the final cloud task scheduling scheme is obtained.

[0062] Among them, the input parameters for adjusting the policy Q-network refer to: according to the improvement suggestions, adjusting the available resource range in the resource information to exclude resources that are not suitable for executing the task; splitting the long-term cloud task into different subtasks for parallel processing to improve the task execution efficiency; adjusting the cloud task scheduling timing, advancing / postponing the start time of the cloud task to avoid resource conflicts, etc.

[0063] This process is not a simple repetition, but on the basis of continuous optimization and adjustment, gradually approaching the optimal solution. The task scheduling evaluation large model will continuously repeat the above processes of evaluation, improvement, update, and rescheduling. Each round of loop is based on the optimization results of the previous round until finally obtaining an optimal task scheduling plan. The final plan will become the core basis for guiding the actual task execution, ensuring that the cloud tasks can be efficiently and orderly promoted.

[0064] Step 6: Execute the cloud task scheduling plan.

[0065] After obtaining the final cloud task scheduling plan, deploy the cloud tasks to specific resources for execution according to the final cloud task scheduling plan.

[0066] During the execution of the cloud task, monitor the execution status of each cloud task in real time. Through monitoring, information such as the actual start time, current execution progress, and estimated completion time of the cloud task can be obtained. If an abnormality is found in a certain cloud task during the execution process, such as an overly long execution time or resource usage exceeding expectations, collect relevant feedback information in a timely manner. This feedback information can be used as a reference for subsequent optimization of the cloud task scheduling plan. At the same time, for the completed cloud tasks, record detailed data such as their actual completion time and resource consumption, which are used to evaluate the overall effect of this cloud task execution and provide data support for further optimizing the scheduling algorithm and evaluation large model.

[0067] Step 7: Iterative optimization.

[0068] Continuously collect data on resource conditions, cloud task information, cloud task scheduling plans, scores, and improvement suggestions. After a period of data collection, adjust the reinforcement learning model according to the improvement suggestions, and perform incremental learning on the reinforcement learning model and the task scheduling evaluation large model based on resource conditions, task information, task scheduling plans, scores, and improvement suggestions, so that the model adapts to the new environment and at the same time avoids catastrophic forgetting, realizing the continuous optimization and iteration of the model.

[0069] The task scheduling evaluation large model will also output some improvement suggestions for the reinforcement learning model. For example, the cloud task scheduling scheme overemphasizes cost and completion time, but has too high energy consumption. It is recommended to add an energy consumption penalty term to the reward function; the cloud task scheduling scheme does not consider whether the resources are preemptible. It is recommended to add a label indicating whether the resources are preemptible to the state space; certain types of tasks are not suitable for using certain resources. For example, tasks with extremely high accuracy requirements are not suitable for using unstable and easily interrupted resources, which restricts the action space and thus increases the constraints for action selection. By adjusting the parameters such as the state space, action space, and reward function of the reinforcement learning model based on these improvement suggestions, the reinforcement learning model can better adapt to the existing environment.

[0070] Embodiment 2 This embodiment provides a cloud task scheduling system based on reinforcement learning and an evaluation model, which specifically includes: A scheduling scheme generation module, which is configured to: obtain cloud task and resource information, and generate a cloud task scheduling scheme through the policy network of the reinforcement learning model; A scheduling scheme update module, which is configured to: for the cloud task scheduling scheme, combine the cloud task scheduling rules, obtain a score and improvement suggestions through the task scheduling evaluation large model, and based on the improvement suggestions, adjust the cloud task and resource information, and then update the cloud task scheduling scheme through the policy network of the trained reinforcement learning model; An incremental learning module, which is configured to: continuously collect resource conditions, cloud tasks, generated cloud task scheduling schemes, scores, and improvement suggestions. After a period of time, adjust the state space, action space, and reward function of the reinforcement learning model according to the improvement suggestions, and perform incremental learning on the reinforcement learning model and the task scheduling evaluation large model according to the resource conditions, cloud tasks, generated cloud task scheduling schemes, scores, and improvement suggestions.

[0071] Furthermore, it further includes a joint training module, which is configured to: jointly train the reinforcement learning model and the task scheduling evaluation large model; The steps of the joint training include: freezing the parameters of the task scheduling evaluation large model; receiving cloud tasks and resource states, generating a cloud task scheduling scheme through the policy network of the reinforcement learning model, obtaining a reward after interacting with the environment, and simultaneously updating the resource states; inputting the generated cloud task scheduling scheme into the task scheduling evaluation large model to obtain a score and improvement suggestions; calculating the weighted sum of the reward and the score of the task scheduling evaluation large model, and optimizing the reinforcement learning model; freezing the parameters of the reinforcement learning model; receiving cloud tasks and resource states, generating a cloud task scheduling scheme through the reinforcement learning model, the task scheduling evaluation large model evaluates the generated cloud task scheduling scheme to obtain a score and improvement suggestions, combining the results of manual evaluation, calculating the loss function, and updating the task scheduling evaluation large model.

[0072] Further, the weights used for calculating the weighted sum of the reward and the task scheduling evaluation large model score are adjusted according to the difference between the normalized reward and score and the number of training times of the policy network.

[0073] Further, the improvement suggestions for adjusting cloud tasks and resource information include: adjusting the available resource range in the resource information; splitting cloud tasks into different subtasks for parallel processing; adjusting the cloud task scheduling timing.

[0074] Further, the improvement suggestions for adjusting the reinforcement learning model include: adding an energy consumption penalty term to the reward function; adding a label indicating whether the resource is preemptable to the state space; restricting the action space.

[0075] It should be noted here that each module in this embodiment corresponds one by one to each step in Embodiment 1, and the specific implementation process is the same, so it will not be repeated here.

[0076] The above are only the preferred embodiments of the present invention and are not used to limit the present invention. For those skilled in the art, the present invention can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A cloud task scheduling method based on reinforcement learning and an evaluation model, characterized in that Including: Obtain cloud task and resource information, and generate a cloud task scheduling plan through the policy network of the reinforcement learning model; For the cloud task scheduling plan, combine the cloud task scheduling rules, obtain a score and improvement suggestions through the task scheduling evaluation large model, and based on the improvement suggestions, after adjusting the cloud task and resource information, update the cloud task scheduling plan through the policy network of the trained reinforcement learning model; Continuously collect resource conditions, cloud tasks, generated cloud task scheduling plans, scores and improvement suggestions. After a period of time, adjust the state space, action space and reward function of the reinforcement learning model according to the improvement suggestions, and perform incremental learning on the reinforcement learning model and the task scheduling evaluation large model according to the resource conditions, cloud tasks, generated cloud task scheduling plans, scores and improvement suggestions.

2. The cloud task scheduling method based on reinforcement learning and evaluation model according to claim 1, characterized in that It also includes: jointly training the reinforcement learning model and the task scheduling evaluation large model; The steps of the joint training include: freezing the parameters of the task scheduling evaluation large model; receiving cloud tasks and resource status, generating a cloud task scheduling plan through the policy network of the reinforcement learning model, obtaining a reward after interacting with the environment, and updating the resource status at the same time; inputting the generated cloud task scheduling plan into the task scheduling evaluation large model to obtain a score and improvement suggestions; calculating the weighted sum of the reward and the score of the task scheduling evaluation large model, and optimizing the reinforcement learning model; freezing the parameters of the reinforcement learning model; receiving cloud tasks and resource status, generating a cloud task scheduling plan through the reinforcement learning model, and the task scheduling evaluation large model evaluates the generated cloud task scheduling plan to obtain a score and improvement suggestions, combining the manual evaluation results, calculating the loss function, and updating the task scheduling evaluation large model.

3. The cloud task scheduling method based on reinforcement learning and evaluation model according to claim 2, wherein, The weights used for calculating the weighted sum of the reward and the score of the task scheduling evaluation large model are adjusted according to the difference between the normalized reward and score and the number of training times of the policy network.

4. The cloud task scheduling method based on reinforcement learning and evaluation model according to claim 1, characterized in that The improvement suggestions for adjusting cloud tasks and resource information include: adjusting the available resource range in the resource information; splitting cloud tasks into different subtasks for parallel processing; adjusting the cloud task scheduling timing.

5. A cloud task scheduling method based on reinforcement learning and an evaluation model according to claim 1, characterized in that, The improvement suggestions for adjusting the reinforcement learning model include: adding an energy consumption penalty term to the reward function; adding a label indicating whether the resource is preemptable to the state space; restricting the action space.

6. A cloud task scheduling system based on reinforcement learning and an evaluation model, characterized in that, Including: A scheduling plan generation module, which is configured to: obtain cloud task and resource information, and generate a cloud task scheduling plan through the policy network of the reinforcement learning model; A scheduling plan update module, which is configured to: for the cloud task scheduling plan, combine the cloud task scheduling rules, obtain a score and improvement suggestions through the task scheduling evaluation large model, and based on the improvement suggestions, after adjusting the cloud task and resource information, update the cloud task scheduling plan through the policy network of the trained reinforcement learning model; An incremental learning module, which is configured to: continuously collect resource status, cloud tasks, generated cloud task scheduling schemes, scores, and improvement suggestions. After a period of time, adjust the state space, action space, and reward function of the reinforcement learning model according to the improvement suggestions, and perform incremental learning on the reinforcement learning model and the task scheduling evaluation large model based on the resource status, cloud tasks, generated cloud task scheduling schemes, scores, and improvement suggestions.

7. A cloud task scheduling system based on reinforcement learning and an evaluation model according to claim 6, characterized in that, It further includes a joint training module, which is configured to: jointly train the reinforcement learning model and the task scheduling evaluation large model; The steps of the joint training include: freezing the parameters of the task scheduling evaluation large model; receiving cloud tasks and resource status, generating a cloud task scheduling scheme through the policy network of the reinforcement learning model, obtaining a reward after interacting with the environment, and simultaneously updating the resource status; inputting the generated cloud task scheduling scheme into the task scheduling evaluation large model to obtain scores and improvement suggestions; calculating the weighted sum of the reward and the score of the task scheduling evaluation large model to optimize the reinforcement learning model; freezing the parameters of the reinforcement learning model; receiving cloud tasks and resource status, generating a cloud task scheduling scheme through the reinforcement learning model, the task scheduling evaluation large model evaluates the generated cloud task scheduling scheme to obtain scores and improvement suggestions, combines the results of manual evaluation, calculates the loss function, and updates the task scheduling evaluation large model.

8. A cloud task scheduling system based on reinforcement learning and an evaluation model according to claim 7, characterized in that, The weights used for calculating the weighted sum of the reward and the score of the task scheduling evaluation large model are adjusted according to the difference between the normalized reward and score and the number of training times of the policy network.

9. A cloud task scheduling system based on reinforcement learning and an evaluation model as described in claim 6, wherein, The improvement suggestions for adjusting cloud tasks and resource information include: adjusting the available resource range in the resource information; splitting cloud tasks into different subtasks for parallel processing; adjusting the cloud task scheduling timing.

10. A cloud task scheduling system based on reinforcement learning and an evaluation model according to claim 6, characterized in that, The improvement suggestions for adjusting the reinforcement learning model include: adding an energy consumption penalty term to the reward function; adding a label indicating whether the resource is preemptable to the state space; restricting the action space.

Images