Task scheduling method and device, equipment, storage medium and program product

Abstract

The invention relates to a task scheduling method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the steps of obtaining a to-be-processed workflow, wherein the to-be-processed workflow comprises a plurality of to-be-processed tasks; obtaining first state data of each to-be-processed task; through a preset task scheduling model based on deep reinforcement learning, target to-be-processed tasks are sequentially selected from all the to-be-processed tasks according to all the first state data, corresponding target servers are allocated to the selected target to-be-processed tasks, and the preset task scheduling model based on deep reinforcement learning is used for scheduling the target to-be-processed tasks according to the target to-be-processed tasks. The method is obtained through training according to historical state data, historical scheduling action data and historical instant reward values of historical workflows in advance. By adopting the method, the workflow task scheduling effect can be improved.

Description

Technical Field

[0001] The present application relates to the field of workflow technology, and in particular to a task scheduling method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] In the field of insurance digitalization, insurance data services involve comprehensive services in multiple aspects such as business intelligence, regulatory reporting, and customer information push. In this field, workflow task scheduling for processing data is extremely important and plays an extremely important role. Reasonably arranging the execution order of workflow tasks in the cloud environment to maximize system performance, reduce resource utilization costs, and meet various constraints has become one of the important means for enterprises to reduce costs and increase efficiency.

[0003] In traditional technologies, workflow data scheduling optimization algorithms are usually heuristic algorithms, such as greedy algorithms, first-in-first-out algorithms, local search, taboo search, etc. Some are meta-heuristic algorithms, such as ant colony algorithm, artificial bee colony ABC, bacterial algorithm, particle swarm optimization, black widow algorithm, etc.

[0004] However, traditional heuristic algorithms and metaheuristic algorithms are subject to the quality of data, and the parameters fitted by the model may reach local extremes, resulting in low efficiency and accuracy of workflow task scheduling and poor workflow task scheduling effects. Summary of the invention

[0005] Based on this, it is necessary to provide a task scheduling method, apparatus, computer equipment, computer-readable storage medium and computer program product that can improve the workflow task scheduling effect in response to the above technical problems.

[0006] In a first aspect, the present application provides a task scheduling method, comprising:

[0007] Obtain a pending workflow, where the pending workflow includes multiple pending tasks;

[0008] Obtaining first status data of each task to be processed;

[0009] Through the preset task scheduling model based on deep reinforcement learning, according to each first state data, the target pending task is selected from each pending task in turn, and the corresponding target server is assigned to the selected target pending task, wherein the preset task scheduling model based on deep reinforcement learning is pre-trained according to the historical state data, historical scheduling action data and historical immediate reward values ​​of the historical workflow.

[0010] In one embodiment, the first state data includes at least one of first task attribute data, first task state data, first task priority data, and first task dependency data; through a preset task scheduling model based on deep reinforcement learning, according to each first state data, a target task to be processed is selected from each task to be processed in turn, and a corresponding target server is allocated to the selected target task to be processed, including:

[0011] Performing vector concatenation on each first state data corresponding to each task to be processed to obtain a first state vector corresponding to each task to be processed;

[0012] Through a preset task scheduling model based on deep reinforcement learning, target tasks to be processed are selected from each task to be processed in turn according to the first state vector corresponding to each task to be processed, and the corresponding target server is allocated to the selected target task to be processed.

[0013] In one embodiment, the preset task scheduling model is a deep Q network, and the deep Q network includes a current value network; through the preset task scheduling model based on deep reinforcement learning, according to each first state data, a target task to be processed is selected from each task to be processed in sequence, and a corresponding target server is allocated to the selected target task to be processed, including:

[0014] Selection step: selecting a target task to be processed that is executed at the current time step from among the tasks to be processed through the current value network according to each first state data, and assigning a corresponding target server to the selected target task to be processed;

[0015] The target pending task is determined as a processed task, and the execution of the selection step is returned.

[0016] In one embodiment, the deep Q network further includes a target value network; through the current value network, according to each first state data, a target task to be processed executed at the current time step is selected from each task to be processed, and a corresponding target server is allocated to the selected target task to be processed, including:

[0017] Through the current value network, according to each first state data, a target task to be processed that is executed at the current time step is selected from each task to be processed, a corresponding target server is assigned to the selected target task to be processed, and a value assessment prediction value corresponding to the target task to be processed is predicted;

[0018] After selecting a target task to be processed to be executed in the current time step from each task to be processed through the current value network according to each first state data, and assigning a corresponding target server to the selected target task to be processed, the method further includes:

[0019] Obtain the second state data of the task to be processed to be executed in the next time step, and predict the future reward value corresponding to the target task to be processed according to the second state data through the target value network;

[0020] Obtain the current environment feedback data corresponding to the target pending task, and detect the immediate reward value corresponding to the target pending task according to each current environment feedback data;

[0021] Aggregate the immediate reward value and the future reward value to obtain the value assessment target value corresponding to the target pending task;

[0022] According to the value assessment prediction value and the value assessment target value, the current value network is optimized.

[0023] In one embodiment, detecting the immediate reward value corresponding to the target task to be processed according to each current environment feedback data includes:

[0024] According to the current environment feedback data, detect the task completion reward value, task failure reward value and resource utilization reward value corresponding to the target pending task;

[0025] Aggregate the task completion reward value, task failure reward value and resource utilization reward value to obtain the immediate reward value corresponding to the target pending task.

[0026] In one embodiment, the first state data includes at least one of an estimated execution time and a task priority; the current environment feedback data includes at least one of action completion result data, actual execution time and actual resource utilization rate; and according to the current environment feedback data, detecting a task completion reward value, a task failure reward value and a resource utilization reward value corresponding to the target pending task includes at least one of the following:

[0027] If it is determined that the target task to be processed has failed to be completed based on the estimated execution time, the action completion result data corresponding to the target task to be processed, and the actual execution time, the task failure reward value corresponding to the target task to be processed is set to a preset negative reward value;

[0028] When the target pending task is determined to be successfully completed based on the estimated execution time, the action completion result data corresponding to the target pending task and the actual execution time, the task completion reward value corresponding to the target pending task is calculated according to the task priority, the estimated execution time and the actual execution time, wherein the task completion reward value is positively correlated with the task priority, and the task completion reward value is negatively correlated with the actual execution time;

[0029] A resource utilization difference value between an actual resource utilization rate and a preset target resource utilization rate is detected, and a resource utilization reward value is calculated based on the resource utilization difference value, wherein the resource utilization reward value is negatively correlated with the resource utilization difference value.

[0030] In a second aspect, the present application also provides a task scheduling device, comprising:

[0031] A first acquisition module is used to acquire a workflow to be processed, where the workflow to be processed includes a plurality of tasks to be processed;

[0032] A second acquisition module is used to acquire first status data of each task to be processed;

[0033] The prediction module is used to select target pending tasks from each pending task in turn according to each first state data through a preset task scheduling model based on deep reinforcement learning, and assign a corresponding target server to the selected target pending task, wherein the preset task scheduling model based on deep reinforcement learning is pre-trained based on historical state data, historical scheduling action data and historical immediate reward values ​​of historical workflows.

[0034] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0035] Obtain a pending workflow, where the pending workflow includes multiple pending tasks;

[0036] Obtaining first status data of each task to be processed;

[0037] Through the preset task scheduling model based on deep reinforcement learning, according to each first state data, the target pending task is selected from each pending task in turn, and the corresponding target server is assigned to the selected target pending task, wherein the preset task scheduling model based on deep reinforcement learning is pre-trained according to the historical state data, historical scheduling action data and historical immediate reward values ​​of the historical workflow.

[0038] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the following steps are implemented:

[0039] Obtain a pending workflow, where the pending workflow includes multiple pending tasks;

[0040] Obtaining first status data of each task to be processed;

[0041] Through the preset task scheduling model based on deep reinforcement learning, according to each first state data, the target pending task is selected from each pending task in turn, and the corresponding target server is assigned to the selected target pending task, wherein the preset task scheduling model based on deep reinforcement learning is pre-trained according to the historical state data, historical scheduling action data and historical immediate reward values ​​of the historical workflow.

[0042] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0043] Obtain a pending workflow, where the pending workflow includes multiple pending tasks;

[0044] Obtaining first status data of each task to be processed;

[0045] Through the preset task scheduling model based on deep reinforcement learning, according to each first state data, the target pending task is selected from each pending task in turn, and the corresponding target server is assigned to the selected target pending task, wherein the preset task scheduling model based on deep reinforcement learning is pre-trained according to the historical state data, historical scheduling action data and historical immediate reward values ​​of the historical workflow.

[0046] The above-mentioned task scheduling method, device, computer equipment, computer-readable storage medium and computer program product first obtain a to-be-processed workflow, which includes multiple to-be-processed tasks, and then obtain the first state data of each to-be-processed task. Through a preset task scheduling model based on deep reinforcement learning, according to each first state data, a target to-be-processed task is selected from each to-be-processed task in turn, and a corresponding target server is assigned to the selected target to-be-processed task, wherein the preset task scheduling model based on deep reinforcement learning is pre-trained according to the historical state data, historical scheduling action data and historical instant reward value of the historical workflow. The deep reinforcement learning model is used to allocate the execution order and server of each to-be-processed task in the to-be-processed workflow. The deep neural network has a strong representation ability and can capture the complex patterns and dependencies in the data, so as to find the global optimal solution in a larger search space. Compared with traditional heuristic algorithms and meta-heuristic algorithms, the deep reinforcement learning model can not only use the known optimal solution, but also improve the generalization ability of the model by capturing the correlation between different tasks and data, and reduce the risk of falling into local extreme values, thereby improving the efficiency and accuracy of workflow task scheduling, making the workflow task scheduling effect better. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0048] Figure 1 A flowchart of a task scheduling method in one embodiment;

[0049] Figure 2 is a flowchart of a task scheduling method in another embodiment;

[0050] Figure 3 A schematic diagram of detection results of response time ratios corresponding to different task scheduling methods in an embodiment;

[0051] Figure 4 A schematic diagram showing how a predicted value of value assessment varies with the number of training times in one embodiment;

[0052] Figure 5 A schematic diagram of the number of tasks executed by a server corresponding to different task scheduling methods in an embodiment;

[0053] Figure 6 is a structural block diagram of a task scheduling device in one embodiment;

[0054] Figure 7 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0056] In an exemplary embodiment, Figure 1 As shown, a task scheduling method is provided. This embodiment takes the method applied to a terminal as an example, wherein the terminal may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things devices may be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. Portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The head-mounted devices may be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. It can be understood that the method can also be applied to servers, and can also be applied to systems including terminals and servers, and implemented through the interaction between terminals and servers. In this embodiment, the method includes the following steps S10-S30. Among them:

[0057] Step S10: obtaining a workflow to be processed, where the workflow to be processed includes a plurality of tasks to be processed.

[0058] A workflow can refer to a group of interrelated tasks that are organized in a certain order and logical relationship to form a workflow to complete a specific business goal or process. Workflow task scheduling can refer to the process of rationally allocating tasks to computing resources and coordinating the execution order of tasks according to predefined rules and policies. A pending workflow can refer to a workflow that has not been scheduled and executed but needs to be scheduled and executed. A pending workflow is formed by a group of interrelated pending tasks organized in a certain order and logical relationship.

[0059] Exemplarily, the business end can generate a pending workflow according to actual business needs, and initiate a workflow task scheduling request to schedule the pending workflow to the terminal, wherein the workflow task scheduling request carries the pending workflow. After receiving the workflow task scheduling request, the terminal can extract the pending workflow and multiple pending tasks contained in the pending workflow from the workflow task scheduling request.

[0060] Step S20, obtaining first status data of each task to be processed.

[0061] Among them, the state data may refer to the representation of the specific situation or configuration of the task in the environment, including at least one of task attribute data, task state data, task priority data, task resource requirement data and task dependency data. Among them, the task attribute data may include at least one of the identification information, type and execution time of the task, and these task attribute data provide a basic description for each task, which is used to identify and distinguish different tasks. The task priority data is used to indicate the urgency of the task. The priority of different tasks can be set according to actual business needs, etc. This embodiment is not limited to this. As an example, the task priority data can be represented by an integer in a certain numerical range. The higher the numerical value of the number, the higher the priority. For example, it can be represented by a number 1-20, where the number 1 indicates the lowest priority and the number 20 indicates the highest priority. The task resource requirement data is used to characterize the resource requirement of each task, including at least one of the CPU (Central Processing Unit) resource requirement and the memory resource requirement, etc., which can be evaluated based on the historical data of historical tasks of the same task type as each task. The task status data is used to characterize the current status of each task, including at least one of 0 commit succeeded, 1 running, 2 prepare to pause, 3 pause, 4 prepare to stop, 5 stop, 6 fail, 7 succeed, 8 need fault tolerance, 9 kill, 10 wait for thread, 11 wait for dependency to complete, etc., where the number can be the state number of each state, and the task status data can be the state number. The corresponding state information can be queried based on the state number, and the change of task status reflects the dynamic characteristics of the workflow. Task dependency data is used to describe the dependency relationship between tasks, such as the existence of dependency, the absence of dependency, the existence of timing dependency, etc. Tasks with dependencies can be scheduled based on the dependencies. For example, tasks with timing dependencies can be executed in sequence in a continuous manner to avoid resource waste and time delay.

[0062] The first state data may refer to the state data of the task to be processed at the current time step. The first state data may be sent to the terminal by the business end when initiating a workflow task scheduling request, or may be detected by the terminal from one or more corresponding functional modules, which is not limited in this embodiment.

[0063] Exemplarily, when the business end generates a pending workflow according to actual business needs, it can obtain the first state data of each pending task in the pending workflow through user input, information query, etc., and then send the first state data together with the workflow task scheduling request to the terminal when initiating a workflow task scheduling request. After the terminal obtains the pending workflow and determines the pending tasks in the pending workflow, it can also extract the first state data corresponding to each pending task from the workflow task scheduling request.

[0064] Step S30, through the preset task scheduling model based on deep reinforcement learning, according to each first state data, select the target task to be processed from each task to be processed in turn, and assign the corresponding target server to the selected target task to be processed, wherein the preset task scheduling model based on deep reinforcement learning is pre-trained according to the historical state data, historical scheduling action data and historical instant reward values ​​of the historical workflow.

[0065] The preset task scheduling model based on deep reinforcement learning may refer to a model that optimizes the task scheduling strategy by using deep reinforcement learning technology through interactive learning with the environment. The preset task scheduling model based on deep reinforcement learning may be used to predict scheduling actions, and the scheduling actions may include task selection and resource allocation, that is, the scheduling action may be to select a pending task from the pending tasks as the target pending task currently being executed, and to allocate a corresponding target server for the target pending task. It can be understood that when there are multiple pending tasks, the number of scheduling actions is the same as the number of pending tasks, that is, it is possible to predict scheduling actions that are executed in sequence at multiple time steps to achieve scheduling of all pending tasks. For example, assuming that the pending workflow includes N pending tasks and M servers are deployed, then in the process of scheduling tasks for the pending workflow, a total of N scheduling actions can be predicted, wherein the first scheduling action can be to select one from the N pending tasks as the target pending task, and assign the selected target pending task to one of the M servers for processing. After the target pending task is executed, the target pending task can be determined as a processed task, thereby removing it from the set of pending tasks, and then predicting the second scheduling action. The second scheduling action can be to select one from the N-1 pending tasks as the target pending task, and assign the selected target pending task to one of the M servers for processing, and so on. The cycle continues until all N pending tasks are executed.

[0066] Exemplarily, after obtaining the first state data of each pending task, each first state data can be input into a pre-trained preset task scheduling model based on deep reinforcement learning, and the preset task scheduling model predicts scheduling actions based on each first state data, and predicts scheduling actions to be executed sequentially in multiple time steps. Among them, for each time step, the preset task scheduling model can predict the scheduling action with the highest estimated value based on the first state data of the currently remaining pending tasks, determine the pending task indicated in the scheduling action with the highest estimated value as the target pending task, and determine the server indicated in the scheduling action with the highest estimated value as the target server.

[0067] Among them, each scheduling action at least includes information about the pending tasks executed at the current moment and information about the server corresponding to the pending tasks. For example, the scheduling action may be to assign pending task A to a server B for processing. Therefore, the preset task scheduling model can predict the scheduling action with the highest estimated value based on the first state data of the currently remaining pending tasks, which may include: generating corresponding scheduling actions based on the pending tasks remaining at the current time step, and the scheduling actions correspond to the pending tasks one by one; combining each first state data and each scheduling action to obtain task scheduling combination data, each task scheduling combination data includes the first state data corresponding to any pending task and the scheduling action corresponding to the pending task; performing estimated value prediction based on each task scheduling combination data through the preset task scheduling model, and selecting the scheduling action with the highest estimated value.

[0068] In some feasible implementations, before selecting target tasks to be processed from each task to be processed in sequence according to each first state data through a preset task scheduling model based on deep reinforcement learning, and assigning a corresponding target server to the selected target tasks to be processed, the preset task scheduling model based on deep reinforcement learning can also be trained first.

[0069] As an example, the process of training a preset task scheduling model based on deep reinforcement learning may include: initializing parameters of the preset deep reinforcement learning model; obtaining historical state data, historical scheduling action data and historical immediate reward values ​​of any historical task in the historical workflow, wherein the historical state data of the historical task includes the third state data of the historical task executed at the current time step and the fourth state data of the historical task executed at the next time step; inputting the historical scheduling action data and the third state data into the deep reinforcement learning model to predict the historical value evaluation prediction value of the historical scheduling action; inputting the fourth state data and the historical immediate reward value into the deep reinforcement learning model to calculate the historical value evaluation target value of the historical scheduling action; calculating the model loss value based on the historical value evaluation target value and the historical value evaluation prediction value, and iteratively optimizing the deep reinforcement learning model based on the model loss value.

[0070] In some feasible implementations, the preset task scheduling model based on deep reinforcement learning is a deep Q network, which includes a current value network and a target value network; before the process of training the preset task scheduling model based on deep reinforcement learning, it can include: initializing the network parameters θ of the current value network and the network parameters θ' of the target value network, defining the experience replay buffer, the discount factor γ and the learning rate α; then, initializing the state s, and selecting the action a, and selecting the action with the maximum Q value according to the ε-greedy strategy; executing the action, observing the reward r and the next state s', and then storing (s, a, r, s') as a sample in the experience replay buffer; repeating this multiple times to build the experience replay buffer. Furthermore, the process of training the preset task scheduling model based on deep reinforcement learning may include: randomly sampling from the experience replay buffer to obtain a batch of training samples; inputting s and a in the training samples into the current value network, using the Q value function in the current value network to calculate the current Q value Q(s, a;θ), inputting r and s' in the training samples into the target value network, and using the Q value function in the target value network to calculate the target Q value, wherein the target Q value y can be expressed as:

[0071]

[0072] Then, the model loss L is calculated based on the target Q value and the current Q value:

[0073]

[0074] Furthermore, the parameter θ is updated using the gradient descent method based on the model loss L, and θ can be periodically copied to the target value network to update θ'.

[0075] In the above task scheduling method, firstly, a pending workflow is obtained, which includes multiple pending tasks, and then the first state data of each pending task is obtained. Through a preset task scheduling model based on deep reinforcement learning, according to each first state data, a target pending task is selected from each pending task in turn, and a corresponding target server is assigned to the selected target pending task, wherein the preset task scheduling model based on deep reinforcement learning is pre-trained according to the historical state data, historical scheduling action data and historical instant reward value of the historical workflow. The deep reinforcement learning model is used to allocate the execution order and server of each pending task in the pending workflow. The deep neural network has a strong representation ability and can capture the complex patterns and dependencies in the data, so as to find the global optimal solution in a larger search space. Compared with the traditional heuristic algorithm and meta-heuristic algorithm, the deep reinforcement learning model can not only use the known optimal solution, but also improve the generalization ability of the model by capturing the correlation between different tasks and data, and reduce the risk of falling into local extreme values, thereby improving the efficiency and accuracy of workflow task scheduling, and making the workflow task scheduling effect better.

[0076] In an exemplary embodiment, the first state data includes at least one of first task attribute data, first task state data, first task priority data, and first task dependency data; through a preset task scheduling model based on deep reinforcement learning, according to each first state data, a target task to be processed is selected from each task to be processed in turn, and a corresponding target server is assigned to the selected target task to be processed, including steps S311 to S312. Wherein:

[0077] Step S311 , performing vector concatenation on each first state data corresponding to each task to be processed to obtain a first state vector corresponding to each task to be processed.

[0078] It should be noted that in the insurance field, businesses such as reporting to the State Administration of Financial Supervision and Administration and showing reports to leaders are time-limited and can be set with a higher priority. Other tasks such as daily data synchronization, data extraction, data integration and data cleaning can be set with a lower priority. Traditional deep Q networks find it difficult to identify the differences in these task priorities, resulting in the task scheduling results being difficult to accurately match the actual priorities of the business end.

[0079] Exemplarily, after the first state data corresponding to each task to be processed is acquired, vector concatenation may be performed on each first state data corresponding to each task to be processed to obtain a first state vector corresponding to each task to be processed.

[0080] Step S312, through the preset task scheduling model based on deep reinforcement learning, according to the first state vector corresponding to each task to be processed, the target task to be processed is selected from each task to be processed in turn, and the corresponding target server is allocated to the selected target task to be processed.

[0081] Exemplarily, after obtaining the first state vector of each pending task, each first state vector can be input into a pre-trained preset task scheduling model based on deep reinforcement learning, and the preset task scheduling model predicts scheduling actions based on each first state vector, and predicts scheduling actions to be executed sequentially in multiple time steps. For each time step, the preset task scheduling model can predict the scheduling action with the highest estimated value based on the first state vector of the currently remaining pending tasks, determine the pending task indicated in the scheduling action with the highest estimated value as the target pending task, and determine the server indicated in the scheduling action with the highest estimated value as the target server.

[0082] In this embodiment, by determining the task priority data as status data, the task scheduling model can learn to give priority to processing tasks with higher priorities. In this way, as long as the business end provides the task priority data when initiating a task scheduling request, the priority of each task to be processed can be fully considered when scheduling, so that the task scheduling results can more accurately match the actual priority requirements of the business end, thereby improving the task scheduling effect.

[0083] In an exemplary embodiment, the preset task scheduling model is a deep Q network, and the deep Q network includes a current value network; through the preset task scheduling model based on deep reinforcement learning, according to each first state data, the target task to be processed is selected from each task to be processed in turn, and the corresponding target server is assigned to the selected target task to be processed, including steps S321 to S322. Wherein:

[0084] Step S321, selection step: through the current value network, according to each first state data, select the target task to be processed to be executed in the current time step from each task to be processed, and allocate the corresponding target server to the selected target task to be processed.

[0085] Among them, Deep Q-Network (DQN) is a type of deep reinforcement learning model that combines neural networks and Q learning algorithms. The goal of DQN is to learn a function Q(s,a;θ) so that its output is as close to the true Q value as possible, where θ is the parameter of the neural network, s can refer to the state data of the task, a can refer to the scheduling action, and the Q value function Q(s,a) represents the total reward that the agent expects to obtain after taking action a in state s.

[0086] The deep Q network can use two neural networks, one is the current value network, which is used to select the scheduling action and calculate the current Q value, that is, the value assessment prediction value, and the other is the target value network, which is used to calculate the target Q value, that is, the value assessment target value. The parameters of the target value network are obtained by regularly copying the parameters of the current value network, usually during the training process, and the copy is performed once every fixed number of steps or cycles. By comparing the difference between the value assessment prediction value and the value assessment target value, the loss can be calculated, and the model parameters of the current value network can be updated by the gradient descent method. In the model application stage, only the current value network can be deployed to select the scheduling action, or the current value network and the target value network can be deployed at the same time. The current value network can be used to select the scheduling action, and the model optimization and update can continue during the model application process.

[0087] Exemplarily, at each time step, the first state data of the remaining pending tasks in the current time step can be input into the current value network for scheduling action prediction. The preset task scheduling model can predict the scheduling action with the highest estimated value based on the first state data of the currently remaining pending tasks, determine the pending task indicated in the scheduling action with the highest estimated value as the target pending task, and determine the server indicated in the scheduling action with the highest estimated value as the target server.

[0088] Step S322, determine the target to-be-processed task as a processed task, and return to the execution selection step.

[0089] Exemplarily, after determining that the target pending task corresponding to the current time step has been completed, the completed target pending task can be determined as a processed task, so that the target pending task is removed from the set of pending tasks, and then the execution selection step is returned to select the target pending task to be executed in the next time step from the remaining pending tasks, and this cycle is repeated until all pending tasks in the pending workflow are completed, completing the task scheduling process for the pending workflow.

[0090] In this embodiment, DQN can approximate the Q-value function by using a deep neural network, thereby achieving a highly automated decision-making process, and can make decisions in real time, reducing the delay of some urgent tasks with time limits, and improving the efficiency of workflow task scheduling.

[0091] In an exemplary embodiment, the deep Q network also includes a target value network; Figure 2 As shown, through the current value network, according to each first state data, a target task to be processed that is executed at the current time step is selected from each task to be processed, and a corresponding target server is allocated to the selected target task to be processed, including:

[0092] Step S331, through the current value network, according to each first state data, select the target pending task to be executed in the current time step from each pending task, assign the corresponding target server to the selected target pending task, and predict the value assessment prediction value corresponding to the target pending task.

[0093] Exemplarily, at each time step, the first state data of the remaining pending tasks at the current time step can be first input into the current value network for scheduling action prediction. The corresponding scheduling actions can be first generated based on the remaining pending tasks at the current time step, and the scheduling actions correspond to the pending tasks one by one; each first state data and each scheduling action are combined to obtain task scheduling combination data, each task scheduling combination data includes the first state data corresponding to any pending task and the scheduling action corresponding to the pending task; and then the Q value function in the current value network is used to perform estimated value prediction based on each task scheduling combination data, and the highest estimated value is determined as the value evaluation prediction value, the pending task indicated in the scheduling action corresponding to the value evaluation prediction value is determined as the target pending task, and the server indicated in the scheduling action corresponding to the value evaluation prediction value is determined as the target server.

[0094] like Figure 2 As shown, through the current value network, according to each first state data, a target task to be processed that is executed at the current time step is selected from each task to be processed, and a corresponding target server is allocated to the selected target task to be processed, and steps S40-S70 are also included, wherein:

[0095] Step S40, obtaining the second state data of the task to be processed to be executed in the next time step, and predicting the future reward value corresponding to the target task to be processed according to the second state data through the target value network.

[0096] Among them, the future reward value can refer to the reward obtained by the agent in the future time step after the current time step. The future reward value reflects the long-term effect of the agent under the current decision and is the long-term feedback of the agent's behavior.

[0097] Exemplarily, after determining the target pending task and the target server, the target pending task can be assigned to the target server for processing at the current time step, and the second state data of the pending task to be executed at the next time step returned by the target server according to the internal state transition rules after completing the target pending task can be received; or after the current value network predicts the pending task to be executed at the next time step, the state data of the pending task predicted by the current value network to be executed at the next time step can be determined as the second state data. Then, the second state data is input into the target value network, and the target value network uses the Q value function to calculate the future reward value corresponding to the target pending task.

[0098] Step S50, obtaining current environment feedback data corresponding to the target task to be processed, and detecting the immediate reward value corresponding to the target task to be processed according to each current environment feedback data.

[0099] Among them, environmental feedback data may refer to environmental feedback information generated after the target pending task is executed, and may include at least one of action completion result data, actual execution time, actual resource utilization rate and energy consumption data, etc. The immediate reward value may refer to the reward immediately returned by the environment after the agent takes an action in a certain state. The immediate reward value can reflect the immediate effect of the agent under the current decision and is the direct feedback of the agent's behavior.

[0100] Exemplarily, after the target task to be processed is completed, the current environment feedback data corresponding to the target task to be processed can be obtained, and the immediate reward value corresponding to the target task to be processed can be calculated based on the current environment feedback data and a preset immediate reward value algorithm.

[0101] Step S60, aggregating the immediate reward value and the future reward value to obtain the value assessment target value corresponding to the target task to be processed.

[0102] Exemplarily, the immediate reward value and the future reward value are aggregated to obtain the value assessment target value corresponding to the target task to be processed. The aggregation method may include at least one of summing, averaging, weighted summing, etc.

[0103] Step S70, optimizing the current value network according to the value assessment prediction value and the value assessment target value.

[0104] Exemplarily, the model loss of the current value network is calculated based on the difference between the value assessment prediction value and the value assessment target value. When the model is judged to be converged based on the model loss, or when the number of iterative optimization reaches a preset upper limit of iterations, there is no need to optimize and update the model parameters. When the model is judged not to have converged based on the model loss, and the number of iterative optimization does not reach a preset upper limit of iterations, the model parameters of the current value network are updated by the gradient descent method based on the model loss.

[0105] As an example, the model parameters of the current value network may be periodically copied to the target value network to update the target value network.

[0106] In this embodiment, the dual Q network can maintain the stability of updates during the training process, prevent the value assessment target value from fluctuating greatly during the training process, and improve the model training effect.

[0107] In an exemplary embodiment, detecting the immediate reward value corresponding to the target task to be processed according to each current environment feedback data includes steps S51 to S52. Among them:

[0108] Step S51, detecting the task completion reward value, task failure reward value and resource utilization reward value corresponding to the target pending task according to the current environment feedback data;

[0109] Among them, the task completion reward value is used to encourage higher priority tasks to be executed first and completed on time. The task failure reward value can be a negative reward to suppress the situation where the pending tasks cannot be completed on time. The resource utilization reward value is used to encourage more reasonable resource allocation for the pending tasks.

[0110] Exemplarily, information related to task priority and information related to the actual execution time of the task can be extracted from the current environment feedback data, and the task completion reward value and task failure reward value corresponding to the target task to be processed can be calculated based on the information related to task priority, information related to the actual execution time of the task and a preset action completion reward function; information related to resource utilization can also be extracted from the current environment feedback data, and the resource utilization reward value corresponding to the target task to be processed can be calculated based on the information related to resource utilization and a preset resource utilization reward function.

[0111] Step S52, aggregating the task completion reward value, the task failure reward value and the resource utilization reward value to obtain the immediate reward value corresponding to the target task to be processed.

[0112] Exemplarily, the task completion reward value, the task failure reward value, and the resource utilization reward value are aggregated to obtain the immediate reward value corresponding to the target pending task. The aggregation method may include at least one of summing, averaging, and weighted summing.

[0113] In this embodiment, the reward function defines the reward of each action in the current state. Through the reward signal, the model can learn which actions are more conducive to obtaining the maximum reward. By setting multiple reward functions to calculate multiple reward values ​​and then aggregating them to obtain the immediate reward value, it is possible to effectively balance multiple requirements such as task execution time, priority requirements, and resource utilization in workflow scheduling. Through comprehensive evaluation, a task scheduling strategy with shorter task execution time, higher priority tasks are executed first, and the most reasonable resource utilization can be obtained.

[0114] In an exemplary embodiment, the first state data includes at least one of the estimated execution time and the task priority; the current environment feedback data includes at least one of the action completion result data, the actual execution time and the actual resource utilization rate; according to the current environment feedback data, detecting the task completion reward value, the task failure reward value and the resource utilization reward value corresponding to the target pending task includes at least one of steps S511 to S513. Wherein:

[0115] Step S511, when it is determined that the target task to be processed has failed to be completed based on the estimated execution time, the action completion result data corresponding to the target task to be processed, and the actual execution time, the task failure reward value corresponding to the target task to be processed is set to a preset negative reward value.

[0116] Among them, the reward function can be used to evaluate the scheduling decisions of the model and guide it to learn the optimal strategy. For example, when a task is successfully completed, a positive reward can be given; for the timely completion of high-priority tasks, a higher positive reward can be given to ensure that critical tasks are given priority; if a task is not completed on time or fails, a negative reward can be given to punish the model for bad decisions.

[0117] The action completion result data is used to indicate whether the target pending task is successfully executed.

[0118] Exemplarily, the action completion result data and the actual execution time corresponding to the target pending task can be extracted from the current environment feedback data, and the estimated execution time corresponding to the target pending task can also be extracted from the first state data; further, when the action completion result data indicates that the target pending task has failed to execute, or when the actual execution time is longer than the estimated execution time, or when the action completion result data indicates that the target pending task has failed to execute and the actual execution time is longer than the estimated execution time, the target pending task is determined to have failed to complete, and the task failure reward value corresponding to the target pending task is set to a preset negative reward value. Among them, the preset negative reward value can be determined according to the actual situation and test results, etc., for example, it can be -20, etc., and this embodiment does not limit this.

[0119] Step S512, when it is determined that the target task to be processed is completed successfully based on the estimated execution time, the action completion result data corresponding to the target task to be processed and the actual execution time, calculate the task completion reward value corresponding to the target task to be processed according to the task priority, the estimated execution time and the actual execution time, wherein the task completion reward value is positively correlated with the task priority, and the task completion reward value is negatively correlated with the actual execution time.

[0120] Exemplarily, the action completion result data and actual execution time corresponding to the target task to be processed can be extracted from the current environment feedback data, and the estimated execution time corresponding to the target task to be processed can also be extracted from the first state data; further, when the action completion result data indicates that the target task to be processed is executed successfully, and the actual execution time is not greater than the estimated execution time, the target task to be processed is determined to be completed successfully, and the task priority corresponding to the target task to be processed can be extracted from the first state data, and the task priority, estimated execution time and actual execution time are input into a preset action completion reward function to calculate the task completion reward value corresponding to the target task to be processed, wherein the preset action completion reward function is used to characterize that the task completion reward value is positively correlated with the task priority, and the task completion reward value is negatively correlated with the actual execution time.

[0121] In one feasible implementation, the preset action completion reward function can be expressed as:

[0122]

[0123] Among them, R complete can be the task completion reward value; α can be the preset first weight value; P is the task priority, T expected is the estimated execution time, T actual The actual execution time.

[0124] Step S513 , detecting a resource utilization difference value between the actual resource utilization rate and the preset target resource utilization rate, and calculating a resource utilization reward value based on the resource utilization difference value, wherein the resource utilization reward value is negatively correlated with the resource utilization difference value.

[0125] Exemplarily, the actual resource utilization rate corresponding to the target pending task can be extracted from the current environmental feedback data, and then the actual resource utilization rate is input into the preset resource utilization reward function to calculate the resource utilization reward value corresponding to the target pending task, wherein the preset action completion reward function is used to characterize the negative correlation between the resource utilization reward value and the resource utilization difference value. The preset target resource utilization rate can be determined based on experience to be a value that can ensure that resources are fully utilized and the number of resource alarms is within an acceptable range, such as 0.6, etc., which can be determined specifically based on actual conditions and test results, etc., and this embodiment does not limit this.

[0126] In one feasible implementation, the preset action completion reward function can be expressed as:

[0127]

[0128] Among them, R u β can be a resource utilization reward value; β can be a preset second weight value; U is the actual resource utilization rate, Utarget is the preset target resource utilization rate, and 0.9 can be the upper limit of the resource utilization rate. The preset first weight value and the preset second weight value can be determined according to actual conditions and test results, and the sum of the two can be equal to 1.

[0129] In some feasible implementations, the immediate reward value reward can be expressed as:

[0130]

[0131] Among them, R complete It can be used to reward the task completion, R u It can be the resource utilization reward value, R fail Can reward value for mission failure.

[0132] In this embodiment, by setting the task failure reward value, the model can be guided to learn a scheduling strategy that completes the pending tasks on time. By setting the task completion reward value, the model can be guided to learn a scheduling strategy that completes the pending tasks on time and in order of priority. By setting the resource utilization reward value, the model can be guided to learn a scheduling strategy that makes resource utilization more reasonable. This can effectively balance multiple requirements such as task execution time, priority requirements, and resource utilization in workflow scheduling. Through comprehensive evaluation, a task scheduling strategy with shorter task execution time, higher priority tasks are executed first, and the most reasonable resource utilization can be obtained.

[0133] In some feasible implementations, the task scheduling method, random algorithm, earliest completion time algorithm and polling algorithm provided in this embodiment are tested for the instruction response time ratio. The test results are as follows: Figure 3 As shown, the instruction response time ratio refers to the instruction set length of the task (in millions of instructions) / the execution time of the task. The execution time of task i = the time for task i to be transferred from the scheduler to the virtual machine + the waiting time of task i in the virtual machine + the execution time of task i in the virtual machine. The higher the instruction response time ratio, the more efficient the optimized algorithm is in executing workflow tasks and can significantly improve the completion time of tasks. IQR refers to the interquartile range. Figure 5 It can be seen that the median of the instruction response time ratio of the random algorithm is 146.2, the median of the instruction response time ratio of the earliest completion time algorithm is 148.7, the median of the instruction response time ratio of the polling algorithm is 149.0, and the median of the instruction response time ratio of this embodiment is 156.9, which is significantly higher than other algorithms. This shows that the optimized algorithm is more efficient in executing workflow tasks and can significantly improve the completion time of the task.

[0134] In some feasible implementations, the changes in the predicted value of the task scheduling method provided in this embodiment during the training process with the number of training times are recorded, such as Figure 4 As shown, the horizontal axis is the number of iterations, and the vertical axis is the value assessment prediction value. After 3000 times, the value assessment prediction value has tended to fluctuate stably. It can be seen that the value assessment prediction value is stable between 5-6.5.

[0135] In some feasible implementations, after the task scheduling method provided by this embodiment is used to schedule tasks, the number of tasks executed on each server is recorded, such as Figure 5 As shown, this embodiment can effectively allocate tasks to different servers and effectively utilize hardware resources. By introducing resource utilization rewards in the reward function, the model not only pays attention to the completion time and priority of the task during the learning process, but also takes into account the efficient utilization of resources, thereby overcoming the technical defects of unbalanced resource allocation in traditional scheduling algorithms and avoiding excessive consumption of resources or long-term idleness.

[0136] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0137] Based on the same inventive concept, the embodiment of the present application also provides a task scheduling device for implementing the task scheduling method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more task scheduling device embodiments provided below can refer to the limitations on the task scheduling method above, and will not be repeated here.

[0138] In an exemplary embodiment, Figure 6 As shown, a task scheduling device is provided, including: a first acquisition module 602, a second acquisition module 604 and a prediction module 606, wherein:

[0139] A first acquisition module 602 is used to acquire a workflow to be processed, where the workflow to be processed includes a plurality of tasks to be processed;

[0140] The second acquisition module 604 is used to acquire the first status data of each task to be processed;

[0141] The prediction module 606 is used to select the target pending task from each pending task in turn according to each first state data through a preset task scheduling model based on deep reinforcement learning, and assign a corresponding target server to the selected target pending task, wherein the preset task scheduling model based on deep reinforcement learning is pre-trained according to the historical state data, historical scheduling action data and historical immediate reward values ​​of the historical workflow.

[0142] In an exemplary embodiment, the first state data includes at least one of first task attribute data, first task state data, first task priority data, and first task dependency data; the prediction module 606 is further configured to:

[0143] Performing vector concatenation on each first state data corresponding to each task to be processed to obtain a first state vector corresponding to each task to be processed;

[0144] Through a preset task scheduling model based on deep reinforcement learning, target tasks to be processed are selected from each task to be processed in turn according to the first state vector corresponding to each task to be processed, and the corresponding target server is allocated to the selected target task to be processed.

[0145] In an exemplary embodiment, the preset task scheduling model is a deep Q network, and the deep Q network includes a current value network; the prediction module 606 is also used to:

[0146] Selection step: selecting a target task to be processed that is executed at the current time step from among the tasks to be processed through the current value network according to each first state data, and assigning a corresponding target server to the selected target task to be processed;

[0147] The target pending task is determined as a processed task, and the execution of the selection step is returned.

[0148] In an exemplary embodiment, the deep Q network further includes a target value network; the prediction module 606 is further used to:

[0149] Through the current value network, according to each first state data, a target task to be processed that is executed at the current time step is selected from each task to be processed, a corresponding target server is assigned to the selected target task to be processed, and a value assessment prediction value corresponding to the target task to be processed is predicted;

[0150] The task scheduling device also includes an optimization module, which is used to:

[0151] Obtain the second state data of the task to be processed to be executed in the next time step, and predict the future reward value corresponding to the target task to be processed according to the second state data through the target value network;

[0152] Obtain the current environment feedback data corresponding to the target pending task, and detect the immediate reward value corresponding to the target pending task according to each current environment feedback data;

[0153] Aggregate the immediate reward value and the future reward value to obtain the value assessment target value corresponding to the target pending task;

[0154] According to the value assessment prediction value and the value assessment target value, the current value network is optimized.

[0155] In an exemplary embodiment, the optimization module is further configured to:

[0156] According to the current environment feedback data, detect the task completion reward value, task failure reward value and resource utilization reward value corresponding to the target pending task;

[0157] Aggregate the task completion reward value, task failure reward value and resource utilization reward value to obtain the immediate reward value corresponding to the target pending task.

[0158] In an exemplary embodiment, the first state data includes at least one of an estimated execution time and a task priority; the current environment feedback data includes at least one of action completion result data, actual execution time and actual resource utilization; and the optimization module is further configured to perform at least one of the following operations:

[0159] If it is determined that the target task to be processed has failed to be completed based on the estimated execution time, the action completion result data corresponding to the target task to be processed, and the actual execution time, the task failure reward value corresponding to the target task to be processed is set to a preset negative reward value;

[0160] When the target pending task is determined to be successfully completed based on the estimated execution time, the action completion result data corresponding to the target pending task and the actual execution time, the task completion reward value corresponding to the target pending task is calculated according to the task priority, the estimated execution time and the actual execution time, wherein the task completion reward value is positively correlated with the task priority, and the task completion reward value is negatively correlated with the actual execution time;

[0161] A resource utilization difference value between an actual resource utilization rate and a preset target resource utilization rate is detected, and a resource utilization reward value is calculated based on the resource utilization difference value, wherein the resource utilization reward value is negatively correlated with the resource utilization difference value.

[0162] Each module in the above task scheduling device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0163] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 7 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, a task scheduling method is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0164] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0165] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0166] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0167] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0168] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0169] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0170] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0171] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A task scheduling method, characterized in that: The method comprises: Acquire a workflow to be processed, wherein the workflow to be processed includes a plurality of tasks to be processed; Acquire the first status data of each of the tasks to be processed; Through a preset task scheduling model based on deep reinforcement learning, according to each of the first state data, a target task to be processed is selected in turn from each of the tasks to be processed, and a corresponding target server is allocated to the selected target task to be processed, wherein the preset task scheduling model based on deep reinforcement learning is pre-trained based on the historical state data, historical scheduling action data and historical instant reward values ​​of the historical workflow.

2. The method according to claim 1, characterized in that The first state data includes at least one of first task attribute data, first task state data, first task priority data and first task dependency data; the preset task scheduling model based on deep reinforcement learning, according to each of the first state data, sequentially selecting a target task to be processed from each of the tasks to be processed, and allocating a corresponding target server to the selected target task to be processed, includes: Performing vector concatenation on the first state data corresponding to each task to be processed to obtain a first state vector corresponding to each task to be processed; Through a preset task scheduling model based on deep reinforcement learning, target tasks to be processed are selected from each of the tasks to be processed in turn according to the first state vector corresponding to each of the tasks to be processed, and a corresponding target server is allocated to the selected target tasks to be processed.

3. The method according to claim 1, characterized in that The preset task scheduling model is a deep Q network, and the deep Q network includes a current value network; the preset task scheduling model based on deep reinforcement learning selects a target task to be processed from each of the tasks to be processed in sequence according to each of the first state data, and allocates a corresponding target server to the selected target task to be processed, including: Selection step: selecting a target task to be processed to be executed at the current time step from the tasks to be processed according to the first state data through the current value network, and assigning a corresponding target server to the selected target task to be processed; The target to-be-processed task is determined as a processed task, and the process returns to execute the selection step.

4. The method according to claim 3, characterized in that The deep Q network also includes a target value network; the current value network is used to select a target task to be processed that is executed at the current time step from each of the tasks to be processed according to each of the first state data, and a corresponding target server is allocated to the selected target task to be processed, including: Through the current value network, according to each of the first state data, a target task to be processed that is executed at the current time step is selected from each of the tasks to be processed, a corresponding target server is allocated to the selected target task to be processed, and a value assessment prediction value corresponding to the target task to be processed is predicted; After selecting a target task to be processed to be executed at the current time step from the tasks to be processed according to the first state data through the current value network and allocating a corresponding target server to the selected target task to be processed, the method further includes: Acquire second state data of the task to be processed to be executed at the next time step, and predict the future reward value corresponding to the target task to be processed according to the second state data through the target value network; Acquire current environment feedback data corresponding to the target task to be processed, and detect the immediate reward value corresponding to the target task to be processed according to each current environment feedback data; Aggregating the immediate reward value and the future reward value to obtain a value assessment target value corresponding to the target task to be processed; The current value network is optimized according to the value assessment prediction value and the value assessment target value.

5. The method according to claim 4, characterized in that The detecting the instant reward value corresponding to the target task to be processed according to each of the current environment feedback data includes: According to the current environment feedback data, detecting the task completion reward value, task failure reward value and resource utilization reward value corresponding to the target pending task; Aggregate the task completion reward value, the task failure reward value and the resource utilization reward value to obtain an immediate reward value corresponding to the target task to be processed.

6. The method according to claim 5, characterized in that The first state data includes at least one of an estimated execution time and a task priority; the current environment feedback data includes at least one of action completion result data, actual execution time and actual resource utilization rate; the detecting, based on the current environment feedback data, a task completion reward value, a task failure reward value and a resource utilization reward value corresponding to the target pending task includes at least one of the following: If it is determined that the target task to be processed fails to be completed based on the estimated execution time, the action completion result data corresponding to the target task to be processed, and the actual execution time, the task failure reward value corresponding to the target task to be processed is set to a preset negative reward value; In the case where it is determined that the target task to be processed is successfully completed based on the estimated execution time, the action completion result data corresponding to the target task to be processed, and the actual execution time, a task completion reward value corresponding to the target task to be processed is calculated according to the task priority, the estimated execution time, and the actual execution time, wherein the task completion reward value is positively correlated with the task priority, and the task completion reward value is negatively correlated with the actual execution time; A resource utilization difference value between the actual resource utilization rate and a preset target resource utilization rate is detected, and a resource utilization reward value is calculated based on the resource utilization difference value, wherein the resource utilization reward value is negatively correlated with the resource utilization difference value.

7. A task scheduling device, characterized in that: The device comprises: A first acquisition module is used to acquire a workflow to be processed, wherein the workflow to be processed includes a plurality of tasks to be processed; A second acquisition module is used to acquire the first status data of each of the tasks to be processed; A prediction module is used to select a target task to be processed from each of the tasks to be processed in turn according to each of the first state data through a preset task scheduling model based on deep reinforcement learning, and assign a corresponding target server to the selected target task to be processed, wherein the preset task scheduling model based on deep reinforcement learning is pre-trained based on historical state data, historical scheduling action data and historical immediate reward values ​​of historical workflows.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images