Large-scale scientific calculation workflow scheduling method based on deep reinforcement learning

Abstract

The invention relates to the field of cloud computing and deep reinforcement learning, and discloses a large-scale scientific computing workflow scheduling method based on deep reinforcement learning. According to the method, the graph attention network is introduced to extract workflow structure features, and a deep reinforcement learning sampling strategy based on a genetic algorithm is designed to generate an initial sample trajectory for a deep reinforcement learning agent, so that the training process is accelerated. Experiments show that a faster training speed can be obtained in a simulated cloud environment, the workflow is executed on the cloud computing platform according to the generated large-scale scientific computing workflow scheduling strategy, and better completion time, energy consumption and reliability can be reflected.

Description

Technical Field

[0001] The present invention relates to the fields of cloud computing and deep reinforcement learning. Background Art

[0002] In the field of scientific computing, a workflow refers to a set of interrelated tasks that need to be executed in a specific order. Workflow scheduling refers to the allocation of appropriate computing resources (such as virtual machines) to tasks in a workflow to meet specific goals or service quality. In recent years, the demand for large-scale scientific computing has increased in various fields, and cloud computing technology has greatly facilitated users to obtain high-performance computing resources. In this context, the scheduling of large-scale scientific workflows on cloud computing platforms has become an important research topic.

[0003] Traditional meta-heuristic algorithms face the challenge of low search efficiency. Therefore, with the development of artificial neural networks and reinforcement learning technology in recent years, some studies have explored the use of deep reinforcement learning to solve workflow scheduling problems.

[0004] Existing technology and its analysis:

[0005] Yang et al. (Yang Z, Nguyen P, Jin H, Nahrstedt K. MIRAS: Model-based reinforcement learning for microservice resource allocation over scientific workflows. 2019 IEEE 39th international conference on distributed computing systems (ICDCS): IEEE; 2019. p. 122-32.) first proposed model-based reinforcement learning to schedule scientific workflows in distributed cloud systems. Yu et al. (Yu X, Wu W, Wang Y. Integrating cognition cost with reliability QoS for dynamic workflow scheduling using reinforcement learning. IEEE Transactions on Services Computing. 2023; 16(4): 2713-26.) proposed a model based on deep reinforcement learning that optimized multiple quality of service objectives. However, the above studies only used explicit raw information of tasks and workflows as state information of deep reinforcement learning models. For large-scale scientific workflows, it is difficult to extract their complex structural features, resulting in poor results in scheduling large-scale scientific workflows and slow training speed.

[0006] Mao et al. (Mao H, Schwarzkopf M, Venkatakrishnan SB, Meng Z, Alizadeh M. Learning scheduling algorithms for data processing clusters. Proceedings of the ACM special interest group on data communication. 2019. p. 270-88.) first proposed to introduce graph neural network into reinforcement learning model to extract global features of workflow, and then the deep reinforcement learning model makes further decisions based on the global features of workflow. However, since this model relies on relatively simple message passing neural network, the scale of workflows it can handle is still limited, and it also faces the problem of low training sampling efficiency on large-scale workflows.

[0007] Zhou et al. (Zhou Y, Li X, Luo J, Yuan M, Zeng J, Yao J. Learning to optimize DAG scheduling in heterogeneous environment. 2022 23rd IEEE International Conference on Mobile Data Management (MDM): IEEE; 2022. p. 137-46.) and Song et al. (Song Y, Li C, Tian L, Song HA reinforcement learning based job scheduling algorithm for heterogeneous computing environment. Computers and Electrical Engineering. 2023; 107: 108653.) used basic message passing neural networks and graph convolutional networks to extract features of workflows and obtain features of task nodes and the entire environment, respectively. However, these methods are not very effective in large-scale scientific workflow scheduling tasks. In addition, the sampling efficiency of existing deep reinforcement learning scheduling methods limits their training speed on large-scale workflows.

[0008] In summary, previous methods pay less attention to complex workflow structures and deep reinforcement learning sampling efficiency, resulting in poor training and performance of large-scale scientific workflow scheduling models, especially when there are multiple quality of service requirements. Summary of the invention

[0009] In view of the problem that the existing deep reinforcement learning technology is difficult to extract the structural information of the workflow when scheduling large-scale scientific computing workflows, and the sampling efficiency of reinforcement learning training is low, the present invention proposes a large-scale scientific computing workflow scheduling method based on deep reinforcement learning. The present invention introduces a graph attention network to extract the structural features of the relationship between each task node of the workflow for the deep reinforcement learning scheduling model. Using the feature representation and structural information obtained by the graph attention network, the deep reinforcement learning agent can show better strategies. Secondly, a deep reinforcement learning sampling strategy based on a genetic algorithm is designed, and a suboptimal scheduling scheme is obtained using the genetic algorithm, which avoids the reinforcement learning agent from performing too much useless exploration in large-scale workflows, thereby improving the training efficiency. The present invention improves the training speed of the deep reinforcement learning scheduling model by effectively extracting and efficiently sampling large-scale workflow features, and the generated scheduling strategy can better meet the service quality indicators including completion time, energy consumption and reliability, thereby providing benefits to users of large-scale scientific computing.

[0010] The technical solution of the present invention is:

[0011] A large-scale scientific computing workflow scheduling method based on deep reinforcement learning with improved structural feature extraction and sampling, including:

[0012] Step 1: Get the task nodes t in the scientific computing workflow i The amount of calculation r i , data volume i , and the sequential dependency relationship between the task nodes of the workflow represented by a directed acyclic graph, the bandwidth B obtained from the cloud computing platform, and the virtual machines VM k The computing power of k , failure rate f k , Energy consumption factor index u k , and based on the information obtained, establish a mathematical optimization model for large-scale scientific computing workflow scheduling with the goal of optimizing service quality;

[0013] Step 2: The scheduling mathematical optimization model established in step 1 with the goal of optimizing service quality is further modeled using the Markov decision process, and the graph attention network is used as the feature extraction layer to extract the complex structural features of large-scale scientific computing workflows as the state input of the subsequent deep reinforcement learning scheduling strategy network; the graph attention network feature extraction layer, the deep reinforcement learning scheduling strategy network, and the deep reinforcement learning critic network together constitute the deep reinforcement learning scheduling agent;

[0014] Step 3, using a genetic algorithm to generate several groups of suboptimal scheduling solutions for the scheduling mathematical optimization model established in step 1 with the goal of optimizing service quality, and converting these scheduling solutions into a series of sampling trajectories consisting of state-action pairs;

[0015] Step 4: Use the sampled trajectories obtained by the genetic algorithm in step 3 to train the deep reinforcement learning scheduling agent, and after several training iterations, place the deep reinforcement learning scheduling agent into a simulation environment for further training, and output a scheduling strategy with the best service quality based on the execution order of each task node.

[0016] Furthermore, in step 1, the order dependency relationship between the task nodes in the workflow in step 1 is reflected as follows: for a task node t i , only when all its predecessor task nodes are completed and data transmission is completed, can it start execution, that is, t i The earliest start time EST(t i ) satisfies the formula:

[0017]

[0018] Where Pred(t i ) is the task node t i The set of all predecessor nodes of , FT(t j ) is the task node t j The end time, CT(t i ) is the task node t i The time required for data transmission, the two meet the following requirements:

[0019] FT(t i )=ST(t i )+ET(t i ,Alc(t i ))

[0020]

[0021] Where Alc(t i ) refers to the task node t i The virtual machine assigned; task node t i Assigned to the virtual machine VM k The calculation execution time ET(t i ,VM k )satisfy:

[0022]

[0023] Furthermore, in step 1, the mathematical optimization model for scheduling large-scale scientific computing workflows described in step 1 aims to optimize the service quality and includes three specific optimization goals, namely, minimizing the workflow completion time MP, minimizing the energy consumption EC required for workflow completion, and maximizing the reliability RB of workflow execution, namely:

[0024] minimize MP,

[0025] minimize EC,

[0026] maximize RB,

[0027] The workflow completion time MP is equivalent to the completion time FT(t i ), that is:

[0028]

[0029] The energy consumption EC required to complete the workflow is the computational energy consumption C(t i ,VM k ), that is:

[0030]

[0031] Where Alc(t i ) refers to the task node t i The virtual machine assigned, assuming that the task node t i Assigned to the virtual machine VM k , then its calculation energy consumption C(t i ,VM k ) and its execution time ET(t i ,VM k ), virtual machine computing power c k and energy consumption factor u k Relevant, namely:

[0032] C(t i ,VM k )=ET(t i ,VM k )×c k u k

[0033] The reliability of workflow execution RB is the probability that all task nodes will execute without failure. It is the product of the reliability of each task node, that is:

[0034]

[0035] Where Alc(t i ) refers to the task node ti The virtual machine assigned, assuming that the task node t i Assigned to the virtual machine VM k , then its reliability D(t i ,VM k ) follows a Poisson distribution, and its execution time ET(t i ,VM k ), virtual machine failure rate f k Related, e is the base of natural logarithms:

[0036]

[0037] Furthermore, in step 2, the Markov decision process model is established, and each step consists of a Markov quaternion M τ ={s τ ,a τ ,P τ ,R τ}, representing the state, action, transition probability and reward at the time τ of training respectively;

[0038] Status τ It is the status of each virtual machine on the current cloud computing platform and the execution status of the current workflow. For the status of the workflow, it is necessary to use the graph attention network to process the directed acyclic graph corresponding to the workflow to extract the topological structure information of the workflow;

[0039] Action a τ It is stipulated that a task node is selected from all currently schedulable task nodes, and a task node t i To be considered schedulable, the following conditions must be met: First, t i All predecessor task nodes of t should have been scheduled, and then t i At least one of the predecessor task nodes has started executing;

[0040] The transition probability P needs to be learned;

[0041] Reward R τ Defined by the following formula, where C1, C2, C3 are weight coefficients, RB τ and EC τ are the overall reliability and energy consumption of all scheduled tasks at time τ; EFT τ is the estimated completion time of all scheduled tasks at time τ:

[0042]

[0043] Furthermore, for the state s τ :

[0044] The information of each virtual machine in the cloud computing environment and the task node t i The information and status of task nodes t i The amount of calculation r i , data volume i , and the sequential dependency relationship between the task nodes of the workflow represented by a directed acyclic graph, the bandwidth B obtained from the cloud computing platform, and the virtual machines VM k The computing power of k , failure rate f k , Energy consumption factor index u k 、Whether the node has been scheduled (t i ), as t i The original feature vector v i ={τ,r i ,d i ,B,c k ,f k ,u k ,Scheduled(t i )}, and then use the graph attention network to process the directed acyclic graph corresponding to the workflow and extract the feature vector h containing the workflow topology information L (t i ), and in each layer of the graph attention network, t i The eigenvector h l (t i ) is calculated as follows:

[0045] h 0 (t i )=v i ,

[0046]

[0047] In the above formula, L is the total number of layers in the graph attention network, h l-1 (t i ) and h l-1 (t j ) is the feature vector of the previous layer, N(t i ) indicates that i The set of all adjacent task nodes, σ is the activation function, W is the learnable weight matrix, α i,j is the task node t i and t j The attention coefficient between

[0048] The global feature H(G) of the workflow is obtained by pooling the features of the Lth layer of all task nodes.

[0049] Furthermore, the action a τCalculation method:

[0050] Based on the feature vectors of all currently schedulable task nodes obtained by the graph attention network, a deep reinforcement learning scheduling strategy network built by a multi-layer perceptron MLP1 and a softmax module is used to calculate the probability of selecting each schedulable task node in the current situation. i Currently schedulable, then choose t i The probability is:

[0051]

[0052] Finally, according to the probability p i Calculate the task node that should be selected currently, where r is a random number satisfying r∈[0,1):

[0053]

[0054] Furthermore, in step 3, a genetic algorithm is used to generate several groups of suboptimal scheduling solutions for the scheduling problem. The scheduling solutions (solution individuals) provided by the genetic algorithm are sequences composed of task nodes arranged in a scheduling order. The genetic algorithm includes the following steps:

[0055] Step 41: define the population selection mechanism and the maximum number of iterations e of the genetic algorithm;

[0056] Step 42: Define the population encoding method of the genetic algorithm and generate a random initial population pop;

[0057] Step 43: For each scheduling scheme in the population pop, calculate its fitness function;

[0058] Step 44: for pop, generate a new population pop′ according to the mutation and crossover mechanism;

[0059] Step 45: Select the best feasible solution in pop′;

[0060] Step 46: recalculate the adaptation function of each scheduling scheme in pop′ and replace pop with pop′;

[0061] Step 47: Repeat steps 44 to 46 until the number of iterations reaches e;

[0062] Step 48: Convert each pop scheduling scheme into a state-action pair {s τ ,a τ ,} trajectory sequence, as the output of the algorithm.

[0063] Further, in step 44, the mutation mechanism of the genetic algorithm comprises the following steps:

[0064] Step 441: randomly select a scheduling scheme I in pop and randomly select a mutation point MutPoint on I;

[0065] Step 442: For the task node t corresponding to MutPoint on I, start from MutPoint and keep moving forward on I until the first predecessor node of t is encountered, or the node moves to the beginning of I, and the position moved to is recorded as start;

[0066] Step 443: For the task node t corresponding to MutPoint on I, start from MutPoint and keep moving backward on I until the first successor node of t is encountered, or the node moves to the end of I, and the position moved to is recorded as end;

[0067] Step 444: For the scheduling scheme I, randomly select a position between start and end, exchange the position and the task node on MutPoint, obtain the scheduling scheme I' and output it.

[0068] Further, in step 44, the crossover mechanism of the genetic algorithm comprises the following steps:

[0069] Step 441′: Randomly select two scheduling schemes I in pop a and I b ;

[0070] Step 442 ′: randomly select a cross point CxPoint;

[0071] Step 443′: Intercept I b [0:CxPoint] as I a’ , intercept I a [0:CxPoint] as I b’ ;

[0072] Step 444′: From I a Starting from the CxPoint task node until I a At the end of I, for each task node t, if t is not in I a’ If t is within I a’ Behind;

[0073] Step 445′: From I b Starting from the CxPoint task node until I b At the end of I, for each task node t, if t is not in I b’ If t is within I b’ Behind;

[0074] Step 446′: Output the crossover scheduling scheme I a’ ,Ib’ .

[0075] Furthermore, in step 4, the basic framework for training the deep reinforcement learning scheduling agent is the PPO algorithm, which is characterized by including two output network modules, namely the deep reinforcement learning scheduling strategy network and the deep reinforcement learning critic network, wherein the critic network is only used in the training phase; the two networks are optimized according to two loss functions, namely the strategy loss L A τ and the critic loss L C τ ;

[0076] Strategy loss L A τ Used to optimize the policy network, calculated by the following formula:

[0077]

[0078] where π θ (a τ |s τ ) is the cloud computing platform and workflow in state s τ Next, select action a τ The probability of the corresponding task node, ∈, is a truncation parameter, generally set to 0.1 to 0.3; τ is the advantage function. Under the premise of a given discount coefficient γ, A τ By the value function V(s τ’+τ ), Reward R τ’+τ Summing gives:

[0079]

[0080] Critic Loss L C τ Used to optimize the critic network's estimate of the value function, it is calculated using the mean square error loss within τ' steps, that is, the following formula:

[0081]

[0082] Where E represents the mathematical expectation, γ is the discount factor mentioned above, and R τ’+τ is the reward mentioned above; and V(s τ ) is the critic network MLP2 estimates the current global feature H(G τ ) is calculated by the following formula:

[0083] V(s τ )=MLP2(H(G τ )),

[0084] By adopting the above technical solution, the beneficial effects of the present invention are:

[0085] By introducing the graph attention network, the structural features of the relationship between the task nodes of the workflow are extracted for the deep reinforcement learning scheduling agent, so that the deep reinforcement learning scheduling agent can adopt better strategies;

[0086] By designing a deep reinforcement learning sampling strategy based on a genetic algorithm, a suboptimal scheduling solution is obtained, and then a sampling trajectory with useful reward information is obtained, which avoids excessive useless exploration of the deep reinforcement learning scheduling agent during the training process, thereby improving the training efficiency;

[0087] The generated scheduling policies can better generate scheduling solutions that meet the quality of service indicators, including shorter workflow computation time, less energy consumption, and higher reliability, thus providing benefits to users when using large-scale scientific computing workflows. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] Figure 1 It is an application example of the large-scale scientific computing workflow scheduling method based on deep reinforcement learning of the present invention;

[0089] Figure 2 It is the main flow chart of the large-scale scientific computing workflow scheduling method based on deep reinforcement learning of the present invention;

[0090] Figure 3 It is a training flowchart for large-scale scientific computing workflow scheduling based on deep reinforcement learning with improved structural feature extraction and sampling;

[0091] Figure 4 This is an example diagram of the mutual conversion between the deep reinforcement learning scheduling strategy and the scheduling scheme obtained by the genetic algorithm in the present invention;

[0092] Figure 5 It is a schematic diagram of the training and network structure of the deep reinforcement learning scheduling agent of the present invention;

[0093] Figure 6 It is a comparison chart of the reward fluctuation curve in the scheduling model training process of the present invention and the experimental results of the prior art. DETAILED DESCRIPTION

[0094] The present invention proposes a large-scale scientific computing workflow scheduling method based on deep reinforcement learning. The application example of this method is as follows: Figure 1As shown in the figure: In the cloud computing environment, users upload scientific computation application jobs in the form of large-scale scientific workflows. The scheduling model is responsible for determining the execution order of tasks in the workflow and assigning them to appropriate virtual machines in cloud computing resources. Finally, the computation results are fed back to the user.

[0095] exist Figure 1 In the scheduling model, the DRL scheduling agent receives information about large-scale scientific computing workflows as state input. After preprocessing, the workflow information is first extracted by a graph attention network (GAT) as a feature extraction layer (Structural feature extraction with GAT), and then input into the policy network (ActorNetwork) of deep reinforcement learning scheduling (while the critic network of deep reinforcement learning is only used for training). The agent network will select the task node that should be scheduled at present as the scheduling decision (Scheduling Decision). The allocation of virtual machines is based on the greedy rule, which finds the virtual machine that makes the task node selected by the current action complete the earliest, and adds it to the task queue of the virtual machine.

[0096] The technical solution of the present invention is further described below in conjunction with the accompanying drawings. The described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments.

[0097] The overall process of the technical solution of the present invention is as follows Figure 2 As shown, including:

[0098] First, obtain the information of each task node and cloud computing platform in the scientific computing workflow (including each task node t i The amount of calculation r i , data volume i , and the sequential dependency relationship between the task nodes of the workflow represented by a directed acyclic graph, the bandwidth B obtained from the cloud computing platform, and the virtual machines VM k The computing power of k , failure rate f k , Energy consumption factor index u k), and based on the information obtained, establish a mathematical optimization model for large-scale scientific computing workflow scheduling with the goal of optimizing service quality;

[0099] Next, the above mathematical optimization model is further modeled using the Markov decision process, and a graph attention network is used to extract the complex structural features of large-scale scientific computing workflows as the state input of the subsequent deep reinforcement learning scheduling model agent;

[0100] Then, for the above mathematical optimization model, a genetic algorithm is used to generate several groups of suboptimal scheduling solutions for the scheduling problem, and these scheduling solutions are transformed into a series of sampling trajectories consisting of state-action pairs;

[0101] Finally, the deep reinforcement learning scheduling agent is trained using the sampling trajectories obtained in the genetic algorithm, and after several training iterations, the deep reinforcement learning scheduling agent is placed in a simulation environment for further training, so that the deep reinforcement learning scheduling agent can output a scheduling strategy with the best service quality based on the execution order of each task node.

[0102] The implementation described is the present invention scheduling Pegasus workflows on a cloud platform. Pegasus is a workflow dataset that contains more than 100 scientific computing workflows with more than 1,000 task nodes.

[0103] The training process of this embodiment is as follows Figure 3 The flowchart shown:

[0104] Step S1: Initialize the genetic algorithm population pop and press Figure 5 Initialize the network of the deep reinforcement learning scheduling agent;

[0105] Step S2: Run the genetic algorithm on the population pop for several iterations to obtain the population pop';

[0106] Step S3: Convert the scheduling scheme in pop' into a state-action pair {s τ ,a τ ,} trajectory sequence and added to the experience pool of the deep reinforcement learning scheduling model, the scheduling scheme generated by the genetic algorithm and the scheduling strategy generated by the deep reinforcement learning scheduling model a τ The conversion relationship is as follows Figure 4 As shown in Figure 1, the scheduling solution generated by the genetic algorithm is a sequence of task node indexes arranged in order of execution, which is also a topological sorting of the workflow. At each decision point, the first schedulable task in the sequence is selected for scheduling.

[0107] Step S4: For each state-action pair {s τ ,a τ,}, and calculate its corresponding reward R based on the following formula:

[0108]

[0109] Deep reinforcement learning scheduling agent network and training Figure 5 As shown in the figure, the state of the cloud computing environment is τ Contains the information of the virtual machine and the task node t i information and status, and take them together as t i The original feature vector v i ={τ,r i ,d i ,B,c k ,f k ,u k ,Scheduled(t i )}, that is, the Raw feature of each task is input into the deep reinforcement learning agent DRL scheduling agent; at the same time, the reward obtained in the previous step is also input into the deep reinforcement learning agent;

[0110] Use the GAT-based feature extraction layer based on graph attention network calculation to process the directed acyclic graph corresponding to the workflow and extract the t of each task node. i Feature vector with workflow topology information Feature of each task:

[0111] h 0 (t i )=v i ,

[0112]

[0113] Where N(t i ) indicates that i The set of all adjacent task nodes, σ is the activation function, W is the weight matrix, α i,j Yes i and t j The attention coefficient between

[0114] The global feature H(G) of the workflow can be obtained by pooling the features of the Lth layer of all task nodes;

[0115] The policy network Actor Network consists of MLP1 and softmax modules. It calculates the probability of selecting each schedulable task node according to the feature of the current schedulable task node: for the current schedulable node t i , select t i The probability is:

[0116]

[0117] According to the calculated decision probability, the task node that the agent should currently schedule is obtained, that is, the action that should be taken, Action a τ ;

[0118] Step S5: Optimize the network of the deep reinforcement learning agent based on the PPO algorithm, which includes two loss functions, namely the strategy loss L A τ and the critic loss L C τ , which are used to optimize the policy network Actor network and the critic network Criticnetwork respectively, and use these two loss functions to back propagate to optimize the deep reinforcement learning scheduling agent network;

[0119] Strategy loss L A τ Used to optimize the policy network, calculated by the following formula:

[0120]

[0121] Critic Loss L C τ It is used to optimize the critic network's estimate of the value function as a mean square error loss, calculated as follows:

[0122]

[0123] V(s τ )=MLP2(H(G τ )),

[0124] Step S6: Repeat steps S4 to S5 until every state-action pair in the experience pool is processed;

[0125] Step S7: Initialize the simulation environment and workflow;

[0126] Step S8: Get the current state, use the formula mentioned above to calculate the feature vector of each task node in the current workflow, and the strategy a that the current deep reinforcement learning scheduling model should adopt according to the following formula: τ :

[0127] According to the probability p i Calculate the task node that should be selected currently, where r is a random number satisfying r∈[0,1):

[0128]

[0129] And calculate the rewards and add them to the experience pool;

[0130] Step S9: For the samples in the experience pool, the network is optimized using the loss function mentioned above;

[0131] Step S10: Repeat steps S9 to S10 until the workflow execution is completed;

[0132] Step S11: Repeat steps S8 to S11 until the specified number of iterations is reached.

[0133] Comparison of experimental results:

[0134] Figure 6 This is a comparison chart of the reward fluctuation curve in the training process of the scheduling model of the present invention and the experimental results of the prior art, where the horizontal axis represents the number of iterative training rounds, the vertical axis is the average reward value, and GAT is the reward fluctuation curve of the present technology. It can be seen that after multiple rounds of iterations, the reward fluctuation curve of the present technology can not only converge, but also the average reward value is significantly higher than that of the prior art.

[0135] The above description is only a description of the preferred embodiments of the present application, and is not intended to limit the scope of the present application. Any changes or modifications made by any person skilled in the art based on the above disclosed technical contents shall be deemed as equivalent effective embodiments and shall fall within the scope of protection of the technical solution of the present application.

Claims

1. A large-scale scientific computing workflow scheduling method based on deep reinforcement learning, characterized in that: include: Step 1: Get the task nodes t in the scientific computing workflow i The amount of calculation r i , data volume i , and the sequential dependency relationship between the task nodes of the workflow represented by a directed acyclic graph, the bandwidth B obtained from the cloud computing platform, and the virtual machines VM k The computing power of k , failure rate f k , Energy consumption factor index u k , and based on the information obtained, establish a mathematical optimization model for large-scale scientific computing workflow scheduling with the goal of optimizing service quality; Step 2: The scheduling mathematical optimization model established in step 1 with the goal of optimizing service quality is further modeled using the Markov decision process, and the graph attention network is used as the feature extraction layer to extract the complex structural features of large-scale scientific computing workflows as the state input of the subsequent deep reinforcement learning scheduling strategy network; the graph attention network feature extraction layer, the deep reinforcement learning scheduling strategy network, and the deep reinforcement learning critic network together constitute the deep reinforcement learning scheduling agent; Step 3, using a genetic algorithm to generate several groups of suboptimal scheduling solutions for the scheduling mathematical optimization model established in step 1 with the goal of optimizing service quality, and converting these scheduling solutions into a series of sampling trajectories consisting of state-action pairs; Step 4: Use the sampled trajectories obtained by the genetic algorithm in step 3 to train the deep reinforcement learning scheduling agent, and after several training iterations, place the deep reinforcement learning scheduling agent into a simulation environment for further training, and output a scheduling strategy with the best service quality based on the execution order of each task node.

2. The method according to claim 1, characterized in that The sequential dependency relationship between the task nodes in the workflow in step 1 is reflected as follows: for a task node t i , only when all its predecessor task nodes are completed and data transmission is completed, can it start execution, that is, t i The earliest start time EST(t i ) satisfies the formula: Where Pred(t i ) is the task node t i The set of all predecessor nodes of , FT(t j ) is the task node t j The end time, CT(t i ) is the task node t i The time required for data transmission, the two meet the following requirements: FT(t i )=ST(t i )+ET(t i ,Alc(t i )) Where Alc(t i ) refers to the task node t i The virtual machine assigned; task node t i Assigned to the virtual machine VM k The calculation execution time ET(t i ,VM k )satisfy:

3. The method according to claim 1, characterized in that The mathematical optimization model for large-scale scientific computing workflow scheduling described in step 1 aims to optimize the service quality and includes three specific optimization goals, namely, minimizing the workflow completion time MP, minimizing the energy consumption EC required for workflow completion, and maximizing the reliability RB of workflow execution, namely: minimize MP, minimize EC, maximize RB, The workflow completion time MP is equivalent to the completion time FT(t i ), that is: The energy consumption EC required to complete the workflow is the computational energy consumption C(t i ,VM k ), that is: Where Alc(t i ) refers to the task node t i The virtual machine assigned, assuming that the task node t i Assigned to the virtual machine VM k , then its calculation energy consumption C(t i ,VM k ) and its execution time ET(t i ,VM k ), virtual machine computing power c k and energy consumption factor u k Relevant, namely: C(t i ,VM k )=ET(t i ,VM k )×c k u k The reliability of workflow execution RB is the probability that all task nodes will execute without failure. It is the product of the reliability of each task node, that is: Where Alc(t i ) refers to the task node t i The virtual machine assigned, assuming that the task node t i Assigned to the virtual machine VM k , then its reliability D(t i ,VM k ) follows a Poisson distribution, and its execution time ET(t i ,VM k ), virtual machine failure rate f k Related, e is the base of natural logarithms: D(t i ,VM k )=e -fk×ET(ti,VMk) .。 4. The method according to claim 1, characterized in that In step 2, the Markov decision process model is established, and each step consists of a Markov quaternion M τ ={s τ ,a τ ,P τ ,R τ }, representing the state, action, transition probability and reward when the training reaches time τ respectively; Status τ It is the status of each virtual machine on the current cloud computing platform and the execution status of the current workflow. For the status of the workflow, it is necessary to use the graph attention network to process the graph structure of the workflow to extract the topological structure information of the workflow; Action a τ It is stipulated that a task node is selected from all currently schedulable task nodes, and a task node t i To be considered schedulable, the following conditions must be met: First, t i All predecessor task nodes of t should have been scheduled, and then t i At least one of the predecessor task nodes has started executing; The transition probability P needs to be learned; Reward R τ Defined by the following formula, where C1, C2, C3 are weight coefficients, RB τ and EC τ are the overall reliability and energy consumption of all scheduled tasks at time τ; EFT τ is the estimated completion time of all scheduled tasks at time τ:

5. The method according to claim 4, characterized in that: In step 2, for state s τ : The information of each virtual machine in the cloud computing environment and the task node t i The information and status of task nodes t i The amount of calculation r i , data volume i , and the sequential dependency relationship between the task nodes of the workflow represented by a directed acyclic graph, the bandwidth B obtained from the cloud computing platform, and the virtual machines VM k The computing power of k , failure rate f k , Energy consumption factor index u k , whether the node has been scheduled (t i ), as t i The original feature vector v i ={τ,r i ,d i ,B,c k ,f k ,u k ,Scheduled(t i )}, and then use the graph attention network to process the directed acyclic graph corresponding to the workflow and extract the feature vector h containing the workflow topology information L (t i ), and in each layer of the graph attention network, t i The eigenvector h l (t i ) is calculated as follows: h 0 (t i )=v i , In the above formula, L is the total number of layers in the graph attention network, h l-1 (t i ) and h l-1 (t j ) is the feature vector of the previous layer, N(t i ) indicates that i The set of all adjacent task nodes, σ is the activation function, W is the learnable weight matrix, α i,j is the task node t i and t j The attention coefficient between The global feature H(G) of the workflow is obtained by pooling the features of the Lth layer of all task nodes.

6. The method according to claim 4, characterized in that: In step 2, action a τ Calculation method: Based on the feature vectors of all currently schedulable task nodes obtained by the graph attention network, a deep reinforcement learning scheduling strategy network built by a multi-layer perceptron MLP1 and a softmax module is used to calculate the probability of selecting each schedulable task node in the current situation; if t i Currently schedulable, then choose t i The probability is: Finally, according to the probability p i Calculate the task node that should be selected currently, where r is a random number satisfying r∈[0,1):

7. The method according to claim 1, characterized in that: In step 4, a genetic algorithm is used to generate several groups of suboptimal scheduling solutions for the scheduling problem. The scheduling solution given is a sequence of task nodes arranged in the scheduling order. The genetic algorithm includes the following steps: Step 41: define the population selection mechanism and the maximum number of iterations e of the genetic algorithm; Step 42: Define the population encoding method of the genetic algorithm and generate a random initial population pop; Step 43: For each scheduling scheme in the population pop, calculate its fitness function; Step 44: for pop, generate a new population pop′ according to the mutation and crossover mechanism; Step 45: Select the best feasible solution in pop′; Step 46: recalculate the adaptation function of each scheduling scheme in pop′ and replace pop with pop′; Step 47: Repeat steps 44 to 46 until the number of iterations reaches e; Step 48: Convert each pop scheduling scheme into a state-action pair {s τ ,a τ ,} trajectory sequence, as the output of the algorithm.

8. The method according to claim 7, characterized in that: In step 4, in order to ensure that the solution obtained by the mutation mechanism in step 44 is still feasible, it includes the following steps: Step 441: randomly select a scheduling scheme I in pop and randomly select a mutation point MutPoint on I; Step 442: For the task node t corresponding to MutPoint on I, start from MutPoint and keep moving forward on I until the first predecessor node of t is encountered, or the node moves to the beginning of I, and the position moved to is recorded as start; Step 443: For the task node t corresponding to MutPoint on I, start from MutPoint and keep moving backward on I until the first successor node of t is encountered, or the node moves to the end of I, and the position moved to is recorded as end; Step 444: For the scheduling scheme I, randomly select a position between start and end, exchange the position and the task node on MutPoint, obtain the scheduling scheme I' and output it.

9. The method according to claim 7, characterized in that: In step 4, in order to ensure that the solution obtained by the crossover mechanism in step 44 is still feasible, it includes the following steps: Step 441′: Randomly select two scheduling schemes I in pop a and I b ; Step 442 ′: randomly select an intersection point CxPoint; Step 443′: Intercept I b [0:CxPoint] as I a’ , intercept I a [0:CxPoint] as I b’ ; Step 444′: From I a Starting from the CxPoint task node until I a At the end of I, for each task node t, if t is not in I a’ If t is within I a’ Behind; Step 445′: From I b Starting from the CxPoint task node until I b At the end of I, for each task node t, if t is not in I b’ If t is within I b’ Behind; Step 446′: Output the crossover scheduling scheme I a’ ,I b’ .

10. The method according to claim 1, characterized in that: In step 4, the basic framework for training the deep reinforcement learning scheduling agent is the PPO algorithm, which is characterized by including two output network modules, namely the deep reinforcement learning scheduling strategy network and the deep reinforcement learning critic network, where the critic network is only used in the training phase; the two networks are optimized according to two loss functions, namely the strategy loss L A τ and the critic loss L C τ ; Strategy loss L A τ Used to optimize the policy network, calculated by the following formula: where π θ (a τ |s τ ) is the cloud computing platform and workflow in state s τ Next, select action a τ The probability of the corresponding task node, ∈, is a truncation parameter, generally set to 0.1 to 0.3; τ is the advantage function. Under the premise of a given discount coefficient γ, the advantage function A τ By the value function V(s τ’+τ ), Reward R τ’+τ The cumulative sum of the differences is: Critic Loss L C τ Used to optimize the critic network's estimate of the value function, it is calculated using the mean square error loss within τ' steps, that is, the following formula: Where E represents the mathematical expectation, γ is the discount factor mentioned above, and R τ’+τ is the reward mentioned above; and V(s τ ) is the critic network MLP2 estimates the current global feature H(G τ ) is calculated by the following formula: V(s τ )=MLP2(H(G τ )),。

Images