A large-scale scientific computing workflow scheduling method based on deep reinforcement learning

Abstract

The application relates to the fields of cloud computing and deep reinforcement learning, and discloses a large-scale scientific computing workflow scheduling method based on deep reinforcement learning. The application introduces a graph attention network to extract workflow structure features, and designs a deep reinforcement learning sampling strategy based on a genetic algorithm to generate initial sample trajectories for a deep reinforcement learning intelligent agent, so that the training process is accelerated. Experiments show that the application can obtain a faster training speed in a simulated cloud environment, and according to the generated large-scale scientific computing workflow scheduling strategy, the workflow can be executed on a cloud computing platform, and better completion time, energy consumption and reliability can be achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of cloud computing and deep reinforcement learning. BACKGROUND

[0002] In the field of scientific computing, a workflow refers to a set of interrelated tasks that need to be executed in a specific order. The scheduling of a workflow refers to the allocation of appropriate computing resources (such as virtual machines) to the tasks in the workflow to meet specific goals or quality of service. In recent years, there has been an increasing demand for large-scale scientific computing in various fields, and cloud computing technology has brought great convenience to users in obtaining high-performance computing resources. Under this background, the scheduling of large-scale scientific workflows on cloud computing platforms has become an important research topic.

[0003] Traditional meta-heuristic algorithms face the challenge of low search efficiency, so in recent years, with the development of artificial neural networks and reinforcement learning techniques, some research has explored the use of deep reinforcement learning to solve workflow scheduling problems.

[0004] Prior art and its analysis:

[0005] Yang et al. (Yang Z, Nguyen P, Jin H, Nahrstedt K. MIRAS: Model-based reinforcement learning for microservice resource allocation over scientific workflows. 2019 IEEE 39th international conference on distributed computing systems (ICDCS): IEEE; 2019. p. 122-32.) first proposed model-based reinforcement learning to schedule scientific workflows in distributed cloud systems. Yu et al. (Yu X, Wu W, Wang Y. Integrating cognition cost with reliability QoS for dynamic workflow scheduling using reinforcement learning. IEEE Transactions on Services Computing. 2023;16(4):2713-26.) proposed a deep reinforcement learning model that optimized multiple quality of service objectives. However, the above studies only utilized explicit raw information of tasks and workflows as state information for the deep reinforcement learning model. For large-scale scientific workflows, it is difficult to extract their complex structural features, resulting in poor performance in scheduling large-scale scientific workflows and slow training speed.

[0006] Mao et al. (Mao H, Schwarzkopf M, Venkatakrishnan SB, Meng Z, Alizadeh M. Learning scheduling algorithms for data processing clusters. Proceedings of the ACM special interest group on data communication. 2019. p. 270-88.) first proposed introducing a graph neural network into the reinforcement learning model to extract the global features of the workflow, and then the deep reinforcement learning model made further decisions based on the global features of the workflow. However, this model, due to its reliance on a relatively simple message passing neural network, is still limited in the size of the workflow it can handle and faces the problem of low training sampling efficiency on large-scale workflows.

[0007] Zhou et al. (Zhou Y, Li X, Luo J, Yuan M, Zeng J, Yao J. Learning to optimize DAG scheduling in heterogeneous environment. 2022 23rd IEEE International Conference on Mobile Data Management (MDM): IEEE; 2022. p. 137-46.) and Song et al. (Song Y, Li C, Tian L, Song H. A reinforcement learning based job scheduling algorithm for heterogeneous computing environment. Computers and Electrical Engineering. 2023;107:108653.) used basic message passing neural network and graph convolution network to extract the features of workflow and the features of task nodes and the whole environment, respectively. However, these methods are not effective in large-scale scientific workflow scheduling tasks. In addition, the sampling efficiency of existing deep reinforcement learning scheduling methods limits the training speed on large-scale workflows.

[0008] In summary, previous methods pay less attention to the complex workflow structure and the sampling efficiency of deep reinforcement learning, which makes the training and performance of large-scale scientific workflow scheduling models unsatisfactory, especially when there are multiple quality of service requirements. SUMMARY

[0009] In view of the problems that the existing deep reinforcement learning technology is difficult to extract the structure information of the workflow when scheduling large-scale scientific computing workflow, and the sampling efficiency of reinforcement learning training is low, the present application provides a large-scale scientific computing workflow scheduling method based on deep reinforcement learning. The present application introduces a graph attention network to extract the structural features of the relationship between each task node of the workflow for the deep reinforcement learning scheduling model. Using the feature representation and structural information obtained by the graph attention network, the deep reinforcement learning agent can show a better strategy. Secondly, a deep reinforcement learning sampling strategy based on genetic algorithm is designed, which uses genetic algorithm to obtain a suboptimal scheduling scheme, avoiding excessive useless exploration of the reinforcement learning agent in large workflows, thereby improving the training efficiency. Through effective extraction of the features of large-scale workflows and efficient sampling, the present application improves the training speed of the deep reinforcement learning scheduling model, and the generated scheduling strategy can better meet the quality of service indicators including completion time, energy consumption and reliability, thereby providing benefits for users of large-scale scientific computing.

[0010] The technical solution of the present invention is:

[0011] A large-scale scientific computing workflow scheduling method based on deep reinforcement learning with improved structural feature extraction and sampling, including:

[0012] Step 1: Get each task node in the scientific computing workflow t i The amount of calculation r i , data volume d i , and the sequential dependency relationship between the task nodes of the workflow represented by a directed acyclic graph, the bandwidth obtained from the cloud computing platform B , each virtual machine VM k Computing power c k , failure rate f k , energy consumption factor index u k , and based on the information obtained, establish a mathematical optimization model for large-scale scientific computing workflow scheduling with the goal of optimizing service quality;

[0013] Step 2: The scheduling mathematical optimization model established in Step 1, which aims to optimize service quality, is further modeled using a Markov decision process. A graph attention network is used as a feature extraction layer to extract the complex structural features of large-scale scientific computing workflows, which serve as the state input for the subsequent deep reinforcement learning scheduling strategy network. The graph attention network feature extraction layer, the deep reinforcement learning scheduling strategy network, and the deep reinforcement learning critic network together constitute the deep reinforcement learning scheduling agent.

[0014] Step 3: Genetic algorithm is used to generate several sets of suboptimal scheduling solutions for the scheduling mathematical optimization model established in step 1 with the goal of optimizing service quality, and these scheduling solutions are converted into a series of sampling trajectories consisting of state-action pairs;

[0015] In step 4, the deep reinforcement learning scheduling agent is trained using the sampling trajectory obtained by the genetic algorithm in step 3. After several training iterations, the deep reinforcement learning scheduling agent is placed in a simulation environment for further training, and the scheduling strategy with the best service quality based on the execution order of each task node is output.

[0016] Furthermore, in step 1, the order dependency between the task nodes in the workflow in step 1 is reflected as follows: t i , can only start execution when all its predecessor task nodes are completed and data transmission is completed, that is,t i the earliest start time of task node EST ( t i ) satisfies the formula:

[0017]

[0018] where Pred( t i ) is a set of all predecessors of task node t i , FT ( t j ) is the end time of task node t j , CT ( t i ) is the time required for data transmission of task node t i , both of which satisfy:

[0019]

[0020]

[0021] where Alc ( t i ) refers to the virtual machine to which task node t i is assigned; the computation execution time of task node t i executed on virtual machine VM k ET ( t i , VM k satisfies:

[0022]

[0023] Further, in step 1, the large-scale scientific computing workflow scheduling mathematical optimization model described in step 1 takes optimizing service quality as the goal, containing three specific optimization objectives, namely minimizing workflow completion time MP , minimizing energy consumption required for workflow completion EC , and maximizing reliability of workflow execution RB , namely:

[0024]

[0025] ​

[0026]

[0027] where T is the completion time of the workflow MP is the completion time of all the task nodes of the workflow FT ( t i ) is the latest value, i.e.

[0028]

[0029] where E is the energy consumption of the workflow EC is the computation energy consumption of all the task nodes of the workflow C ( t i , VM k ) is the sum, i.e.

[0030]

[0031] where VMi Alc ( t i ) is the virtual machine assigned to task node Ti t i , assuming that task node Ti t i is assigned to virtual machine VMi VM k , then its computation energy consumption Ei C ( t i , VM k ) is related to its execution time Ti ET ( t i , VM k ), the computation capability of virtual machine VMi c k and the energy consumption factor of virtual machine VMi u k , i.e.

[0032]

[0033] where R is the reliability of the workflow execution RB is the probability that all the task nodes execute without failure, which is the product of the reliabilities of the task nodes, i.e.

[0034]

[0035] where VMi Alc ( t i ) is the virtual machine assigned to task node Ti ti the virtual machine to which the task node is assigned t i the virtual machine to which the task node is assigned VM k the reliability of the virtual machine to which the task node is assigned D ( t i , VM k is subject to a Poisson distribution, and its execution time ET ( t i , VM k ), the virtual machine failure rate f k is related to e is the base of the natural logarithm:

[0036]

[0037] Further, in step 2, the Markov decision process model is established, and each step is determined by a Markov quadruple M τ = { s τ , a τ , P τ , R τ} respectively represent the state, action, transition probability and reward of the training at the time τ ;

[0038] State s τ is the state of each virtual machine of the current cloud computing platform and the execution state of the current workflow, wherein for the state of the workflow, a graph attention network is used to process the directed acyclic graph corresponding to the workflow to extract the topological structure information of the workflow;

[0039] Action a τ is defined as selecting a task node from all currently schedulable task nodes, and a task node t i is considered schedulable if it meets the following conditions: first t i all predecessor task nodes of the task node have been scheduled, and second t i at least one predecessor task node of the task node has started execution;

[0040] Transition probability P needs to be learned;

[0041] award R τ It is defined by the following formula, where C 1, C 2, C 3 is the weight coefficient, RB τ and EC τ They are τ The overall reliability and energy consumption of all scheduled tasks at all times; EFT τ yes τ Estimated completion time of all scheduled tasks at this moment:

[0042]

[0043] Furthermore, for the state s τ :

[0044] The information of each virtual machine in the cloud computing environment and the task nodes t i Information and status of task nodes t i The amount of calculation r i , data volume d i , and the sequential dependency relationship between the task nodes of the workflow represented by a directed acyclic graph, the bandwidth obtained from the cloud computing platform B , each virtual machine VM k Computing power c k , failure rate f k , energy consumption factor index u k , whether the node has been scheduled Scheduled ( t i ), as t i The original eigenvector of v i ={ τ , r i , d i , B , c k , f k , u k , Scheduled (t i )}, and then use the graph attention network to process the directed acyclic graph corresponding to the workflow and extract the feature vector containing the workflow topology information h L ( t i ), and in each layer of the graph attention network t i The eigenvector of h l ( t i ) is calculated as follows:

[0045]

[0046]

[0047] In the above formula, L is the total number of layers in the graph attention network, h l-1 ( t i )and h l-1 ( t j ) is the feature vector of the previous layer, N ( t i ) indicates that t i The set of all adjacent task nodes, σ is the activation function, W is a learnable weight matrix, α i,j It is a task node t i and t j The attention coefficient between

[0048] Global features of workflows H ( G ) is determined by the first L The features of the layer are obtained through pooling.

[0049] Furthermore, the action a τ Calculation method:

[0050] Based on the feature vectors of all currently schedulable task nodes obtained by the graph attention network, a multi-layer perceptron is used to MLP 1 and the deep reinforcement learning scheduling strategy network built with the softmax module to calculate the probability of selecting each schedulable task node in the current situation.t i If current schedulable, then select t i with probability

[0051]

[0052] Finally, according to the probability p i calculate the task node that should be selected currently, where r is a random number satisfying r ∈ [0, 1):

[0053]

[0054] Further, in step 3, a genetic algorithm is generated for the scheduling problem to generate a number of sets of suboptimal scheduling schemes, and the scheduling scheme (solution individual) given by the genetic algorithm is a sequence formed by arranging each task node in a scheduling order. The genetic algorithm includes the following steps:

[0055] Step 41: define the population selection mechanism and the maximum number of iteration rounds of the genetic algorithm e ;

[0056] Step 42: define the population coding mode of the genetic algorithm, and generate a random initial population pop ;

[0057] Step 43: for each scheduling scheme in the population pop , calculate its fitness function according to

[0058] Step 44: for pop , generate a new population pop ′ according to the mutation and crossover mechanism

[0059] Step 45: select the optimal feasible solution in pop ′;

[0060] Step 46: recalculate the fitness function of each scheduling scheme in pop ′, replace pop ′ with pop ;

[0061] Step 47: repeat steps 44-46 until the number of iteration rounds reaches e ;

[0062] Step 48: convert each scheduling scheme of pop into a state-action pair s τ , a τ , trajectory sequence as the output of the algorithm.

[0063] Furthermore, in step 44, the mutation mechanism of the genetic algorithm includes the following steps:

[0064] Step 441: Random Selection pop A scheduling scheme in I and in I Randomly select mutation points MutPoint ;

[0065] Step 442: For MutPoint exist I The corresponding task node t ,from MutPoint Start in I Keep moving forward until you encounter the first t 's predecessor node, or move to I The beginning of the , the position moved to is recorded as start ;

[0066] Step 443: For MutPoint exist I The corresponding task node t ,from MutPoint Start in I Keep moving back until you encounter the first t The successor node of I The end of the move is recorded as end ;

[0067] Step 444: Scheduling plan I ,exist start and end Randomly select a position between MutPoint The task node on the I’ And output.

[0068] Furthermore, in step 44, the crossover mechanism of the genetic algorithm includes the following steps:

[0069] Step 441′: Random Selection pop Two scheduling schemes in I a and I b ;

[0070] Step 442′: Randomly select intersection points CxPoint ;

[0071] Step 443′: Interception Ib [0: CxPoint ] as Ia’ , intercept Ia [0: CxPoint ] asIb

[0072] Step 444': for each task node from the first task node of the Ia CxPoint Ia t， If t is not in Ia’ , then t is appended to the end of Ia’ ;

[0073] Step 445': for each task node from the first task node of the Ib CxPoint Ib t If t is not in Ib’ , then t is appended to the end of Ib’ ;

[0074] Step 446': output the cross-scheduled scheme I a’ 、 I b’ .

[0075] Further, in step 4, the basic framework of training the deep reinforcement learning scheduling agent is the PPO algorithm, which has the characteristics of containing two output network modules, namely the deep reinforcement learning scheduling policy network and the deep reinforcement learning critic network, wherein the critic network is only used in the training phase; the two networks are respectively optimized according to two loss functions, namely the policy loss L A τ and the critic loss L C τ ;

[0076] The policy loss L A τ is used to optimize the policy network, which is calculated by the following formula:

[0077]

[0078] Wherein π θ ( a τ |s τ ) is the state of the cloud computing platform and the workflow s τ , and the action a τ ​​​​​​​probabilities of the corresponding task nodes, ϵ is a truncation parameter, generally set to 0.1~0.3; A τ is a utility function, under the premise of a given discount factor γ , A τ is obtained by summing the value function τ’ in the step V ( s τ’+τ ),reward R τ’+τ

[0079]

[0080] critic loss L C τ is used to optimize the critic network's estimate of the value function, which uses the mean squared error loss in the step to calculate, that is, the following formula: τ’

[0081]

[0082] where E denotes the mathematical expectation, γ is the discount factor mentioned above, R τ’+τ is the reward mentioned above; and V ( s τ ) is the value function obtained by the critic network MLP 2 estimate the current global features of the workflow H ( G τ ) is calculated by the following formula:

[0083]

[0084] With the above technical solutions, the beneficial effects of the present application are:

[0085] By introducing the graph attention network, the structural features of the relationship between the task nodes of the workflow are extracted for the deep reinforcement learning scheduling agent, so that the deep reinforcement learning scheduling agent can adopt a better strategy;

[0086] By designing a deep reinforcement learning sampling strategy based on a genetic algorithm, a suboptimal scheduling scheme is obtained, and then a sampling trajectory with useful reward information is obtained, which avoids excessive useless exploration of the deep reinforcement learning scheduling agent in the training process, thereby improving the training efficiency;

[0087] ​The generated scheduling policy can better generate scheduling solutions that meet the quality of service indicators, including shorter workflow computation time, less energy consumption, and higher reliability, thus providing benefits to users when using large-scale scientific computing workflows. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] Figure 1 This is an application example of the large-scale scientific computing workflow scheduling method based on deep reinforcement learning of the present invention;

[0089] Figure 2 This is the main flow chart of the large-scale scientific computing workflow scheduling method based on deep reinforcement learning of the present invention;

[0090] Figure 3 A training flowchart for scheduling large-scale scientific computing workflows based on deep reinforcement learning with improved structural feature extraction and sampling;

[0091] Figure 4 This is an example diagram of the mutual conversion between the deep reinforcement learning scheduling strategy and the scheduling scheme obtained by the genetic algorithm in the present invention;

[0092] Figure 5 This is a diagram of the training and network structure of the deep reinforcement learning scheduling agent of the present invention;

[0093] Figure 6 It is a comparison chart of the reward fluctuation curve during the scheduling model training process of the present invention and the experimental results of the prior art. DETAILED DESCRIPTION

[0094] This paper proposes a large-scale scientific computing workflow scheduling method based on deep reinforcement learning. Figure 1 As shown in the figure, in a cloud computing environment, users upload scientific computation application jobs in the form of large-scale scientific workflows. The scheduling model is responsible for determining the execution order of tasks in the workflow and assigning them to appropriate virtual machines in cloud computing resources. Finally, the computation results are fed back to the user.

[0095] exist Figure 1In the scheduling model, a deep reinforcement learning scheduling agent (DRLscheduling agent) receives information about large-scale scientific computing workflows as state input. After preprocessing, the workflow information is first extracted using a graph attention network (GAT) as a feature extraction layer. This feature vector is then input into the DRL scheduling policy network (the critic network in DRL scheduling is used only for training). The agent network then selects the task node to be scheduled as the scheduling decision. Virtual machines are allocated based on a greedy rule, finding the virtual machine that will complete the task node selected by the current action the earliest and adding it to the virtual machine's task queue.

[0096] The following further describes the technical solution of the present invention in conjunction with the accompanying drawings. The described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0097] The overall process of the technical solution of the present invention is as follows Figure 2 Shown, including:

[0098] First, obtain the information of each task node and cloud computing platform in the scientific computing workflow (including each task node t i The amount of calculation r i , data volume d i , and the sequential dependency relationship between the task nodes of the workflow represented by a directed acyclic graph, the bandwidth obtained from the cloud computing platform B , each virtual machine VM k Computing power c k , failure rate f k , energy consumption factor index u k ), and based on the information obtained, establish a mathematical optimization model for large-scale scientific computing workflow scheduling with the goal of optimizing service quality;

[0099] Next, the mathematical optimization model is further modeled using a Markov decision process, and a graph attention network is used to extract the complex structural features of large-scale scientific computing workflows, which serve as the state input for the subsequent deep reinforcement learning scheduling model agent.

[0100] Then, a genetic algorithm is used to generate several sets of suboptimal scheduling solutions for the above mathematical optimization model, and these scheduling solutions are converted into a series of sampling trajectories consisting of state-action pairs;

[0101] Finally, the deep reinforcement learning scheduling agent is trained using the sampling trajectories obtained in the genetic algorithm. After several training iterations, the deep reinforcement learning scheduling agent is placed in a simulation environment for further training, so that the deep reinforcement learning scheduling agent can output a scheduling strategy with the best service quality based on the execution order of each task node.

[0102] The implementation described is the present invention for scheduling Pegasus workflows on a cloud platform. Pegasus is a workflow dataset that contains more than 100 scientific computing workflows with more than 1,000 task nodes.

[0103] The training process of this embodiment is as follows Figure 3 The flowchart shown:

[0104] Step S1: Initialize the genetic algorithm population pop , and press Figure 5 Initialize the network of the deep reinforcement learning scheduling agent;

[0105] Step S2: In the population pop Run the genetic algorithm for several iterations and get the population pop’ ;

[0106] Step S3: pop’ The scheduling scheme in deep reinforcement learning is transformed into state-action pairs { s τ , a τ , The trajectory sequence is added to the experience pool of the deep reinforcement learning scheduling model, and the scheduling scheme generated by the genetic algorithm and the scheduling strategy generated by the deep reinforcement learning scheduling model are a τ The conversion relationship is as follows Figure 4 As shown in Figure 2, the scheduling solution generated by the genetic algorithm is a sequence of task node indexes arranged in the order of execution. This is also a topological sorting of the workflow. At each decision point, the first schedulable task in the sequence is selected for scheduling.

[0107] Step S4: For each state-action pair in the experience pool { s τ ,a τ ,}, calculate the corresponding reward based on the following formula R :

[0108]

[0109] Deep reinforcement learning scheduling agent network and training Figure 5 As shown. The state of the cloud computing environment Environment s τ Contains information about the virtual machine and the task node t i information and status, which together serve as t i The original eigenvector of v i ={ τ , r i , d i , B , c k , f k , u k , Scheduled ( t i )}, that is, the raw feature of each task is input into the deep reinforcement learning agent DRL scheduling agent; at the same time, the reward obtained in the previous step is also input into the deep reinforcement learning agent;

[0110] Use the feature extraction layer GAT-based feature extraction based on graph attention network calculation to process the directed acyclic graph corresponding to the workflow and extract the t i Feature vector with workflow topology information Feature of each task:

[0111]

[0112]

[0113] in N ( t i ) indicates that t i The set of all adjacent task nodes, σ is the activation function, W is the weight matrix, α i,j yes t i and t j The attention coefficient between

[0114] Global Feature of Workflow H ( G ) can be obtained by the first L The features of the layer are obtained through pooling;

[0115] The policy network Actor Network consists of MLP 1 and the softmax module, which calculates the probability of selecting each schedulable task node according to the feature of each schedulable task node: for the current schedulable node t i ,choose t i The probability is:

[0116]

[0117] According to the calculated decision probability, the task node that the agent should currently schedule, that is, the action that should be taken, is obtained. a τ ;

[0118] Step S5: Optimize the network of the deep reinforcement learning agent based on the PPO algorithm, which includes two loss functions, namely the policy loss L A τ and critics' losses L C τ , respectively used to optimize the policy network Actor network and the critic network Criticnetwork, and use these two loss functions for back propagation to optimize the deep reinforcement learning scheduling agent network;

[0119] Strategy loss L A τ Used to optimize the policy network, calculated by the following formula:

[0120]

[0121]

[0122] Critics' losses L C τ The value function used to optimize the critic network is estimated as the mean squared error loss, which is calculated as follows:

[0123]

[0124]

[0125] Step S6: repeat steps S4-S5 until each state-action pair in the experience pool is processed;

[0126] Step S7: initialize the simulation environment and workflow;

[0127] Step S8: obtain the current state, calculate the feature vector of each task node of the current workflow using the above-mentioned formula, and the policy that the current deep reinforcement learning scheduling model should take according to the following formula a τ :

[0128] According to the probability p i Calculate the task node that should be selected at present, wherein r is a random number satisfying r ∈[0, 1):

[0129]

[0130] and calculate the reward, which is added to the experience pool;

[0131] Step S9: for the samples in the experience pool, optimize the network using the above-mentioned loss function;

[0132] Step S10: repeat steps S9-S10 until the end of workflow execution;

[0133] Step S11: repeat steps S8-S11 until the specified number of iterations is reached.

[0134] Comparison of experimental results:

[0135] Figure 6 The reward fluctuation curve in the training process of the scheduling model of the present application and the experimental results of the prior art are compared in the figure, wherein the horizontal axis represents the number of iteration training rounds, and the vertical axis is the average reward value. GAT is the reward fluctuation curve of the present technology. It can be seen that after multiple iterations, the reward fluctuation curve of the present technology not only converges, but also has a significantly higher average reward value than the prior art.

[0136] The above description is only a description of the preferred embodiments of the present application, and is not any limitation on the scope of the present application. Any modification or modification made by any person skilled in the art according to the above-mentioned technical content shall be regarded as an equivalent effective embodiment, and shall fall within the scope of the technical scheme protected by the present application.

Claims

1. A large-scale scientific computing workflow scheduling method based on deep reinforcement learning, characterized by: include: Step 1: Get each task node in the scientific computing workflow t i The amount of calculation r i , data volume d i , and the sequential dependency relationship between the task nodes of the workflow represented by a directed acyclic graph, the bandwidth obtained from the cloud computing platform B , each virtual machine VM k Computing power c k , failure rate f k , energy consumption factor index u k , and based on the information obtained, establish a mathematical optimization model for large-scale scientific computing workflow scheduling with the goal of optimizing service quality; Step 2: The scheduling mathematical optimization model established in Step 1, which aims to optimize service quality, is further modeled using a Markov decision process. A graph attention network is used as a feature extraction layer to extract the complex structural features of large-scale scientific computing workflows, which serve as state inputs for the subsequent deep reinforcement learning scheduling strategy network. The graph attention network feature extraction layer, the deep reinforcement learning scheduling strategy network, and the deep reinforcement learning critic network together constitute the deep reinforcement learning scheduling agent. Step 3: Genetic algorithm is used to generate several sets of suboptimal scheduling solutions for the scheduling mathematical optimization model established in step 1 with the goal of optimizing service quality, and these scheduling solutions are converted into a series of sampling trajectories consisting of state-action pairs; Step 4: Use the sampled trajectories obtained by the genetic algorithm in step 3 to train the deep reinforcement learning scheduling agent. After several training iterations, the deep reinforcement learning scheduling agent is placed in the simulation environment for further training, and the scheduling strategy with the best service quality based on the execution order of each task node is output; In step 2, the information of each virtual machine in the cloud computing environment and the task node t i Information and status of task nodes t i The amount of calculation r i , data volume d i , and the sequential dependency relationship between the task nodes of the workflow represented by a directed acyclic graph, the bandwidth obtained from the cloud computing platform B , each virtual machine VM k Computing power c k , failure rate f k , energy consumption factor index u k , whether the node has been scheduled Scheduled ( t i ), as t i The original eigenvector of v i ={ τ , r i , d i , B , c k , f k , u k , Scheduled ( t i )}, and then use the graph attention network to process the directed acyclic graph corresponding to the workflow and extract the feature vector containing the workflow topology information h L ( t i ), and in each layer of the graph attention network t i The eigenvector of h l ( t i ) is calculated as follows: In the above formula, L is the total number of layers in the graph attention network, h l-1 ( t i )and h l-1 ( t j ) is the feature vector of the previous layer, N ( t i ) indicates that t i The set of all adjacent task nodes, σ is the activation function, W is a learnable weight matrix, α i,j It is a task node t i and t j The attention coefficient between Global features of workflows H ( G ) is determined by the first L The features of the layer are obtained through pooling; In step 4, the basic framework for training the deep reinforcement learning scheduling agent is the PPO algorithm, which is characterized by including two output network modules, namely the deep reinforcement learning scheduling policy network and the deep reinforcement learning critic network, where the critic network is only used in the training phase; these two networks are optimized according to two loss functions, namely the policy loss and the target loss. L A τ and critics' losses L C τ ; Strategy loss L A τ Used to optimize the policy network, calculated by the following formula: in π θ ( a τ |s τ ) is the cloud computing platform and workflow in a state s τ Next, select Action a τ The probability of the corresponding task node, ϵ is a cutoff parameter, set to 0.1~0.3; A τ Then it is the advantage function, given the discount coefficient γ Under the premise of A τ Depend on τ' Value function within a step V ( s τ’+τ ),award R τ’+τ The cumulative sum of the differences is: Critics' losses L C τ It is used to optimize the critic network's estimate of the value function, which uses τ' The mean square error loss within the step is calculated as follows: in E represents the mathematical expectation, γ is the discount factor mentioned above, R τ’+τ is the reward mentioned above; and V ( s τ ) is the critic network MLP 2. Estimate the current global characteristics of the workflow H ( G τ ) is calculated by the following formula:

2. The method according to claim 1, characterized in that The sequential dependency relationship between the task nodes in the workflow in step 1 is reflected as follows: for a task node t i , can only start execution when all its predecessor task nodes are completed and data transmission is completed, that is, t i Earliest start time EST ( t i ) satisfies the formula: Where Pred( t i ) is the task node t i The set of all predecessor nodes of FT ( t j ) is the task node t j The end time, CT ( t i ) is the task node t i The time required for data transmission, the two respectively meet: in Alc ( t i ) refers to the task node t i Assigned virtual machine; task node t i Assigned to the virtual machine VM k Execution time of calculations performed on ET ( t i , VM k )satisfy:

3. The method according to claim 1, characterized in that The mathematical optimization model for large-scale scientific computing workflow scheduling described in step 1 aims to optimize service quality and includes three specific optimization goals, namely, minimizing workflow completion time. MP , minimize the energy consumption required to complete the workflow EC , maximize the reliability of workflow execution RB ,Right now: The workflow completion time MP Equivalent to the completion time of all nodes in the workflow FT ( t i ), that is: Energy consumption required to complete the workflow EC is the computational energy consumption of all task nodes in the workflow C ( t i , VM k ), that is: in Alc ( t i ) refers to the task node t i The assigned virtual machine, assuming the task node t i Assigned to a virtual machine VM k , then its computing energy consumption C ( t i , VM k ) and its execution time ET ( t i , VM k ), virtual machine computing power c k and energy consumption factor u k Relevant, namely: Reliability of workflow execution RB is the probability that all task nodes execute without failure, which is the product of the reliability of each task node, that is: in Alc ( t i ) refers to the task node t i The assigned virtual machine, assuming the task node t i Assigned to a virtual machine VM k , then its reliability D ( t i , VM k ) follows a Poisson distribution, and its execution time ET ( t i , VM k ), virtual machine failure rate f k related, e is the base of natural logarithms:

4. The method according to claim 1, characterized in that In step 2, the Markov decision process model is established, and each step consists of a Markov quadruple M τ = { s τ , a τ , P τ , R τ } decided to represent the training τ The state, action, transition probability and reward at the moment; state s τ It is the status of each virtual machine in the current cloud computing platform and the execution status of the current workflow. For the status of the workflow, it is necessary to use the graph attention network to process the graph structure of the workflow to extract the topological structure information of the workflow; action a τ It is stipulated that a task node is selected from all currently schedulable task nodes, and a task node t i To be considered schedulable, the following conditions must be met: First t i All predecessor task nodes should have been scheduled, and then t i At least one of the predecessor task nodes has started executing; Transition probability P Need to learn to obtain; award R τ It is defined by the following formula, where C 1, C 2, C 3 is the weight coefficient, RB τ and EC τ They are τ The overall reliability and energy consumption of all scheduled tasks at all times; EFT τ yes τ Estimated completion time of all scheduled tasks at this moment:

5. The method according to claim 4, characterized in that In step 2, the action a τ Calculation method: Based on the feature vectors of all currently schedulable task nodes obtained by the graph attention network, a multi-layer perceptron is used to MLP 1 and the deep reinforcement learning scheduling strategy network built with the softmax module to calculate the probability of selecting each schedulable task node in the current situation; if t i Currently schedulable, then choose t i The probability is: Finally, according to the probability p i Calculate the task node that should be selected currently, where r is satisfied r ∈[0,1) random number:

6. The method according to claim 1, characterized in that In step 4, a genetic algorithm is used to generate several suboptimal scheduling solutions for the scheduling problem. The scheduling solution provided is a sequence of task nodes arranged in the scheduling order. The genetic algorithm includes the following steps: Step 41: Define the population selection mechanism and maximum number of iterations for the genetic algorithm e ; Step 42: Define the population encoding method of the genetic algorithm and generate a random initial population pop ; Step 43: Population pop For each scheduling scheme in , calculate its fitness function; Step 44: Right pop , generating new populations based on mutation and crossover mechanisms pop '; Step 45: Pick pop The optimal feasible solution in ′; Step 46: Recalculate pop The adaptation function of each scheduling scheme in ′ is used pop 'replace pop ; Step 47: Repeat steps 44 to 46 until the number of iterations reaches e ; Step 48: pop Each scheduling scheme is converted into a state-action pair { s τ , a τ }Trajectory sequence, as the output of the algorithm.

7. The method according to claim 6, characterized in that In step 4, to ensure that the solution obtained by the mutation mechanism in step 44 is still feasible, it includes the following steps: Step 441: Random Selection pop A scheduling scheme in I and in I Randomly select mutation points MutPoint ; Step 442: For MutPoint exist I The corresponding task node t ,from MutPoint Start, in I Keep moving forward until you encounter the first t 's predecessor node, or move to I The beginning of the , the position moved to is recorded as start ; Step 443: For MutPoint exist I The corresponding task node t ,from MutPoint Start, in I Keep moving back until you encounter the first t The successor node of I The end of the move is recorded as end ; Step 444: Scheduling plan I ,exist start and end Randomly select a position between MutPoint The task node on the I’ And output.

8. The method according to claim 6, characterized in that In step 4, to ensure that the solution obtained by the crossover mechanism in step 44 is still feasible, it includes the following steps: Step 441′: Random Selection pop Two scheduling schemes in I a and I b ; Step 442′: Randomly select intersection points CxPoint ; Step 443′: Interception Ib [0: CxPoint ] as Ia' , intercept Ia [0: CxPoint ] as Ib '; Step 444′: From Ia No. CxPoint Task nodes start until Ia At the end of t， if t Not present Ia' Inside, then t Append to Ia' Behind Step 445′: From Ib No. CxPoint Task nodes start until Ib At the end of t ,if t Not present Ib' Inside, then t Append to Ib' Behind Step 446′: Output the scheduling plan after crossover I a’ 、 I b’ .

Images