Heterogeneous resource-oriented adaptive task scheduling method based on deep reinforcement learning
Through the dual-agent model of deep reinforcement learning, combined with the MAAC algorithm and attention mechanism, the adaptability and load balancing problems of task scheduling in the heterogeneous computing power platform are solved, and the efficiency and flexibility of task scheduling are achieved.
Patent Information
- Application Number
- CN202410970990.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-19
- Publication Date
- 2024-11-08
AI Technical Summary
The prior art lacks adaptive capabilities when scheduling tasks in heterogeneous computing power platforms, fails to effectively consider the preferences of tasks and the advantages of nodes, and single-objective optimization leads to load imbalance, making it difficult to cope with complex task scheduling environments.
Adopting a dual-agent model based on deep reinforcement learning, through the collaborative optimization of task agents and node agents, combined with MAAC algorithms and attention mechanisms, an adaptive task scheduling strategy is established to optimize scheduling time and load balancing.
The adaptability and load balancing of task scheduling are realized, scheduling efficiency and adaptability are improved, and the appropriate nodes can be matched according to task needs, and the scheduling time and load balancing are optimized.
Smart Images

Figure CN118916132A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of heterogeneous computing power platforms, and in particular to an adaptive task scheduling method for heterogeneous resources based on deep reinforcement learning. Background Art
[0002] With the advent of the cloud computing era, a large number of heterogeneous tasks are difficult to run on a single server and need to rely on more complex heterogeneous computing platforms. Modern high-performance computing applications usually use DAG-based task graph models to represent the dependencies between tasks. DAG task scheduling on heterogeneous computing platforms is an NP-hard problem and is even more complicated in actual scheduling.
[0003] Most task scheduling only considers a single type of task, without considering the preference of the task and the excellence of the node, and has poor adaptability. At the same time, the current heuristic-based task scheduling algorithm lacks the versatility of scenarios and is difficult to cope with more complex task scheduling environments; the task scheduling method based on machine learning is also a research hotspot, but due to the complexity of the algorithm, the modeling is difficult and the training time is long. In addition, most of the current task scheduling strategies use a single goal as the optimization goal, and do not fully consider the performance of heterogeneous computing platforms when scheduling tasks. Summary of the invention
[0004] In order to overcome the shortcomings of the prior art, the purpose of the present invention is to provide an adaptive task scheduling method for heterogeneous resources based on deep reinforcement learning, which takes into account the two optimization goals of minimizing the scheduling duration of tasks and maximizing the load balancing degree, while considering the type of tasks and the characteristics of nodes, to achieve the best task scheduling strategy for the DAG tasks input by users.
[0005] To achieve the above object, the present invention provides the following solutions:
[0006] An adaptive task scheduling method for heterogeneous resources based on deep reinforcement learning, comprising:
[0007] Obtaining the labeled DAG task input by the user, and extracting the DAG task to obtain DAG task information;
[0008] Obtain resource information of each node of the heterogeneous computing platform;
[0009] Establish and train a dual-agent deep reinforcement learning model based on the various resource information and DAG task information of the heterogeneous computing power platform;
[0010] In a simulated heterogeneous computing platform environment, the DAG tasks are scheduled using the output of the trained dual-agent deep reinforcement task scheduling learning model.
[0011] Preferably, the DAG task information includes: the computing overhead, network overhead and storage overhead of the task to be scheduled on different nodes and the running time of the task to be scheduled on different nodes.
[0012] Preferably, the resource information includes computing resources, storage resources and network resources.
[0013] Preferably, the DAG task is extracted to obtain DAG task information, including:
[0014] The matrix W = V × Q is used to represent the computational overhead of the tasks to be scheduled on different nodes; where Q is the number of nodes of the q heterogeneous computing platform, and V is the number of tasks to be scheduled;
[0015] Use matrix X = V × Q to represent the storage overhead of the scheduled tasks on different nodes;
[0016] The matrix Y = V × Q is used to represent the communication overhead of the scheduled tasks on different nodes;
[0017] Using ST(v i ,q j ) represents task v i At node q j The start time of the operation, ST(v i ,q j )=max{q j The earliest free time, AFT}, where AFT represents the task v i The end time of all predecessor tasks.
[0018] Preferably, obtaining resource information of each node of the heterogeneous computing power platform includes:
[0019] The computing resources of each node in the heterogeneous computing platform are represented by a four-tuple w res-i indicates; w res-i ={c core-i ,c speed-i ,g i ,g speed-i},i=1,...,q; where c core-i Indicates the number of CPU cores of the current node, c speed-i Indicates the CPU computing speed of the current node, g i Indicates the size of the current node GPU, g speed-i Indicates the current GPU computing speed;
[0020] The storage resources of each node in the heterogeneous computing platform are represented by a triple x res-i Indicates; x res-i ={m i ,d i ,rw i},i=1,...,q; where m i Indicates the memory size of the current node, d i Indicates the disk size of the current node, rw i Indicates the read and write speed of the current node;
[0021] The network resources of each node in the heterogeneous computing platform are represented by a triplet y res-i indicates; y res-i = {b i ,l i ,tr i},i=1,...,q; where b i Indicates the network bandwidth of the current node, l i Indicates the network delay of the current node, tr i Indicates the data transmission rate of the current node.
[0022] Preferably, obtaining resource information of each node of the heterogeneous computing power platform includes:
[0023] Scheduling time is optimized to minimize makespan; where makespan = FT(v n )-ST(v0), where FT(v n ) is the end time of the terminal task, and ST(v0) is the start time of the starting task;
[0024] Calculate the load balance degree σ of resources respectively w , the load balance of storage resourcesσ x and the load balancing degree of network resources σ y ;in σ y =σ b is the standard deviation of bandwidth usage, σ cpu is the standard deviation of CPU utilization, σ gpu is the standard deviation of GPU utilization, σ m is the standard deviation of memory usage, σ d is the standard deviation of cache usage.
[0025] Preferably, the steps of establishing a dual-agent deep reinforcement learning model include:
[0026] Construct a task agent; the optimization goal of the task agent is to minimize the scheduling time; each task agent is composed of its own actor network and critic network; the expression of the actor network state input S is: S = {n, ST (v i ,q j)|j=1,...,q}; where n represents the number of currently executable tasks, q is q heterogeneous computing nodes, and j is the number of heterogeneous computing nodes; the reward function R for each time step t The calculation formula is
[0027] where v i is the action output by the task agent, i.e. the task to be scheduled, q q The action output by the node agent, i.e. the node number;
[0028] Construct a node agent; the node agent is used to start different resource agents according to the action network output by the task agent. There are three options, including: computing resource agent, storage resource agent and network resource agent; the task label is 0, the computing resource agent is started, the task label is 1, the storage resource agent is started, and the task label is 2, the network resource agent is started;
[0029] The optimization goal of the computing resource agent is to minimize the computing resource σ w , the expression of actor network state input S is: S = {w res-i ,w ij |j=1,...,p}, the reward function for each time step is: in It is task v i The standard deviation of the computing resource utilization of the node after scheduling is completed. It is task v i-1 The standard deviation of the computing resource utilization of the node after scheduling is completed;
[0030] The optimization goal of the storage resource agent is to minimize σ x , the expression of actor network state input S is: S = {x res-i ,x ij |j=1,...,p}, the reward function for each time step is: in It is task v i The standard deviation of the node's storage resource utilization after scheduling is completed. It is task v i-1 The standard deviation of the node’s storage resource utilization after scheduling is completed;
[0031] The optimization goal of the network resource agent is to minimize σ y , actor network state input S = {y res-i ,y ij |j=1,...,p}, the reward function for each time step is: in It is task v i After scheduling is completed, the standard deviation of the node's network resource utilization is It is task v i-1 The standard deviation of the network resource utilization of nodes after scheduling is completed;
[0032] Preferably, the steps of training the dual-agent deep reinforcement learning model include:
[0033] The MAAC algorithm is used to update and train the established dual-agent deep reinforcement learning model to obtain an adaptive dual-agent deep reinforcement learning model.
[0034] The attention mechanism is used to model the interaction between the task agent and the node agent, update their respective actor networks and critic networks, and use the joint loss function to update the critic network separately. The update formula is: in,
[0035] is the value function, o is the state observation set of the agent, a is the action set of the agent, D is the experience replay pool, Given the local state o observed i Select action a i The probability of θ is the network parameter, r i is the immediate reward, γ is the discount factor, and α is the attention weight;
[0036] When calculating the Q value, the attention mechanism is used to consider the relationship between the two agents at the same time, and the state observation and action information of the two agents i and j are used as the input of the critic network. The formula is: Among them, f is realized by deep neural network, o is the state observation set of the two agents, a is the action set of the two agents, and g i is the embedding operation of the MLP layer, x i is the state observation and action information of agent j, the formula is x i =αv j =αh(Vg j (o j ,a j )); where α is the attention weight of agent i to agent j, v j is the embedding of the state and action of agent j, that is, first use g j The observation state and action information are embedded and encoded, then linearly transformed using the linear matrix V, and finally a nonlinear operation h is performed; the attention weights α of the two agents are expressed by similarity and calculated using the Euclidean distance. Where X=(x1,x2,...,x n )=(o1,a1) represents the set of state observations and actions of the task agent, o1 is the state observation vector of the task agent, a1 is the action output of the task agent, Y=(y1,y2,...,y n )=(o2,a2) represents the set of state observations and actions of the node agent, o2 is the state observation vector of the node agent, and a2 is the action output of the node agent.
[0037] Preferably, the strategies of the actor networks of the two agents are updated by stochastic policy gradients; the update formula of the stochastic policy gradients is: Among them, b(s,a \i ) using a counterfactual baseline for computing the advantage function, is the value function, o is the state observation set of the agent, a is the action set of the agent, D is the experience replay pool, Given the local state o it observes i Select action a i probability.
[0038] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:
[0039] The present invention provides an adaptive task scheduling method for heterogeneous resources based on deep reinforcement learning, including: obtaining a labeled DAG task input by a user, and extracting the DAG task to obtain DAG task information; obtaining resource information of each node of the heterogeneous computing power platform; establishing and training a dual-agent deep reinforcement learning model according to various types of resource information and DAG task information of the heterogeneous computing power platform; scheduling the DAG task using the output of the trained dual-agent deep reinforcement task scheduling learning model in a simulated heterogeneous computing power platform environment. The present invention. The present invention fully considers the resource demand preference of the task and the resource advantage of the node, and can match the corresponding node according to the task demand to improve the scheduling efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0041] Figure 1 A flow chart of a method provided by an embodiment of the present invention;
[0042] Figure 2A schematic diagram of a technical route provided for an embodiment of the present invention;
[0043] Figure 3 Schematic diagram of the dual-agent deep reinforcement learning model provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0044] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0045] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.
[0046] Figure 1 A flow chart of a method provided by an embodiment of the present invention, such as Figure 1 As shown, the present invention provides an adaptive task scheduling method for heterogeneous resources based on deep reinforcement learning, comprising:
[0047] Step 100: Obtain the DAG task with a label input by the user, and extract the DAG task to obtain DAG task information;
[0048] Step 200: Obtain resource information of each node of the heterogeneous computing platform;
[0049] Step 300: Establish and train a dual-agent deep reinforcement learning model according to various resource information and DAG task information of the heterogeneous computing power platform;
[0050] Step 400: In a simulated heterogeneous computing platform environment, the DAG task is scheduled using the output of the trained dual-agent deep reinforcement task scheduling learning model.
[0051] like Figure 2 As shown, the specific implementation method of this method includes the following parts:
[0052] Step 1: Get the labeled DAG task input by the user and use a tuple G = (V, E) to represent the DAG task, where the node V = {v1, v2, v3, ..., v n} is a task to be scheduled, edge E = {e1, e2, e3, ..., e n} indicates the order of tasks. ready ={v1,v2,...v n} represents schedulable tasks, that is, the predecessor tasks of these tasks have all been scheduled. For each task v i , use v before-i ={v1,v2,...,v n} represents its predecessor task. In addition, the computational cost w of all tasks on different nodes is obtained ij , storage overhead x ij and network overhead ij and the start time of the task ST(v i ,q j ).
[0053] Specifically, in step 1 of this embodiment, information is extracted from the DAG task input by the user to obtain the task overhead, specifically in the following manner:
[0054] The matrix W = V × Q represents the computational overhead of the scheduled tasks on different nodes, where Q is the number of nodes on the heterogeneous computing platform, and w ij For task v i At node q j The computational overhead.
[0055] The matrix X = V × Q represents the storage overhead of the scheduled tasks on different nodes, where Q is the number of nodes in the heterogeneous computing platform, and x ij For task v i At node q j storage overhead.
[0056] The matrix Y = V × Q represents the communication overhead of the scheduled tasks on different nodes, where Q is the number of nodes of the q heterogeneous computing platforms, and y ij For task v i At node q j communication overhead.
[0057] Use ST(v i ,q j ) indicates that the task is in v i At the start time of node qj, ST(v i ,q j )=max{q j The earliest free time, AFT}, where AFT represents the task v i The end time of all predecessor tasks.
[0058] Step 2: Get all node information of the heterogeneous computing platform, including: computing resources w res-i , storage resources x res-i and network resources res-i The computing resources include the number of CPU cores c core-i 、cpu calculation speed cspeed-i ,gpu size g i and gpu computing speed g speed-i ; Storage resources include memory size m i , disk size d i Indicates the disk size and read / write speed of the current node. i .
[0059] Specifically, in step 2 of this embodiment, the resource information of the heterogeneous computing power platform is obtained in the following manner:
[0060] The computing resources of each node in the heterogeneous computing platform are represented by a four-tuple w res-i It means that w res-i ={c core-i ,c speed-i ,g i ,g speed-i},i=1,...,q,where c core-i Indicates the number of CPU cores of the current node, c speed-i Indicates the CPU computing speed of the current node, g i Indicates the size of the current node GPU, g speed-i Indicates the current GPU computing speed.
[0061] The storage resources of each node in the heterogeneous computing platform are represented by a triple x res-i Indicates that x res-i ={m i ,d i ,rw i},i=1,...,q, where m i Indicates the memory size of the current node, d i Indicates the disk size of the current node, rw i Indicates the read and write speed of the current node.
[0062] The network resources of each node in the heterogeneous computing platform are represented by a triplet y res-i Indicates that y res-i = {b i ,l i ,tr i},i=1,...,q,where b i Indicates the network bandwidth of the current node, l i Indicates the network delay of the current node, tr i Indicates the data transmission rate of the current node.
[0063] Furthermore, in step 3, when constructing the dual-agent deep reinforcement learning model, the dual-objective optimization of minimizing scheduling time and maximizing load balancing is considered at the same time. The specific method is as follows:
[0064] Scheduling time is optimized to minimize makespan, makespan = FT(v n )-ST(v0), where FT(v n ) is the end time of the endpoint task, and ST(v0) is the start time of the origin task.
[0065] Load balancing includes computing resources, storage resources, and network resources, and is expressed as w , σ x , σ y They represent the balance of computing resources, storage resources and network resources respectively, which is to minimize the standard deviation. σ y =σ b is the standard deviation of bandwidth usage.
[0066] Step 3: Based on the acquired DAG task information and the node information of the heterogeneous computing platform, a dual-agent deep reinforcement learning model is constructed. The model of the dual-agent deep reinforcement learning model is as follows: Figure 3 shown.
[0067] Each agent is composed of its own actor and critic network, which are independent of each other. In this embodiment, the task agent is first constructed to minimize the scheduling time as the optimization goal, with S = {n, ST (v i ,q j )|j=1,...,q} as the actor network state input, n represents the number of currently executable tasks, and the following formula is used as the reward function
[0068]
[0069] where v i is the action output by the task agent, i.e. the task to be scheduled, q q It is the action output by the node agent, that is, the node number.
[0070] Next, we construct a node agent. The node agent starts different resource agents according to the action output by the task agent, including three types: computing resource agent, storage resource agent, and network resource agent. When the task label is 0, the computing resource agent is started. When the task label is 1, the storage resource agent is started. When the task label is 2, the network resource agent is started. During training, different agents are started according to the action task type output by the task agent, and training updates are performed. Among them, the computing resource agent takes minimizing the standard deviation of computing resource utilization as the optimization goal, with S = {w res-i ,w ij |j=1,...,p} is used as the state input of the actor network, and the following formula is used as the reward function:
[0071]
[0072] in It is task v i The standard deviation of the computing resource utilization of the node after scheduling is completed. It is task v i-1 The standard deviation of the computing resource utilization of the node after scheduling is completed.
[0073] The storage resource agent takes minimizing the standard deviation of storage resource utilization as the optimization goal, with S = {x res-i ,x ij |j=1,...,p} is used as the state input of the actor network, and the following formula is used as the reward function:
[0074]
[0075] in It is task v i The standard deviation of the node's storage resource utilization after scheduling is completed. It is task v i-1 The standard deviation of the node's storage resource utilization after scheduling is completed.
[0076] The network resource agent takes minimizing the standard deviation of network resource utilization as the optimization goal, with S = {y res-i ,y ij |j=1,...,p} is used as the state input of the actor network, and the following formula is used as the reward function:
[0077]
[0078] Next, after the agent is built, the model training begins. First, the MAAC algorithm is used to update the established dual-agent deep reinforcement learning model to obtain an adaptive dual-agent deep reinforcement learning model. ready Output an action a1, which is the task v to be scheduled i , the node agent according to v i The label selects the corresponding agent, which outputs an action a2, i.e., the scheduled node. Next, the state observation s1 and output action a1 of the task agent and the state observation s2 and output action a2 of the corresponding node agent are input into their respective critic networks, and the reward of each time step is calculated. At the same time, the weight α between the two agents is obtained by using the attention mechanism, which is calculated as the Euclidean distance. Since the critic networks of the two agents use shared parameters, a joint loss function is used to update the critics separately, as shown in the following formula:
[0079] in
[0080] in
[0081] Among them, o is the state observation set of the two agents, a is the action set of the two agents, and g i is the embedding operation of the MLP layer, x i is the state observation and action information of agent j, and the formula is as follows:
[0082] x i =αv j =αh(Vg j (o j ,a j ))
[0083] Among them, α is the attention weight of agent i to agent j, v j is the embedding of the state and action of agent j, that is, first use g j The observation state and action information are embedded and encoded, then linearly transformed using the linear matrix V, and finally a nonlinear operation h is performed.
[0084] Furthermore, the attention weights α of the two agents are expressed as similarities, and the calculation method is the Euclidean distance,
[0085]
[0086] Where X=(x1,x2,...,x n )=(o1,a1) represents the set of state observations and actions of the task agent, o1 is the state observation vector of the task agent, a1 is the action output of the task agent, Y=(y1,y2,...,y n )=(o2,a2) represents the set of state observations and actions of the node agent, o2 is the state observation vector of the node agent, and a2 is the action output of the node agent.
[0087] Secondly, the actor policies of the two agents are updated using the following formula using stochastic policy gradients:
[0088]
[0089] Step 4: Set the appropriate number of iterations and time steps to obtain the trained dual-agent deep reinforcement learning model, and use the trained model to schedule the DAG tasks input by the user to achieve dual-objective optimization of scheduling time and load balancing.
[0090] In summary, the present invention solves the complexity of computing power types and load imbalance in heterogeneous computing power platforms when scheduling user DAG tasks, and optimizes scheduling time and load balancing as dual-objective optimization to achieve the best scheduling strategy. The present invention is based on deep reinforcement learning, actor-critic network as the basis, and uses the MAAC algorithm to establish a task scheduling model with adaptive capabilities. At the same time, the preference of the task is considered, and the corresponding intelligent agent is started according to the task label. In addition, the influence between intelligent agents is considered, and the network is updated using the attention mechanism, which can improve the scheduling level of DAG tasks.
[0091] The beneficial effects of the present invention are as follows:
[0092] (1) The present invention fully considers the resource demand preference of the task and the resource advantage of the node, and can match the corresponding node according to the task demand, thereby improving the scheduling efficiency.
[0093] (2) The present invention has good adaptive performance, can adapt to dynamic environmental changes and make timely adjustments, and output appropriate scheduling strategies.
[0094] (3) The present invention considers dual optimization objectives when modeling, combining load balance while optimizing scheduling time. At the same time, it uses the attention mechanism to comprehensively consider the states and actions between intelligent agents, thereby achieving high-performance task scheduling.
[0095] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.
[0096] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of the present invention. At the same time, for those skilled in the art, according to the idea of the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.
Claims
1. An adaptive task scheduling method for heterogeneous resources based on deep reinforcement learning, characterized in that: include: Obtaining the labeled DAG task input by the user, and extracting the DAG task to obtain DAG task information; Obtain resource information of each node of the heterogeneous computing platform; Establish and train a dual-agent deep reinforcement learning model based on the various resource information and DAG task information of the heterogeneous computing power platform; In a simulated heterogeneous computing platform environment, the DAG tasks are scheduled using the output of the trained dual-agent deep reinforcement task scheduling learning model.
2. The method for adaptive task scheduling based on deep reinforcement learning for heterogeneous resources according to claim 1, characterized in that: The DAG task information includes: the computing overhead, network overhead and storage overhead of the task to be scheduled on different nodes and the running time of the task to be scheduled on different nodes.
3. The method for adaptive task scheduling based on deep reinforcement learning for heterogeneous resources according to claim 1, characterized in that: The resource information includes computing resources, storage resources and network resources.
4. The method for adaptive task scheduling based on deep reinforcement learning for heterogeneous resources according to claim 2 is characterized in that: Extract the DAG task to obtain DAG task information, including: The matrix W = V × Q is used to represent the computational overhead of the tasks to be scheduled on different nodes; where Q is the number of nodes of the q heterogeneous computing platforms, and V is the number of tasks to be scheduled; Use matrix X = V × Q to represent the storage overhead of the scheduled tasks on different nodes; The matrix Y = V × Q is used to represent the communication overhead of the scheduled tasks on different nodes; Using ST(v i ,q j ) represents task v i At the start of the running time of node qj, ST(v i ,q j )=max{q j The earliest free time, AFT}, where AFT represents the task v i The end time of all predecessor tasks.
5. The method for adaptive task scheduling based on deep reinforcement learning for heterogeneous resources according to claim 3 is characterized in that: Obtain resource information of each node in the heterogeneous computing platform, including: The computing resources of each node in the heterogeneous computing platform are represented by a four-tuple w res-i indicates; w res-i ={c core-i ,c speed-i ,g i ,g speed-i },i=1,...,q; where c core-i Indicates the number of CPU cores of the current node, c speed-i Indicates the CPU computing speed of the current node, g i Indicates the size of the current node GPU, g speed-i Indicates the current GPU computing speed; The storage resources of each node in the heterogeneous computing platform are represented by a triple x res-i Indicates; x res-i ={m i ,d i ,rw i },i=1,...,q; where m i Indicates the memory size of the current node, d i Indicates the disk size of the current node, rw i Indicates the read and write speed of the current node; The network resources of each node in the heterogeneous computing platform are represented by a triplet y res-i indicates; y res-i = {b i ,l i ,tr i },i=1,...,q; where b i Indicates the network bandwidth of the current node, l i Indicates the network delay of the current node, tr i Indicates the data transmission rate of the current node.
6. The adaptive task scheduling method for heterogeneous resources based on deep reinforcement learning according to claim 3 is characterized in that: Obtain resource information of each node in the heterogeneous computing platform, including: Scheduling time is optimized to minimize makespan; where makespan = FT(v n )-ST(v0), where FT(v n ) is the end time of the terminal task, and ST(v0) is the start time of the starting task; Calculate the load balance degree σ of resources respectively w , the load balance of storage resourcesσ x and the load balancing degree of network resources σ y ;in σ y =σ b is the standard deviation of bandwidth usage, σ cpu is the standard deviation of CPU utilization, σ gpu is the standard deviation of GPU utilization, σ m is the standard deviation of memory usage, σ d is the standard deviation of cache usage.
7. The method for adaptive task scheduling based on deep reinforcement learning for heterogeneous resources according to claim 5, characterized in that: The steps to build a dual-agent deep reinforcement learning model include: Construct a task agent; the optimization goal of the task agent is to minimize the scheduling time; each task agent is composed of its own actor network and critic network; the expression of the actor network state input S is: S = {n, ST (v i ,q j )|j=1,...,q}; where n represents the number of currently executable tasks, q is q heterogeneous computing nodes, and j is the number of heterogeneous computing nodes; the reward function R for each time step t The calculation formula is where v i is the action output by the task agent, i.e. the task to be scheduled, q q The action output by the node agent, i.e. the node number; Construct a node agent; the node agent is used to start different resource agents according to the action network output by the task agent. There are three options, including: computing resource agent, storage resource agent and network resource agent; the task label is 0, the computing resource agent is started, the task label is 1, the storage resource agent is started, and the task label is 2, the network resource agent is started; The optimization goal of the computing resource agent is to minimize the computing resource σ w , the expression of actor network state input S is: S = {w res-i ,w ij |j=1,...,p}, the reward function for each time step is: in It is task v i After scheduling is completed, the standard deviation of the node's computing resource utilization, σ wi-1 It is task v i-1 The standard deviation of the computing resource utilization of the node after scheduling is completed; The optimization goal of the storage resource agent is to minimize σ x , the expression of actor network state input S is: S = {x res-i ,x ij |j=1,...,p}, the reward function for each time step is: in It is task v i The standard deviation of the node's storage resource utilization after scheduling is completed. It is task v i-1 The standard deviation of the node’s storage resource utilization after scheduling is completed; The optimization goal of the network resource agent is to minimize σ y , actor network state input S = {y res-i ,y ij |j=1,...,p}, the reward function for each time step is: in It is task v i After scheduling is completed, the standard deviation of the node's network resource utilization is It is task v i-1 The standard deviation of the network resource utilization of the nodes after scheduling is completed.
8. The adaptive task scheduling method for heterogeneous resources based on deep reinforcement learning according to claim 7 is characterized in that: The steps to train a dual-agent deep reinforcement learning model include: The MAAC algorithm is used to update and train the established dual-agent deep reinforcement learning model to obtain an adaptive dual-agent deep reinforcement learning model. The attention mechanism is used to model the interaction between the task agent and the node agent, update their respective actor networks and critic networks, and use the joint loss function to update the critic network separately. The update formula is: in, is the value function, o is the state observation set of the agent, a is the action set of the agent, D is the experience replay pool, Given the local state o observed i Select action a i The probability of θ is the network parameter, r i is the immediate reward, γ is the discount factor, and α is the attention weight; In calculating the value function When the value is , the attention mechanism is used to consider the relationship between the two agents at the same time, and the state observation and action information of the two agents i and j are used as the input of the critic network. The formula is Among them, f is realized by deep neural network, g i is the embedding operation of the MLP layer, x i is the state observation and action information of agent j, the formula is x i =αv j =αh(Vg j (o j ,a j )); where α is the attention weight of agent i to agent j, v j is the embedding of the state and action of agent j, that is, first use g j The observation state and action information are embedded and encoded, then linearly transformed using the linear matrix V, and finally a nonlinear operation h is performed; the attention weights α of the two agents are expressed by similarity and calculated using the Euclidean distance. Where X=(x1,x2,...,x n )=(o1,a1) represents the set of state observations and actions of the task agent, o1 is the state observation vector of the task agent, a1 is the action output of the task agent, Y=(y1,y2,...,y n )=(o2,a2) represents the set of state observations and actions of the node agent, o2 is the state observation vector of the node agent, and a2 is the action output of the node agent.
9. The method for adaptive task scheduling based on deep reinforcement learning for heterogeneous resources according to claim 8, characterized in that: The strategies of the actor networks of the two agents are updated by stochastic policy gradients; the update formula of the stochastic policy gradient is: Among them, b(s,a \i ) using a counterfactual baseline for computing the advantage function.
Citation Information
Cited By
Heterogeneous resource intelligent scheduling method and system of large model training and pushing all-in-one machine
CN119781991A
Heterogeneous resource intelligent scheduling method and system for large-scale model training and push-pull integrated machine
CN119781991B
A method and device for collaborative optimization of energy and computing power based on reinforcement learning
CN119784034A
