Calculation task scheduling method and device, terminal equipment and storage medium

Through the multi-agent reinforcement learning model, the task path planning scheme is generated on the drone, which solves the problem of large delay in the offload of fixed base stations and improves the efficiency of computing tasks of power equipment in remote or harsh environments.

CN118939398APending Publication Date: 2024-11-12GUANGDONG POWER GRID CO LTD +1
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202410989663.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-23
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

The existing computing task scheduling methods will generate large delays when unloading computing tasks to fixed base stations, resulting in low processing efficiency, especially in unmanned or harsh environments, which are difficult to efficiently handle computing-intensive tasks generated by power equipment.

Method used

Using multi-agent reinforcement learning model, by initializing agents and behavioral networks, iteratively generates the probability distribution of computing node selection and link weights based on network configuration data, and determines the task path planning scheme, so that the drone can handle both computing tasks and As a repeater, offload tasks to a fixed base station to reduce delay.

Benefits of technology

It effectively reduces the delay of calculation task offloading and improves processing efficiency, especially in remote or harsh environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118939398A_ABST
    Figure CN118939398A_ABST
Patent Text Reader

Abstract

The invention discloses a calculation task scheduling method and device, terminal equipment and a storage medium, and the method comprises the steps: initializing the model parameters of a multi-agent reinforcement learning model, and creating an agent for each unmanned plane in a target region; setting an initial state according to the network configuration data, and starting to iterate the multi-agent reinforcement learning model; utilizing a behavior network corresponding to the intelligent agent to output probability distribution selected by calculation nodes, and generating a first action array based on the probability distribution selected by the calculation nodes; the probability distribution of the link weight is output by using the behavior network, and a second action array is generated based on the probability distribution of the link weight; a comprehensive action is formed based on the first action array and the second action array, a task path planning scheme is determined according to the comprehensive action, and the path planning scheme comprises a destination node and a forwarding path selected by a calculation task. According to the method, the time delay of unloading the calculation task can be effectively reduced, so that the processing efficiency of the calculation task can be effectively improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of computing task scheduling, and in particular to a computing task scheduling method, device, terminal equipment and storage medium. Background Art

[0002] As a comprehensive power business network transmission platform, the ubiquitous power data communication network is the basis for the interconnection of people, machines, objects and applications in the power company system. On the basis of the power Internet of Things, it fully applies modern information technologies such as mobile Internet, artificial intelligence, IPv6+, and advanced network communication technologies around all aspects of the power system to realize the interconnection of all aspects of the power system. The intelligent service system has the characteristics of comprehensive state perception, efficient information processing, and convenient and flexible application, which supports power production operation and maintenance and business management decisions. my country's electricity demand is getting higher and higher, and the business in the power data network is also increasing. The number of power Internet of Things terminals has increased sharply, and the amount of network transmission data has exploded. Traditional cloud computing can no longer meet all the needs, so edge computing has received more and more attention. Edge computing aims to solve the data processing, storage and transmission problems encountered by cloud computing. It can use edge gateways for distributed deployment, data collection, processing, protocol conversion and data analysis nearby, which greatly alleviates the pressure on network transmission and data centers. Combined with IPv6+ network innovation technology, it ensures flexible scheduling and low-latency interaction of terminal services on the edge side, speeds up processing, and improves service response performance and quality.

[0003] Typically, computing tasks generated by user devices can be offloaded to fixed edge servers for computation. However, some power grid equipment, such as thermal power, hydropower, nuclear power, photovoltaic power generation, wind power and other energy systems and equipment, may be set up in unmanned areas or in relatively harsh environments, and power equipment will still generate many computing-intensive tasks. In this scenario, since ground computing servers are usually deployed in fixed base stations (BS), which are far away from user devices that generate computing tasks, existing computing task scheduling methods will generate large delays when offloading computing tasks to fixed base stations, resulting in low processing efficiency of computing tasks. Summary of the invention

[0004] The present invention provides a computing task scheduling method, device, terminal equipment and storage medium to solve the technical problem that some computing task scheduling methods will generate a large delay when unloading computing tasks to a fixed base station, resulting in low processing efficiency of computing tasks.

[0005] The present invention provides a computing task scheduling method, comprising:

[0006] Initializing model parameters of a multi-agent reinforcement learning model; creating an agent for each UAV in the target area, wherein each agent in the multi-agent reinforcement learning model corresponds to a behavior network, and all agents share an evaluation network;

[0007] Setting an initial state according to network configuration data and starting to iterate the multi-agent reinforcement learning model; the network configuration data includes network topology, task list, computing node list and network link status;

[0008] The behavior network corresponding to the agent outputs a probability distribution of computing node selection, and generates a first action array based on the probability distribution of computing node selection; the behavior network corresponding to the agent outputs a probability distribution of link weights, and generates a second action array based on the probability distribution of link weights;

[0009] A comprehensive action is formed based on the first action array and the second action array, and a task path planning scheme is determined according to the comprehensive action. The path planning scheme includes a destination node and a forwarding path selected by the calculation task.

[0010] Furthermore, after forming a comprehensive action based on the first action array and the second action array and determining a task path planning scheme according to the comprehensive action, the method further includes:

[0011] A node network is formed based on drone nodes, network performance statistics are calculated according to network status data in the node network, and a shared reward is calculated according to a weight index of the network performance statistics; the network status data includes link utilization and transmission delay data, and the network performance statistics include link utilization variance, maximum waiting operation time variance and average task completion time;

[0012] The next state is set with the current network configuration data, and the initial state, the comprehensive action, the shared reward and the next state are stored in the experience replay pool until the experience replay pool is full;

[0013] The evaluation network and the behavior network are updated based on the storage records of the experience replay pool, the current iteration is ended, and the next iteration is performed. When the current iteration number reaches the set maximum iteration number, the iteration is stopped.

[0014] Furthermore, the expression of the first action array is as follows:

[0015]

[0016] Among them, a c is the first action array, λ kIndicates the computing nodes to which task k is assigned in the task list are numbered 1 to n, and the size of the first action array is the length of the task list;

[0017] The expression of the second action array is as follows:

[0018] a n =[w1,w2,…,w |ε| ]

[0019] Among them, a n is the second action array, and w is the weight of each link.

[0020] Furthermore, the updating of the evaluation network and the behavior network based on the storage records of the experience replay pool includes:

[0021] Input the next state and all states in the experience replay pool into the evaluation network, and calculate the advantage function of the current strategy;

[0022] Calculating the loss function of the evaluation network according to the advantage function of the current strategy;

[0023] Based on the loss function of the evaluation network, the network parameters of the evaluation network are updated through back propagation to obtain an updated evaluation network.

[0024] Furthermore, the expression of the loss function of the evaluation network is as follows:

[0025]

[0026] Where c_loss is the loss function of the evaluation network. is the advantage function, is the square of the advantage function

[0027] Furthermore, the updating of the evaluation network and the behavior network based on the storage records of the experience replay pool includes:

[0028] Determine the old behavior network and the new behavior network corresponding to the behavior network of each agent;

[0029] Inputting all states in the experience replay pool into the old behavior network and the new behavior network respectively, to obtain a first probability distribution and a second probability distribution respectively;

[0030] Obtaining first action probabilities and second action probabilities corresponding to the first probability distribution and the second probability distribution for all comprehensive actions in the experience replay pool;

[0031] Dividing the second action probability by the first action probability to obtain an importance weight;

[0032] Calculating the loss function of the behavior network according to the importance weight, the advantage function of the current strategy and the learning rate;

[0033] Based on the loss function of the behavior network, the network parameters of the behavior network are updated through back propagation to obtain an updated behavior network.

[0034] Furthermore, the expression of the loss function of the behavior network is as follows:

[0035]

[0036] Where a_loss is the loss function of the network, ratio is the importance weight value, ∈ is the learning rate, and clip(ratio,1-∈,1+∈) means clipping the importance weights that exceed the range of (1-∈,1+∈).

[0037] The present invention also provides a computing task scheduling device, comprising:

[0038] An agent creation module is used to initialize the model parameters of the multi-agent reinforcement learning model and create an agent for each UAV in the target area. Each agent in the multi-agent reinforcement learning model corresponds to a behavior network, and all agents share an evaluation network.

[0039] An iteration start module, used to set a next state according to network configuration data and start iterating the multi-agent reinforcement learning model; the network configuration data includes network topology, task list, computing node list and network link status;

[0040] An action array generation module, configured to use the behavior network corresponding to the agent to output the probability distribution of computing node selection, and generate a first action array based on the probability distribution of computing node selection; use the behavior network corresponding to the agent to output the probability distribution of link weights, and generate a second action array based on the probability distribution of link weights;

[0041] The task path planning scheme determination module is used to form a comprehensive action based on the first action array and the second action array, and determine the task path planning scheme according to the comprehensive action. The path planning scheme includes calculating the destination node and forwarding path selected by the task.

[0042] The present invention also provides a terminal device, comprising: a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, and the processor implements the computing task scheduling method as described above when executing the computer program.

[0043] The present invention also provides a computer-readable storage medium, which includes a stored computer program; wherein, when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the computing task scheduling method as described above.

[0044] The present invention utilizes the behavior network corresponding to the intelligent agent to output the probability distribution of computing node selection, and generates a first action array based on the probability distribution of computing node selection, utilizes the behavior network corresponding to the intelligent agent to output the probability distribution of link weights, and generates a second action array based on the probability distribution of link weights, and composes a comprehensive action based on the first action array and the second action array, determines the task path planning scheme according to the comprehensive action, and uses the computing node selection and link weights as the comprehensive action, so that the drone can be used as a computing power node to process computing tasks, and can also be used as a repeater to offload computing tasks to a fixed base station on the ground, thereby effectively reducing the delay of unloading computing tasks, and then effectively improving the processing efficiency of computing tasks.

[0045] Furthermore, the embodiments of the present invention can select a suitable drone to process corresponding computing tasks, and after processing the computing tasks, can select the optimal transmission path to transmit back to the user node, thereby further improving the processing efficiency of the computing tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a flowchart of a computing task scheduling method provided by an embodiment of the present invention;

[0047] Figure 2 It is a schematic diagram of the task scheduling of the UAV-assisted power data network provided by an embodiment of the present invention;

[0048] Figure 3 is a schematic diagram of a computing data transmission path provided by an embodiment of the present invention;

[0049] Figure 4 Another flowchart of the computing task scheduling method provided by the embodiment of the present invention is

[0050] Figure 5 It is a structural diagram of a computing task scheduling device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0051] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0052] In the description of this application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the feature. In the description of this application, unless otherwise specified, "plurality" means two or more.

[0053] In the description of this application, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0054] The embodiments of the present invention can be applied to electric power data computing networks to implement scheduling of computing tasks generated by user terminals.

[0055] See also Figure 1 The present invention provides a computing task scheduling method, comprising:

[0056] S1. Initialize the model parameters of the multi-agent reinforcement learning model (MAPPO model), create an agent for each drone in the target area, each agent in the multi-agent reinforcement learning model corresponds to a behavior network (Actor network), and all agents share an evaluation network (Critic network);

[0057] In the embodiment of the present invention, the current number of iterations is set to 0, the maximum number of iterations is set to i_max, an experience replay pool is set with a length of n, and the number of updates per round is set to u.

[0058] In this embodiment of the present invention, agent A n Used to enable the drone to act as an edge server for computing and forwarding.

[0059] S2. Set the initial state according to the network configuration data and start iterating the multi-agent reinforcement learning model; the network configuration data includes the network topology, task list, computing node list and network link status;

[0060] In the embodiment of the present invention, the initial state (V c ,T,G,N) is set as the network topology V c , a combination of task list T, computing node list G and network link status N.

[0061] S3, using the behavior network corresponding to the agent to output the probability distribution of the calculation node selection, and generating a first action array based on the probability distribution of the calculation node selection; using the behavior network corresponding to the agent to output the probability distribution of the link weight, and generating a second action array based on the probability distribution of the link weight;

[0062] In the embodiment of the present invention, step S3 is to start iteration, so that the value of i increases by 1. Each agent generates an action: Agent A of overall control of data forwarding c The Actor network output computes the probability distribution of node selection a c , and then generate the first action array by sampling from the probability distribution λ k Indicates the computing nodes 1 to n to which task k is assigned in the task list, and the array size is the length of the task list. c It is a multi-dimensional discrete action space, and the dimension is equal to the number of computing tasks*the number of computing nodes. Therefore, the embodiment of the present invention first groups the output layer of the neural network, and then uses softmax to normalize each group of outputs to the probability of occurrence of each dimensional discrete action.

[0063] Each drone is considered as an agent A n The Actor network outputs the probability distribution of link weights and generates the second action array represents the weight of each link. In the embodiment of the present invention, since the second action data a n It is a multidimensional continuous action space, and the multivariate normal distribution is used as the probability distribution type of the action. The output layer of the neural network is the mean and variance of the weight of each link, and the dimension is equal to the number of network links*2.

[0064] S4. Based on the first action array and the second action array, a comprehensive action is formed, and a task path planning scheme is determined according to the comprehensive action. The path planning scheme includes a destination node and a forwarding path selected by the calculation task.

[0065] In the embodiment of the present invention, after the calculation data reaches the drone node, the next action can be selected for the calculation task according to the calculation result of the intelligent agent on the drone. c The mapping of the calculated tasks to the drone nodes and the agent A on each drone n The given link weights form a comprehensive action Action = (a c ,a n ), the destination node (i.e., the destination node of the task data flow) and the forwarding path p selected for the computing task can be determined f , is applied to the network as a path planning scheme, so that computing tasks can be scheduled based on the task path planning scheme.

[0066] In an embodiment of the present invention, multiple unmanned aerial vehicles (UAVs) are equipped with computing servers as auxiliary equipment to help user devices perform computing offload. Among them, the unmanned aerial vehicle (UAV) can be used as a computing power node to process computing tasks, and can also be used as a repeater to offload computing tasks to a fixed base station (BS) on the ground. Multiple drones can patrol along a fixed route with a certain distance in front and behind, and provide computing services for user devices along the route. Faced with power network equipment in remote areas, or scenarios where ground networks are paralyzed after natural disasters, computing-intensive tasks generated by user devices can only be processed locally or offloaded to drones. Considering the offloading decision of computing tasks and the flight path of drones, embodiments of the present invention can select a suitable drone to process computing tasks. After processing the computing tasks, the drone selects the optimal transmission path to transmit back to the user node, thereby effectively reducing the transmission delay of computing task offloading, and thus effectively improving the efficiency of task processing.

[0067] See also Figure 2 , is a schematic diagram of a UAV-assisted power data network task scheduling provided by an embodiment of the present invention.

[0068] See also Figure 3 , is a schematic diagram of a computing data transmission path provided by an embodiment of the present invention. The computing tasks generated by the user equipment can be offloaded to the drone, or handed over to the ground fixed base station for calculation. The transmission path selection when the computing data is sent up and the data is returned will also greatly affect the overall task scheduling delay. The computing data of the embodiment of the present invention can be transmitted through the ground network or through the drone.

[0069] See also Figure 4 In one embodiment, a multi-agent reinforcement learning model is used to implement computing task scheduling. The relevant parameters of the multi-agent reinforcement learning model are as follows:

[0070] State Space:

[0071] Under the UAV-assisted grid computing task scheduling, the state space consists of the unfinished task queue, the remaining energy of the UAV, the channel state, and the generated computing task amount. That is, the state space of the agent n in a certain time slot is defined as:

[0072]

[0073] Among them, ρ n (t) represents the unfinished task queue, E UAV (t) represents the remaining energy of UAV, h mi (t) represents the channel state between terminal device n and computing node i, Φ n (t) represents the computing task generated by terminal device n in the current time slot. When t = 1, ρn (t) = 0, E UAV (t) = E max , Φ n (t)∈{1,2,…,M}.

[0074] Action Space:

[0075] Based on the current state n (t) and the experience of other agents, each agent independently selects actions to schedule computing tasks. In one embodiment, the action selected by the agent can be a binary task offloading decision, that is, And x nmi (t)∈{0,1}.

[0076] Reward function:

[0077] In one embodiment, the long-term weighted sum of the delay in calculating task completion and the energy consumption generated by the drone can be used as the goal to define a function to maximize the global reward. The formula is as follows:

[0078]

[0079] Each agent chooses an action As the output of the actor network, the critic network uses the value function to evaluate the action. The loss function of its value function is:

[0080]

[0081]

[0082] The policy gradient update formula of the actor network in this algorithm is:

[0083]

[0084] Online actor network θ μ and the online critic network θ Q The gradient update method is used for online real-time update, while the parameters of the target actor network and the target critic network are updated by soft update. The specific update formula is:

[0085]

[0086] In one embodiment, after forming a comprehensive action based on the first action array and the second action array and determining a task path planning scheme according to the comprehensive action, the method further includes:

[0087] S5. A node network is formed based on drone nodes, network performance statistics are calculated based on the network status data in the node network, and a shared reward is calculated based on the weight index of the network performance statistics; the network status data includes link utilization and transmission delay data, and the network performance statistics include link utilization variance, maximum waiting operation time variance and average task completion time;

[0088] In an embodiment of the present invention, a corresponding task data stream can be generated in the network composed of drone nodes to train the intelligent agent, network performance statistics can be calculated based on the network status data in the node network, and a shared reward Reward can be calculated based on the weight index of the network performance statistics.

[0089] S6. Set the next state with the current network configuration data, and store the initial state, comprehensive action, shared reward and next state in the experience replay pool until the experience replay pool is full;

[0090] In the embodiment of the present invention, the next state State′ of the agent is set to the combination of the current network topology, task list, computing node list and network link state (State, Action = (a c ,a n ), Reward, State′) are stored in the experience replay pool for reference of model iterative learning.

[0091] In the embodiment of the present invention, steps S3-S6 are repeated until the experience replay pool is full.

[0092] S7. Update the evaluation network and behavior network based on the storage records in the experience replay pool, end the current iteration, and proceed to the next iteration. When the current iteration number reaches the set maximum iteration number, stop the iteration.

[0093] After updating the evaluation network and the behavior network based on the storage records of the experience replay pool, the embodiment of the present invention repeatedly updates the evaluation network and the behavior network until a preset number of updates per round is reached to obtain the final evaluation network and behavior network of the current iteration.

[0094] In one embodiment, the expression of the first action array in step S3 is as follows:

[0095]

[0096] Among them, a c is the first action array, λ k Indicates the computing nodes to which task k is assigned in the task list are numbered 1 to n, and the size of the first action array is the length of the task list;

[0097] The expression of the second action array is as follows:

[0098] a n =[w1,w2,…,w |ε| ];

[0099] Among them, a n is the second action array, and w is the weight of each link.

[0100] In one embodiment, step S7, updating the evaluation network and the behavior network based on the storage records of the experience replay pool, includes:

[0101] S71, input the next state and all states in the experience replay pool into the evaluation network, and calculate the advantage function of the current strategy;

[0102] In the embodiment of the present invention, the latest state and all states in the experience replay pool are input into the evaluation network to obtain the V_values ​​of all states, and the advantage function is calculated based on the V_values.

[0103]

[0104] Among them, γλ is the attenuation factor, δ t′ is the advantage function, For shared rewards, V(s t ) is the output obtained by taking the global state as the input of the evaluation network.

[0105] S72, calculating the loss function of the evaluation network according to the advantage function of the current strategy;

[0106] In this embodiment of the present invention, the expression of the loss function of the evaluation network is as follows:

[0107]

[0108] Where c_loss is the loss function of the evaluation network. is the advantage function, is the square of the advantage function.

[0109] S73. Based on the loss function of the evaluation network, update the network parameters of the evaluation network through back propagation to obtain an updated evaluation network.

[0110] In one embodiment, step S7, updating the evaluation network and the behavior network based on the storage records of the experience replay pool, includes:

[0111] S701, determining the old behavior network and the new behavior network corresponding to the behavior network of each agent;

[0112] S702, input all states in the experience replay pool into the old behavior network (Actor-old network) and the new behavior network (Actor-new network), respectively, to obtain a first probability distribution dist1 and a second probability distribution dist2;

[0113] S703, obtaining the first action probability prob1 and the second action probability prob2 corresponding to the first probability distribution and the second probability distribution for all comprehensive actions in the experience replay pool;

[0114] S704, dividing the second action probability by the first action probability to obtain an importance weight;

[0115] S705, calculating the loss function of the behavior network according to the importance weight, the advantage function of the current strategy and the learning rate;

[0116] In the embodiment of the present invention, the expression of the loss function of the behavior network is as follows:

[0117]

[0118] Where a_loss is the loss function of the network, ratio is the importance weight value, ∈ is the learning rate (usually set to 0.2), and clip(ratio,1-∈,1+∈) means clipping the importance weights that exceed the range of (1-∈,1+∈).

[0119] S706 . Based on the loss function of the behavior network, update the network parameters of the behavior network through back propagation to obtain an updated behavior network.

[0120] The implementation of the embodiments of the present invention has the following beneficial effects:

[0121] The embodiment of the present invention utilizes the behavior network corresponding to the intelligent agent to output the probability distribution of computing node selection, and generates a first action array based on the probability distribution of computing node selection, utilizes the behavior network corresponding to the intelligent agent to output the probability distribution of link weights, and generates a second action array based on the probability distribution of link weights, and composes a comprehensive action based on the first action array and the second action array, determines the task path planning scheme according to the comprehensive action, and uses the computing node selection and link weights as the comprehensive action, so that the drone can be used as a computing power node to process computing tasks, and can also be used as a repeater to offload computing tasks to a fixed base station on the ground, thereby effectively reducing the delay of unloading computing tasks, and then effectively improving the processing efficiency of computing tasks.

[0122] Furthermore, the embodiments of the present invention can select a suitable drone to process corresponding computing tasks, and after processing the computing tasks, can select the optimal transmission path to transmit back to the user node, thereby further improving the processing efficiency of the computing tasks.

[0123] See also Figure 5 Based on the same technical concept as the above embodiment, the present invention also provides a computing task scheduling device, including:

[0124] The agent creation module 10 is used to initialize the model parameters of the multi-agent reinforcement learning model, create an agent for each UAV in the target area, each agent in the multi-agent reinforcement learning model corresponds to a behavior network, and all agents share an evaluation network;

[0125] An iteration start module 20 is used to set the next state according to the network configuration data and start iterating the multi-agent reinforcement learning model; the network configuration data includes the network topology, the task list, the computing node list and the network link state;

[0126] The action array generation module 30 is used to use the behavior network corresponding to the intelligent agent to output the probability distribution of the calculation node selection, and generate a first action array based on the probability distribution of the calculation node selection; use the behavior network corresponding to the intelligent agent to output the probability distribution of the link weight, and generate a second action array based on the probability distribution of the link weight;

[0127] The task path planning scheme determination module 40 is used to form a comprehensive action based on the first action array and the second action array, and determine the task path planning scheme according to the comprehensive action. The path planning scheme includes calculating the destination node and forwarding path selected by the task.

[0128] In one embodiment, the computing task scheduling device further includes:

[0129] A shared reward calculation module is used to form a node network based on drone nodes, calculate network performance statistics according to network status data in the node network, and calculate shared rewards according to the weight index of network performance statistics; network status data includes link utilization and transmission delay data, and network performance statistics include link utilization variance, maximum waiting operation time variance and average task completion time;

[0130] The data storage module is used to set the next state with the current network configuration data, and store the initial state, comprehensive action, shared reward and next state in the experience replay pool until the experience replay pool is full;

[0131] The network update module is used to update the evaluation network and behavior network based on the storage records of the experience replay pool, end the current iteration, and proceed to the next iteration. When the current iteration number reaches the set maximum iteration number, the iteration is stopped.

[0132] In one embodiment, the expression of the first action array is as follows:

[0133]

[0134] Among them, a c is the first action array, λ k Indicates the computing nodes to which task k is assigned in the task list are numbered 1 to n, and the size of the first action array is the length of the task list;

[0135] The expression of the second action array is as follows:

[0136] a n =[w1,w2,…,w |ε| ];

[0137] Among them, a n is the second action array, and w is the weight of each link.

[0138] In one embodiment, the network update module is further configured to:

[0139] Input the next state and all states in the experience replay pool into the evaluation network to calculate the advantage function of the current strategy;

[0140] Calculate the loss function of the evaluation network based on the advantage function of the current strategy;

[0141] Based on the loss function of the evaluation network, the network parameters of the evaluation network are updated through back propagation to obtain the updated evaluation network.

[0142] In one embodiment, the expression of the loss function of the evaluation network is as follows:

[0143]

[0144] Where c_loss is the loss function of the evaluation network. is the advantage function, is the square of the advantage function.

[0145] In one embodiment, the network update module is further configured to:

[0146] Determine the old behavior network and the new behavior network corresponding to the behavior network of each agent;

[0147] Input all states in the experience replay pool into the old behavior network and the new behavior network respectively, and obtain the first probability distribution and the second probability distribution respectively;

[0148] Obtain the first action probability and the second action probability corresponding to the first probability distribution and the second probability distribution for all comprehensive actions in the experience replay pool;

[0149] Divide the second action probability by the first action probability to get the importance weight;

[0150] Calculate the loss function of the behavior network based on the importance weight, the advantage function of the current strategy, and the learning rate;

[0151] Based on the loss function of the behavior network, the network parameters of the behavior network are updated through back propagation to obtain the updated behavior network.

[0152] In one embodiment, the loss function of the behavior network is expressed as follows:

[0153]

[0154] Where a_loss is the loss function of the network, ratio is the importance weight value, ∈ is the learning rate, and clip(ratio,1-∈,1+∈) means clipping the importance weights that exceed the range of (1-∈,1+∈).

[0155] Correspondingly, an embodiment of the present invention also provides a terminal device, including: a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, and when the processor executes the computer program, the computing task scheduling method of any one of the above embodiments is implemented.

[0156] The terminal device of this embodiment includes: a processor, a memory, and a computer program and computer instructions stored in the memory and executable on the processor. When the processor executes the computer program, each step in the above embodiment 1 is implemented, for example Figure 1 Alternatively, when the processor executes the computer program, the functions of each module / unit in the above device embodiment, such as the action array generation module 30, are implemented.

[0157] Exemplarily, the computer program can be divided into one or more modules / units, one or more modules / units are stored in a memory and executed by a processor to complete the present invention. One or more modules / units can be a series of computer program instruction segments that can perform specific functions, and the instruction segments are used to describe the execution process of the computer program in a terminal device. For example, the action array generation module 30 is used to use the behavior network corresponding to the intelligent agent to output the probability distribution of the selection of the computing node, and generate a first action array based on the probability distribution of the selection of the computing node; use the behavior network corresponding to the intelligent agent to output the probability distribution of the link weight, and generate a second action array based on the probability distribution of the link weight.

[0158] The terminal device may be a computing device such as a desktop computer, a notebook, a PDA, and a cloud server. The terminal device may include, but is not limited to, a processor and a memory. Those skilled in the art will appreciate that the schematic diagram is merely an example of a terminal device and does not constitute a limitation on the terminal device. The terminal device may include more or fewer components than shown in the diagram, or may combine certain components, or different components. For example, the terminal device may also include an input / output device, a network access device, a bus, etc.

[0159] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the terminal device, and uses various interfaces and lines to connect various parts of the entire terminal device.

[0160] The memory can be used to store computer programs and / or modules. The processor realizes various functions of the terminal device by running or executing the computer programs and / or modules stored in the memory and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function, etc.; the data storage area can store data created according to the use of the mobile terminal, etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0161] Among them, if the module / unit integrated in the terminal device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electric carrier signals and telecommunication signals.

[0162] Accordingly, an embodiment of the present invention further provides a computer-readable storage medium, the computer-readable storage medium including a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the computing task scheduling method of any one of the above embodiments.

[0163] The above specific embodiments further illustrate the purpose, technical solutions and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. It is particularly pointed out that for those skilled in the art, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of protection of the present invention.

Claims

1. A computing task scheduling method, It is characterized in that include: Initializing model parameters of a multi-agent reinforcement learning model, creating an agent for each UAV in the target area, wherein each agent in the multi-agent reinforcement learning model corresponds to a behavior network, and all agents share an evaluation network; Setting an initial state according to network configuration data and starting to iterate the multi-agent reinforcement learning model; the network configuration data includes network topology, task list, computing node list and network link status; The behavior network corresponding to the agent outputs a probability distribution of computing node selection, and generates a first action array based on the probability distribution of computing node selection; the behavior network corresponding to the agent outputs a probability distribution of link weights, and generates a second action array based on the probability distribution of link weights; A comprehensive action is formed based on the first action array and the second action array, and a task path planning scheme is determined according to the comprehensive action. The path planning scheme includes a destination node and a forwarding path selected by the calculation task.

2. The computing task scheduling method according to claim 1, It is characterized in that After forming a comprehensive action based on the first action array and the second action array and determining a task path planning scheme according to the comprehensive action, the method further includes: A node network is formed based on drone nodes, network performance statistics are calculated according to network status data in the node network, and a shared reward is calculated according to a weight index of the network performance statistics; the network status data includes link utilization and transmission delay data, and the network performance statistics include link utilization variance, maximum waiting operation time variance and average task completion time; The next state is set with the current network configuration data, and the initial state, the comprehensive action, the shared reward and the next state are stored in the experience replay pool until the experience replay pool is full; The evaluation network and the behavior network are updated based on the storage records of the experience replay pool, the current iteration is ended, and the next iteration is performed. When the current iteration number reaches the set maximum iteration number, the iteration is stopped.

3. The computing task scheduling method according to claim 1, It is characterized in that The expression of the first action array is as follows: Among them, a c is the first action array, λ k Indicates the computing nodes to which task k is assigned in the task list, numbered 1 to n. The size of the first action array is the length of the task list. The expression of the second action array is as follows: a n =[w 1 ,w 2 ,...,w |ε| ] Among them, a n is the second action array, and w is the weight of each link.

4. The computing task scheduling method according to claim 2, It is characterized in that The updating of the evaluation network and the behavior network based on the storage records of the experience replay pool includes: Input the next state and all states in the experience replay pool into the evaluation network, and calculate the advantage function of the current strategy; Calculating the loss function of the evaluation network according to the advantage function of the current strategy; Based on the loss function of the evaluation network, the network parameters of the evaluation network are updated through back propagation to obtain an updated evaluation network.

5. The computing task scheduling method according to claim 4, It is characterized in that The expression of the loss function of the evaluation network is as follows: Where c_loss is the loss function of the evaluation network. is the advantage function, is the square of the advantage function.

6. The computing task scheduling method according to claim 4, It is characterized in that The updating of the evaluation network and the behavior network based on the storage records of the experience replay pool includes: Determine the old behavior network and the new behavior network corresponding to the behavior network of each agent; Inputting all states in the experience replay pool into the old behavior network and the new behavior network respectively, to obtain a first probability distribution and a second probability distribution respectively; Obtaining first action probabilities and second action probabilities corresponding to the first probability distribution and the second probability distribution for all comprehensive actions in the experience replay pool; Dividing the second action probability by the first action probability to obtain an importance weight; Calculating the loss function of the behavior network according to the importance weight, the advantage function of the current strategy and the learning rate; Based on the loss function of the behavior network, the network parameters of the behavior network are updated through back propagation to obtain an updated behavior network.

7. The computing task scheduling method according to claim 6, It is characterized in that The expression of the loss function of the behavior network is as follows: Where a_loss is the loss function of the network, ratio is the importance weight value, ∈ is the learning rate, and clip(ratio,1-∈,1+∈) means clipping the importance weights that exceed the range of (1-∈,1+∈).

8. A computing task scheduling device, It is characterized in that include: An agent creation module is used to initialize the model parameters of the multi-agent reinforcement learning model and create an agent for each UAV in the target area. Each agent in the multi-agent reinforcement learning model corresponds to a behavior network, and all agents share an evaluation network. An iteration start module, used to set an initial state according to network configuration data and start iterating the multi-agent reinforcement learning model; the network configuration data includes network topology, task list, computing node list and network link status; An action array generation module, configured to use the behavior network corresponding to the agent to output the probability distribution of computing node selection, and generate a first action array based on the probability distribution of computing node selection; use the behavior network corresponding to the agent to output the probability distribution of link weights, and generate a second action array based on the probability distribution of link weights; The task path planning scheme determination module is used to form a comprehensive action based on the first action array and the second action array, and determine the task path planning scheme according to the comprehensive action. The path planning scheme includes calculating the destination node and forwarding path selected by the task.

9. A terminal device, It is characterized in that include: A processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor implements the computing task scheduling method as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, It is characterized in that The computer-readable storage medium includes a stored computer program; wherein, when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the computing task scheduling method according to any one of claims 1 to 7.

Citation Information

Cited By

  • Resource-aware intelligent edge gateway system and operation method

    CN119545435A