Multi-agent depth deterministic gradient strategy calculation unloading method

Abstract

The invention relates to the technical field of edge computing, in particular to a multi-agent depth deterministic gradient strategy computing unloading method, which specifically comprises the following steps of: establishing a terminal equipment task unloading decision model in a cooperative application scene of multiple internet of things terminal equipment and multiple edge computing server nodes; a Stackelberg game model is constructed by analyzing a competitive relationship between edge nodes and Internet of Things terminal equipment, then a corresponding multi-agent Markov decision process is designed, and finally, the Nash equilibrium of the game model is solved by using the designed multi-agent training. Compared with the prior art, the method has the advantages that the cost of an edge computing system can be effectively reduced, the problem that a traditional deep reinforcement learning algorithm is caught in a local optimal solution due to long space dimension of exploration action is solved, and a dynamic computing unloading decision in a multi-terminal and multi-edge node cooperative emergency scene is realized; the timeliness of sensing information transmission in an emergency scene is improved, and meanwhile, the unloading cost is reduced.

Description

Technical Field

[0001] The present invention relates to the field of edge computing technology, and in particular to a multi-agent deep deterministic gradient strategy calculation offloading method. Background Art

[0002] With the rapid growth and application of IoT devices, such as smartphones, sensors, and wearable devices, a large number of computationally intensive tasks need to be transferred from IoT devices to cloud servers for execution. However, the transfer process of these intensive tasks involves a large amount of data transmission, which will lead to high latency in IoT applications. The emergence of mobile edge computing (MEC) can effectively alleviate this challenge. Mobile edge computing can migrate complex tasks from computationally intensive tasks in IoT devices to edge servers, thereby providing computing services for IoT devices. By leveraging the computing and decision-making capabilities of edge servers and cloud servers to reduce computing latency and energy, the user experience quality can be improved.

[0003] In order to cope with the large number of computing-intensive tasks brought by the explosive growth of IoT devices, it is usually necessary to deploy multiple edge node servers. However, due to cost considerations, the computing power and computing resources of multiple edge node servers are usually different. Therefore, how to offload computing tasks to terminal devices in edge computing systems and which edge node to offload to, that is, the offloading strategy of terminal devices, will have a great impact on the system's latency and energy consumption. Using traditional intelligent optimization algorithms or deep reinforcement learning algorithms to solve and explore the action space dimension is long, resulting in long iteration time or easy to fall into the local optimal solution problem, which does not meet the requirements of low latency and high service quality of edge computing. Summary of the invention

[0004] The purpose of the present invention is to provide a multi-agent deep deterministic gradient strategy calculation offloading method, which aims to solve the problem of traditional deep reinforcement learning algorithms falling into local optimal solutions due to the long dimension of exploring action space, improve the timeliness of perception information transmission in emergency scenarios while reducing offloading costs.

[0005] To achieve the above object, the present invention provides a multi-agent deep deterministic gradient strategy calculation offloading method, comprising the following steps:

[0006] Step 1: Construct a Stackelberg game model including multiple IoT terminals and multiple edge nodes;

[0007] Step 2: Design the Markov decision process;

[0008] Step 3: Multiple agents select actions based on their observations of the environment;

[0009] Step 4: Extract a certain number of samples from the experience replay pool and update the multi-agent online policy network parameters θ by calculating the target reward value, loss function and policy gradient μ and the online Q network parameter θ Q , and adopt the soft update target strategy network parameter θ μ ' and the target Q network parameters θ Q '.

[0010] Optionally, the environmental scenario in step 1 is constructed under the application scenario of multiple IoT devices and multiple edge computing server nodes, specifically including multiple edge computing server nodes with different computing capabilities and multiple IoT terminal devices, that is, multiple intelligent agents. The service cycle of the entire system is divided into time slots. The terminal device will generate a computing-intensive task in each time slot. The terminal device offloads the computing task to the edge node for auxiliary processing through the wireless network.

[0011] Optionally, the utility functions of the IoT terminal devices and edge nodes in the Stackelberg game model are constructed, and the utility function of the edge node server is as follows:

[0012]

[0013] Among them, U c The utility of edge node i, P i represents the price of computing resources per unit of edge node i, D t represents the amount of task data unloaded from the terminal device to the edge node in the edge computing system time slot t, γ i represents the proportion of tasks offloaded from IoT terminal devices to edge nodes, and C represents the unit resource cost of edge nodes;

[0014] The utility function of the IoT terminal device is as follows:

[0015]

[0016] Among them, C user Indicates the unit resource computing cost of IoT terminal devices, It indicates the proportion of local computing tasks generated by IoT terminal device i.

[0017] Optionally, the edge nodes in step 2 are used as the state space S of the agent c , action space A c , reward function design strategy R c The expression is as follows:

[0018] S c ={C c ,P c ,γ}

[0019] A c =P c

[0020] R c =(P c -C c )×γ×D

[0021] Among them, C c is the unit resource cost set of all edge nodes, P c is the price set of all edge node unit resources. γ is the set of offloading strategies for all IoT terminal devices.

[0022] Optionally, in step 2, the state space S of the IoT terminal device as an intelligent agent user 、Action Space Design Strategy A user , reward function R user The expression is as follows:

[0023] S user ={D,P c}

[0024] A user =γ

[0025] R user =(-P c -C user -C Tr )*D.

[0026] Optionally, in step 3, the action selection strategy of the multi-agents is that each agent selects an action in the system time slot t according to its partial observation o(t) and strategy μ θ :Φ→Α select action a(t):

[0027] a(t)=μ θ (o(t);θ).

[0028] Optionally, each agent in step 4 contains 4 neural networks: Actor network with parameters θ μ ; Critic network, parameter is θ Q ; Target Actor network, parameter is θ μ '; Target Critic network, parameter is θ Q ';

[0029] The Actor network is used to explore the environment state s and generate a predicted action a. The Critic network further evaluates the value of action a and outputs Q(s,a|θ Q ), the cumulative expected reward of the agent performing action a in state s is calculated as follows:

[0030]

[0031] Among them, s'~P represents the state transition probability P to get the next state s', Q'(s',a') represents the action value obtained by performing action a' in state s', and r represents the reward value obtained by performing action a' in state s'; then the parameter θ Q The target network Q value can be expressed as:

[0032] y=r+γQ'(s',μ'(s'|θ μ ')|θ Q ')

[0033] By calculating the mean square error between the estimated value and the actual value, the loss function of the Critic network is expressed as:

[0034] L(θ Q )=E μ '[(Q(s,a|θ Q )-y 2 )]

[0035] The target network uses a soft update strategy to update parameters, that is, each time the update is performed, the parameters of the target network will gradually approach the parameters of the online network to a certain extent to reduce the mutation caused by the update; specifically, the parameters θ in the target Actor network and the target Critic network are updated iteratively. μ ' and θ Q ':

[0036] θ μ '←ω1θ μ +(1-ω1)θ μ '

[0037] θ Q '←ω2θ Q +(1-ω2)θ Q '.

[0038] The present invention provides a multi-agent deep deterministic gradient strategy calculation offloading method, specifically, establishing a terminal device task offloading decision model in a collaborative application scenario of multiple IoT terminal devices and multiple edge computing server nodes, constructing a Stackelberg game model by analyzing the competitive relationship between edge nodes and IoT terminal devices, and then designing a corresponding multi-agent Markov decision process, and finally using the designed multi-agent training to solve the Nash equilibrium of the game model. Compared with the prior art, it can effectively reduce the cost of edge computing systems, and solve the problem of traditional deep reinforcement learning algorithms falling into local optimal solutions due to the long dimension of the exploration action space, and realize dynamic calculation offloading decisions in multi-terminal and multi-edge node collaborative emergency scenarios, improve the timeliness of perceptual information transmission in emergency scenarios, and reduce the offloading cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0040] Figure 1 It is a schematic diagram of a forest fire prevention and firefighting scene according to a specific embodiment of the present invention.

[0041] Figure 2 It is a schematic diagram of performance comparison between the method of the present invention and other benchmark algorithms.

[0042] Figure 3 It is a schematic diagram of the performance comparison between the method of the present invention and other benchmark algorithms under different channel bandwidths of the system.

[0043] Figure 4 It is a schematic diagram of the performance comparison between the method of the present invention and other benchmark algorithms at the CPU main frequency of the system Internet of Things terminal device. DETAILED DESCRIPTION

[0044] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0045] The present invention provides a multi-agent deep deterministic gradient strategy calculation offloading method, comprising the following steps:

[0046] Step 1: Construct a Stackelberg game model including multiple IoT terminals and multiple edge nodes;

[0047] Step 2: Design the Markov decision process;

[0048] Step 3: Multiple agents select actions based on their observations of the environment;

[0049] Step 4: Extract a certain number of samples from the experience replay pool and update the multi-agent online policy network parameters θ by calculating the target reward value, loss function and policy gradient μ and the online Q network parameter θ Q , and adopt the soft update target strategy network parameter θ μ ' and the target Q network parameters θ Q '.

[0050] See also Figures 1 to 4 , the following is further described in conjunction with specific embodiments and implementation steps:

[0051] Specifically implemented in monitoring scenarios such as forest fire prevention and firefighting, it can effectively handle the computing offload tasks generated by IoT monitoring devices and minimize the latency and energy consumption of the entire edge computing monitoring system, as shown in the attached Figure 1 As shown, the environmental conditions are as follows:

[0052] (1) Computing environment: Assume that a fire breaks out in a forest area and rescue workers are needed to carry out firefighting and rescue work. In the area, there are N IoT monitoring devices and multiple drones equipped with micro edge computing servers as air edge nodes, and a communication quality assurance vehicle as a ground edge node.

[0053] (2) Computational tasks: Assume that the service cycle of the entire system is divided into T time slots. The terminal device will generate a computationally intensive task D (the monitored data that needs to be analyzed) in each time slot, which is sent wirelessly to the edge node for calculation. The edge node then transmits the analysis results back to the emergency command center for fire disaster assessment and to provide information support to rescue personnel.

[0054] (3) Task offloading strategy: The offloading strategy of the terminal device is a partial offloading strategy, that is, the task can be divided into multiple parts and sent to multiple edge nodes for calculation.

[0055] (4) Optimization goal of the present invention: The goal of the multi-agent deep deterministic gradient strategy calculation offloading method based on Stackelberg game of the present invention is to reduce the system delay and weighted sum of energy consumption. t And energy consumption E t The formula is as follows:

[0056]

[0057]

[0058] The corresponding implementation steps are as follows:

[0059] S1. Construct Stackelberg game model;

[0060] S101: Constructing the utility function U of the communication support vehicle bs :

[0061]

[0062] in, represents the unit resource price of the communication support vehicle for the IoT terminal device i, D irepresents the size of the computing task generated by the IoT terminal device, C bs represents the unit resource calculation cost of the communication support vehicle, It indicates the proportion of IoT terminal device i offloading tasks to the communication support vehicle.

[0063] S102: Construct drone utility function:

[0064]

[0065] in, represents the unit resource price of the drone for IoT terminal device i, C uav represents the unit resource computing cost of the drone, It indicates the proportion of IoT terminal device i offloading tasks to drones.

[0066] S103: Constructing the utility function of the IoT terminal device:

[0067]

[0068] in, The proportion of tasks computed locally by IoT terminal devices, C user Indicates the unit resource computing cost of IoT terminal devices.

[0069] S2. Design a multi-agent Markov decision process. The Markov decision process of the algorithm of the present invention is designed according to the Stackelberg model of step S1, and each edge node is independently designed as an agent, and all IoT terminal devices are designed as an agent.

[0070] S201: The Markov decision process of the drone, i.e. the state space S UAV , action space A uav And the reward function R UAV The design is as follows:

[0071] S UAV ={C UAV ,P o_uav ,P CSV ,γ}

[0072] A uav =P uav

[0073] R UAV =(P UAV -C UAV )×γ UAV ×D

[0074] Among them, P o_uav is the unit resource price of other drones, P CSVThe communication support vehicle's unit resource price collection for all IoT devices.

[0075] S202: The Markov decision process of the communication support vehicle, i.e., the state space S CSV , action space A CSV And the reward function R CSV The design is as follows:

[0076] S CSV ={C CSV ,P UAV ,γ}

[0077] A CSV =P CSV

[0078] R CSV =(P CSV -C CSV )×γ CSV ×D

[0079] S203: The Markov decision process of IoT terminal devices, i.e., the state space S TD , action space A TD And the reward function R TD The design is as follows:

[0080] S TD ={D,P CSV ,P UAV}

[0081] A TD =γ

[0082] R TD =(-P CSV -P UAV -C UAV -Tr)*D

[0083] S3, initialization environment, unit cost and resource price of edge nodes and terminal devices, size of agent training experience pool C, batch size N;

[0084] S4. Update the current task information of the terminal device:

[0085] S5. The agent makes action decisions based on the environment a(t) = μ θ (o(t);θ);

[0086] S501: Calculate reward r(t) based on agent action a(t)

[0087] S502: The state transfer experience tuple (s i (t),a i (t),r i (t),si (t+1)) stored in the experience pool

[0088] S6. Randomly sample N experiences from the experience pool:

[0089] S601: Calculated through the target Actor network

[0090] S602: Calculation through target critic network

[0091] S603: Update the Critic network parameters by minimizing the mean square error and gradient descent: L m (ω m )=E[(Q m (o m (t),a m (t);ω m )-y m ) 2 ];

[0092] S604: Update actor network parameters via policy gradients

[0093] S605, soft update target Actor network:

[0094] S606, soft update target critic network:

[0095] Furthermore, the present invention is also explained by comparing with other benchmark algorithms:

[0096] like Figures 2 to 4 As shown in the figure, the English term abbreviations and benchmark algorithms mentioned in the figure are defined as follows:

[0097] SG-MADDPG: Stackelberg-Based Multi-Agent Deep Deterministic PolicyGradient, a multi-agent deep deterministic gradient policy calculation offloading method based on Stackelberg game of the present invention;

[0098] Only UAV: ​​All tasks generated by the terminal device are offloaded to the UAV, and offloading to the communication rescue vehicle is not considered;

[0099] Ony CSV: All tasks generated by the terminal device are offloaded to the communication rescue vehicle, and offloading to the drone is not considered;

[0100] DDPG: Single-agent Deep Deterministic Policy Gradient Algorithm;

[0101] Random: The terminal device is randomly unloaded to any edge node.

[0102] Figure 2 The performance comparison of SG-MADDPG and other benchmark algorithms in different time slots is shown. As can be seen from the figure, the performance of the method of the present invention is the best under different time slots and numbers of users. The reason is that the SG-MADDPG method is designed based on the difference in device computing power and the relative advantages and disadvantages of channel conditions. The offloading decision is closer to the optimal value. At the same time, multi-agent reinforcement learning can reduce the problem of too high dimension of the state space, so its performance is better than DDPG. The Only CSV and Only UAV strategies are because when all tasks are offloaded to a certain type of server, the communication interference between users will increase, which will lead to an increase in transmission delay and energy consumption. The All local strategy has high computing delay and energy consumption due to the low computing power of the local device.

[0103] Figure 3 The relationship between the system delay and the weighted sum of energy consumption and the channel bandwidth under different strategies is shown. As can be seen from the figure, the SG-MADDPG strategy is better than other strategies under different channel bandwidths. This is because the Markov decision process constructed by SG-MADDPG combined with the Stackelberg game enables multi-agents to efficiently learn action strategies and effectively respond to dynamic changes in the environment. In addition, when the bandwidth is small, the Only CSV strategy will produce a large decline as the channel bandwidth continues to increase. This is because the channel conditions between CSV and the terminal device are poor and the path loss is large. The increase in bandwidth can make up for its shortcomings to a certain extent. However, when the channel bandwidth increases to a certain threshold, the benefits of all offloading strategies are relatively stable, indicating that the reduction in transmission delay cannot further reduce the system delay and the weighted sum of energy consumption, and the computing delay has become the main factor affecting the system cost.

[0104] Figure 4 The performance comparison between SG-MADDPG and other benchmark algorithms at different terminal device CPU frequencies is shown. As can be seen from the figure, with the continuous improvement of the computing power of the terminal device, the latency and energy consumption weighted and performance gap brought by the SG-MADDPG offloading strategy and the Alllocal strategy are shrinking. For example, when the CPU frequency of the terminal device reaches about 12-14GHz, the system performance under the two strategies is not much different. The reason is that when the computing power of the terminal device is small, the All local strategy will bring higher computing latency to the system. Further increasing the computing power of the terminal device can significantly reduce the computing latency, so its performance is close to the SG-MADDPG strategy. It also shows that the multi-agent system based on the SG-MADDPG strategy has strong adaptability.

[0105] What is disclosed above is only a preferred embodiment of the present invention, and it certainly cannot be used to limit the scope of rights of the present invention. Ordinary technicians in this field can understand that all or part of the processes of the above embodiment and equivalent changes made according to the claims of the present invention still fall within the scope of the invention.

Claims

1. A multi-agent deep deterministic gradient strategy calculation offloading method, characterized in that: The following steps are involved: Step 1: Construct a Stackelberg game model including multiple IoT terminals and multiple edge nodes; Step 2: Design the Markov decision process; Step 3: Multiple agents select actions based on their observations of the environment; Step 4: Extract a certain number of samples from the experience replay pool and update the multi-agent online policy network parameters θ by calculating the target reward value, loss function and policy gradient μ and the online Q network parameter θ Q , and adopt the soft update target strategy network parameter θ μ' and the parameters θ of the target Q network Q' .

2. The multi-agent deep deterministic gradient strategy calculation offloading method according to claim 1, characterized in that: The environmental scenario in step 1 is constructed under the application scenario of multiple IoT devices and multiple edge computing server nodes, specifically including multiple edge computing server nodes with different computing capabilities and multiple IoT terminal devices, that is, multiple intelligent agents. The service cycle of the entire system is divided into time slots. The terminal device will generate a computing-intensive task in each time slot. The terminal device offloads the computing task to the edge node for auxiliary processing through the wireless network.

3. The multi-agent deep deterministic gradient strategy calculation offloading method according to claim 2, characterized in that: The utility function of the IoT terminal device and edge node in the Stackelberg game model is constructed, and the utility function of the edge node server is as follows: Among them, U c The utility of edge node i, P i represents the price of computing resources per unit of edge node i, D t represents the amount of task data unloaded from the terminal device to the edge node in the edge computing system time slot t, γ i represents the proportion of tasks offloaded from IoT terminal devices to edge nodes, and C represents the unit resource cost of edge nodes; The utility function of the IoT terminal device is as follows: Among them, C user Indicates the unit resource computing cost of IoT terminal devices, It indicates the proportion of local computing tasks generated by IoT terminal device i.

4. The multi-agent deep deterministic gradient strategy calculation offloading method according to claim 3, characterized in that: In step 2, the edge nodes serve as the state space S of the agent. c , action space A c , reward function design strategy R c The expression is as follows: S c J{C c ,P c ,γ} A c =P c R c =(P c -C c )×γ×D Among them, C c is the unit resource cost set of all edge nodes, P c is the price set of all edge node unit resources, and γ is the set of offloading strategies for all IoT terminal devices.

5. The multi-agent deep deterministic gradient strategy calculation offloading method according to claim 4, characterized in that: In step 2, the state space S of the IoT terminal device as an intelligent agent user 、Action Space Design Strategy A user , reward function R user The expression is as follows: S user ={D,P c } A user =c R user =(-P c -C user -C Tr )*D。 6. The multi-agent deep deterministic gradient strategy calculation offloading method according to claim 5, characterized in that: In step 3, the action selection strategy of the multi-agent is that each agent selects an action in the system time slot t according to its partial observation o(t) and strategy μ θ :Φ→Α select action a(t): a(t)=μ θ (o(t);θ).

7. The multi-agent deep deterministic gradient strategy calculation offloading method according to claim 6, characterized in that: In step 4, each agent contains four neural networks: Actor network with parameters θ μ ; Critic network, parameter is θ Q ; Target Actor network, parameter is θ μ' ; Target Critic network, parameter is θ Q' ; The Actor network is used to explore the environment state s and generate a predicted action a. The Critic network further evaluates the value of action a and outputs Q(s,a|θ Q ), the cumulative expected reward of the agent performing action a in state s is calculated as follows: Among them, s'~P represents the state transition probability P to get the next state s', Q'(s',a') represents the action value obtained by performing action a' in state s', and r represents the reward value obtained by performing action a' in state s'; then the parameter θ Q' The target network Q value can be expressed as: y=r+γQ'(s',μ'(s'|θ μ' )|θ Q' ) By calculating the mean square error between the estimated value and the actual value, the loss function of the Critic network is expressed as: L(θ Q )=E μ' [(Q(s,a|θ Q )-y 2 )] The target network uses a soft update strategy to update parameters, that is, each time the update is performed, the parameters of the target network will gradually approach the parameters of the online network to a certain extent to reduce the mutation caused by the update; specifically, the parameters θ in the target Actor network and the target Critic network are updated iteratively. μ' and θ Q' : i μ' ←ω1θ μ +(1-ω1)θ μ' i Q' ←ω2θ Q +(1-ω2)θ Q' 。

Images