A Multi-Agent Deep Deterministic Gradient Policy Computation Offloading Method

Abstract

This invention relates to the field of edge computing technology, specifically to a multi-agent deep deterministic gradient policy computation offloading method. Specifically, it involves establishing a terminal device task offloading decision model in a collaborative application scenario involving multiple IoT terminal devices and multiple edge computing server nodes. By analyzing the competitive relationship between edge nodes and IoT terminal devices, a Stackelberg game model is constructed, followed by the design of a corresponding multi-agent Markov decision process. Finally, the designed multi-agent training method is used to solve the Nash equilibrium of the game model. Compared with existing technologies, this method effectively reduces the cost of edge computing systems and solves the problem of traditional deep reinforcement learning algorithms getting trapped in local optima due to the long dimension of the action space. It achieves dynamic computation offloading decision-making in collaborative emergency scenarios involving multiple terminals and multiple edge nodes, improving the timeliness of sensing information transmission in emergency scenarios while reducing offloading costs.

Description

Technical Field

[0001] This invention relates to the field of edge computing technology, and more specifically to a method for offloading multi-agent deep deterministic gradient policy computation. Background Technology

[0002] With the rapid growth and application of IoT devices, such as smartphones, sensors, and wearables, a large number of computationally intensive tasks need to be migrated from IoT devices to cloud servers. However, the migration of these intensive tasks involves a large amount of data transmission, leading to high latency in IoT applications. Mobile edge computing (MEC) can effectively alleviate this challenge. MEC migrates complex, computationally intensive tasks from IoT devices to edge servers, thereby providing computing services to IoT devices. By leveraging the computing and decision-making capabilities of edge servers and cloud servers, computational latency and energy consumption are reduced, thus improving the user experience.

[0003] To cope with the surge in computationally intensive tasks brought about by the proliferation of IoT devices, multiple edge node servers are typically deployed. However, due to cost considerations, the computing power and resources of these edge node servers usually vary. Therefore, how terminal devices offload computing tasks and to which edge node they do so significantly impacts system latency and energy consumption. Using traditional intelligent optimization algorithms or deep reinforcement learning algorithms for problem-solving involves a long action space dimension, leading to long iteration times or a tendency to get trapped in local optima, which does not meet the requirements of low latency and high service quality in edge computing. Summary of the Invention

[0004] The purpose of this invention is to provide a multi-agent deep deterministic gradient policy calculation offloading method, which aims to solve the problem of traditional deep reinforcement learning algorithms getting trapped in local optima due to the long dimension of the action space exploration, improve the timeliness of perception information transmission in emergency scenarios and reduce offloading costs.

[0005] To achieve the above objectives, the present invention provides a method for offloading multi-agent deep deterministic gradient policy computation, comprising the following steps:

[0006] Step 1: Construct a Stackelberg game model that includes multiple IoT terminals and multiple edge nodes;

[0007] Step 2: Design a Markov decision process;

[0008] Step 3: The multi-agent system selects actions based on its observations of the environment;

[0009] Step 4: Draw a certain number of samples from the experience replay pool, and update the online policy network parameters of the multi-agent system by calculating the target reward value, loss function, and policy gradient. and online Network parameters And adopt a soft update target strategy for network parameters. and target Network parameters .

[0010] Optionally, the environment scenario in step 1 is constructed in an application scenario with multiple IoT devices and multiple edge computing server nodes. Specifically, it includes multiple edge computing server nodes with different computing capabilities and multiple IoT terminal devices, i.e. multiple intelligent agents. The service cycle of the entire system is divided into time slots. In each time slot, the terminal device generates a computationally intensive task. The terminal device offloads the computing task to the edge node for auxiliary processing through the wireless network.

[0011] Optionally, the utility functions of the IoT terminal devices and edge nodes in the Stackelberg game model are constructed as follows:

[0012]

[0013] in, The utility of edge node i This represents the selling price of one unit of computing resources at edge node i. This represents the amount of task data that is offloaded from the terminal device to the edge node in time slot t of the edge computing system. This indicates the proportion of tasks that are offloaded from IoT terminal devices to edge nodes. This represents the unit resource cost of an edge node;

[0014] The utility function for IoT terminal devices is as follows:

[0015]

[0016] in, This represents the unit resource computing cost of IoT terminal devices. This indicates the proportion of computing tasks generated by IoT terminal device i that are performed locally.

[0017] Optionally, in step 2, the edge nodes serve as the state space of the agent. Action space Reward function design strategy The expression is as follows:

[0018]

[0019]

[0020]

[0021] in, This is the set of unit resource costs for all edge nodes. This is the set of unit resource prices for all edge nodes. This is a set of offloading strategies for all IoT terminal devices.

[0022] Optionally, in step 2, the state space of the IoT terminal device as an intelligent agent... Action space design strategy Reward function The expression is as follows:

[0023]

[0024]

[0025] .

[0026] Optionally, in step 3, the multi-agent action selection strategy involves each agent selecting actions based on its partial observations within system time slot t. and strategies Select Action :

[0027] .

[0028] Optionally, in step 4, each agent contains four neural networks: an Actor network with parameters as follows: ; Critic network, parameters are ;Target Actor network, parameters are ; Target Critic network, parameters are ;

[0029] Actor networks are used to explore environmental states. And generate predicted actions Further evaluation of actions through the Critic network. Value and output Calculate the agent's state Next action The cumulative expected reward is shown in the following formula:

[0030]

[0031] in, Represents the state transition probability Get the next state , Indicates the state Execute action The obtained action value, Indicates the state Execute action The obtained reward value; then the parameter Target network The value can be represented as:

[0032]

[0033] By calculating the mean square error between the estimated and actual values, the loss function of the Critic network is expressed as:

[0034]

[0035] The target network employs a soft update strategy to update its parameters. This means that with each update, the target network's parameters gradually approach the online network's parameters to minimize abrupt changes caused by the update. Specifically, the parameters in the target Actor network and the target Critic network are updated iteratively. and :

[0036]

[0037] .

[0038] This invention provides a multi-agent deep deterministic gradient policy computation offloading method. Specifically, it establishes a terminal device task offloading decision model in a collaborative application scenario involving multiple IoT terminal devices and multiple edge computing server nodes. By analyzing the competitive relationship between edge nodes and IoT terminal devices, a Stackelberg game model is constructed, and a corresponding multi-agent Markov decision process is designed. Finally, the designed multi-agent training method is used to solve the Nash equilibrium of the game model. Compared with existing technologies, this method effectively reduces the cost of edge computing systems and solves the problem of traditional deep reinforcement learning algorithms getting trapped in local optima due to the long dimension of the action space. It achieves dynamic computation offloading decision-making in collaborative emergency scenarios involving multiple terminals and multiple edge nodes, improving the timeliness of sensing information transmission in emergency scenarios while reducing offloading costs. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 This is a schematic diagram of a forest fire prevention and firefighting scenario according to a specific embodiment of the present invention.

[0041] Figure 2 This is a schematic diagram comparing the performance of the method of this invention with other benchmark algorithms.

[0042] Figure 3 This diagram illustrates the performance comparison between the method of this invention and other benchmark algorithms under different channel bandwidths in the system.

[0043] Figure 4 This is a schematic diagram comparing the performance of the method of this invention with other benchmark algorithms at the CPU clock frequency of the system's IoT terminal device. Detailed Implementation

[0044] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0045] This invention provides a method for offloading the computation of multi-agent deep deterministic gradient policies, comprising the following steps:

[0046] Step 1: Construct a Stackelberg game model that includes multiple IoT terminals and multiple edge nodes;

[0047] Step 2: Design a Markov decision process;

[0048] Step 3: The multi-agent system selects actions based on its observations of the environment;

[0049] Step 4: Draw a certain number of samples from the experience replay pool, and update the online policy network parameters of the multi-agent system by calculating the target reward value, loss function, and policy gradient. and online Network parameters And adopt a soft update target strategy for network parameters. and target Network parameters .

[0050] Please see Figures 1 to 4The following description, in conjunction with specific embodiments and implementation steps, provides further details:

[0051] Specifically implemented in monitoring scenarios such as forest fire prevention and fighting, it can effectively handle the computational offloading tasks generated by IoT monitoring devices, minimizing the latency and energy consumption of the entire edge computing monitoring system, as shown in the attached figure. Figure 1 As shown, the environmental conditions are as follows:

[0052] (1) Calculation environment: Assume that a fire has occurred in a forest area, and rescue personnel are needed to carry out fire fighting and rescue work. Fires are unevenly distributed in this area. The system includes an IoT monitoring device and multiple drones, each equipped with a micro edge computing server as an airborne edge node, and a communications quality assurance vehicle as a ground edge node.

[0053] (2) Computational task: Assume the service cycle of the entire system is divided into... Each time slot generates a computationally intensive task for the terminal device. The monitored data that needs to be analyzed is wirelessly transmitted to edge nodes for calculation. The edge nodes then transmit the analysis results back to the emergency command center for fire situation assessment, providing information support to rescue personnel.

[0054] (3) Task offloading strategy: The offloading strategy of the terminal device is a partial offloading strategy, that is, the task can be divided into multiple parts and sent to multiple edge nodes for computation.

[0055] (4) Optimization objective of the present invention: The objective of the multi-agent deep deterministic gradient strategy calculation offloading method based on Stackelberg game in this invention is to reduce the system's time delay and energy consumption weighted sum, and to reduce the system's time slots. latency and energy consumption The formula is shown below:

[0056]

[0057]

[0058] The corresponding implementation steps are as follows:

[0059] S1. Construct the Stackelberg game model;

[0060] S101: Construct a communication guarantee vehicle utility function :

[0061]

[0062] in, This indicates the unit resource price of communication support vehicle for IoT terminal device i. Indicates the size of the computing tasks generated by IoT terminal devices. This indicates the unit resource calculation cost of the communication support vehicle. This indicates the proportion of tasks that IoT terminal devices i offload to the communication support vehicle.

[0063] S102: Construct the drone utility function:

[0064]

[0065] in, This indicates the unit resource price of drones for IoT terminal device i. This indicates the unit resource computation cost of the drone. This indicates the proportion of tasks that IoT terminal devices i offload to drones.

[0066] S103: Constructing the utility function for IoT terminal devices:

[0067]

[0068] in, The proportion of tasks computed locally by IoT terminal devices This represents the unit resource computing cost of IoT terminal devices.

[0069] S2. Design a multi-agent Markov decision process; The Markov decision process algorithm of this invention is designed based on the Stackelberg model in step S1, with each edge node designed independently as an agent and all IoT terminal devices as a single agent;

[0070] S201: Markov Decision Process of Unmanned Aerial Vehicles, i.e., State Space Action space and reward function The design is as follows:

[0071]

[0072]

[0073]

[0074] in, The unit resource price of other drones, The unit resource price set for all IoT devices in the communication support vehicle.

[0075] S202: The Markov Decision Process (State Space) of the Communication Support Vehicle Action space and reward function The design is as follows:

[0076]

[0077]

[0078]

[0079] S203: Markov decision process, i.e., state space, for IoT terminal devices. Action space and reward function The design is as follows:

[0080]

[0081]

[0082]

[0083] S3, initialization environment, unit cost and resource price of edge nodes and terminal devices, and size of the agent training experience pool. Batch size ;

[0084] S4. Update the current task information of the terminal device:

[0085] S5, the intelligent agent makes action decisions based on the environment. ;

[0086] S501: Based on the agent's actions Calculate rewards

[0087] S502: Transition state tuple Store in experience pool

[0088] S6. Randomly sample from the experience pool. One piece of experience:

[0089] S601: Calculation via the target Actor network ;

[0090] S602: Calculation via target Critic network

[0091] S603: Gradient descent updates the Critic network parameters by minimizing the mean squared error. ;

[0092] S604: Update via policy gradient Network parameters :

[0093] S605, Soft Update Target Actor Network:

[0094] S606, Soft Update Target Critic Network: .

[0095] Furthermore, this invention is illustrated by comparison with other benchmark algorithms:

[0096] like Figures 2 to 4 As shown in the figure, the abbreviations of English terms and the definitions of the benchmark algorithms are as follows:

[0097] SG-MADDPG: Stackelberg-Based Multi-Agent Deep Deterministic Policy Gradient, the present invention is a method for unloading multi-agent deep deterministic gradient policy computation based on Stackelberg game theory;

[0098] Only UAV: ​​All tasks generated by the terminal device are offloaded to the UAV, and offloading to the communication rescue vehicle is not considered.

[0099] Ony CSV: All tasks generated by terminal devices are offloaded to the communication rescue vehicle, and offloading to drones is not considered;

[0100] DDPG: Single-agent deep deterministic policy gradient algorithm;

[0101] Random: Terminal devices are randomly offloaded to any edge node.

[0102] Figure 2 The performance comparison of SG-MADDPG with other benchmark algorithms under different time slots is presented. As shown in the figure, the proposed method exhibits the best performance under different time slots and numbers of users. This is because the SG-MADDPG method is designed based on differences in device computing power and relative channel conditions, resulting in offloading decisions closer to the optimal value. Furthermore, multi-agent reinforcement learning reduces the problem of excessively high state space dimensionality, thus its performance is superior to DDPG. The Only CSV and Only UAV strategies suffer from increased communication interference between users when all tasks are offloaded to a certain type of server, leading to increased transmission latency and energy consumption. The All local strategy suffers from high computational latency and energy consumption due to insufficient local device computing power.

[0103] Figure 3The relationship between the weighted sum of system latency and energy consumption and channel bandwidth under different strategies is illustrated. As shown in the figure, the SG-MADDPG strategy outperforms other strategies under different channel bandwidths. This is because the Markov decision process constructed by SG-MADDPG combined with Stackelberg game theory enables multi-agents to efficiently learn action strategies and effectively respond to dynamic environmental changes. Furthermore, when the bandwidth is small, only the Only CSV strategy shows a significant decrease in performance as the channel bandwidth increases. This is because the channel conditions between CSV and the terminal device are poor, resulting in high path loss, which can be partially compensated for by increasing bandwidth. However, when the channel bandwidth increases to a certain threshold, the returns of all offloading strategies become relatively stable, indicating that reducing transmission latency cannot further reduce the weighted sum of system latency and energy consumption; computational latency becomes the main factor affecting system cost.

[0104] Figure 4 This paper presents a performance comparison of SG-MADDPG with other benchmark algorithms under different terminal device CPU frequencies. As shown in the figure, with the continuous improvement of terminal device computing power, the performance gap in latency and energy consumption weighted by the SG-MADDPG offloading strategy and the Alllocal strategy is constantly narrowing. For example, when the terminal device CPU frequency reaches around 12-14 GHz, the system performance under the two strategies is not significantly different. This is because when the terminal device computing power is low, the Alllocal strategy introduces high computational latency to the system. Further increasing the terminal device computing power can significantly reduce computational latency, thus its performance is close to that of the SG-MADDPG strategy. This also indicates that the multi-agent system based on the SG-MADDPG strategy has strong adaptability.

[0105] The above description discloses only one preferred embodiment of the present invention, and should not be construed as limiting the scope of the present invention. Those skilled in the art will understand that all or part of the processes of the above embodiments can be implemented, and equivalent changes made in accordance with the claims of the present invention are still within the scope of the invention.

Claims

1. A multi-agent deep deterministic gradient policy computation offloading method, characterized in that, Comprising the following steps: Step 1: Constructing a Stackelberg game model including multiple Internet of Things terminals and multiple edge nodes; The environment scenario in step 1 is constructed under the application scenario of multiple Internet of Things devices and multiple edge computing server nodes, specifically including multiple edge computing server nodes with different computing capabilities and multiple Internet of Things terminal devices, i.e. multiple agents, the service period of the whole system is divided into time slots, and the terminal device will generate a computing-intensive task in each time slot, and the terminal device will offload the computing task to the edge node for auxiliary processing through a wireless network; The utility function of the Internet of Things terminal device and the edge node in the Stackelberg game model is constructed, and the utility function of the edge node server is as follows: ； wherein, represents the utility function of edge node i, represents the selling price of a unit computing resource of edge node i, represents the amount of task data offloaded by terminal devices to edge node i at time slot t in the edge computing system, represents the proportion of tasks of the Internet of Things terminal device offloaded to edge node i, represents the unit resource cost of edge node i; The utility function of the Internet of Things terminal device is as follows: ； wherein, denotes the computing cost of the unit resource of the IoT terminal device, denotes the proportion of the computing task generated by the IoT terminal device i to be calculated locally; Step 2: Designing a Markov decision process; The state space of the edge node in step 2 as an agent , action space , reward function design strategy The expression is as follows: ； ； ； wherein, is a set of unit resource costs for all edge nodes, is a set of unit resource selling prices for all edge nodes, is a set of offloading policies for all IoT terminal devices; State space of the IoT terminal device in step 2 as an agent , Action space design policy , Reward function The expression is as follows: ； ； ； Step 3: Multiple agents select actions according to their observations of the environment; Step 4: Extract a certain number of samples from the experience replay pool, update the online policy network parameters of the multi-agent by calculating the target return value, loss function and policy gradient and online network parameters , and adopt soft update target policy network parameters and the parameters of the target network.

2. The multi-agent deep deterministic gradient policy computing offloading method according to claim 1, wherein, The action selection policy of the multi-agent in step 3, each agent at system time slot t according to its partial observation and policy selects an action : 。

Images