Deep deterministic gradient policy computation offloading method based on alternating iterative optimization

Abstract

The present application relates to the technical field of deep reinforcement learning, and particularly relates to a deep deterministic gradient policy calculation unloading method based on alternating iterative optimization, a task unloading decision model is established in a cooperative application scene of Internet of Things devices and edge computing server nodes, and coupled terminal device unloading strategies, edge node resource allocation and location deployment variables in the model are modeled into three agents, a corresponding Markov process is designed, and the agents are alternately trained and learned to solve; an optimization goal of the present application is to reduce a time delay and energy consumption weighted sum of a system; compared with the prior art, the problem that a traditional deep reinforcement learning algorithm falls into a local optimal solution due to a long exploration action space dimension is solved, fast unloading decision in a dynamic computing task unloading environment is realized, and user experience is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deep reinforcement learning, and particularly relates to a deep deterministic gradient policy calculation offloading method based on alternating iterative optimization. BACKGROUND

[0002] With the rapid growth and application of Internet of Things (IoT) devices such as smart phones, sensors and wearable devices, a large number of computationally intensive tasks need to be transferred from IoT devices to cloud servers for execution. However, the transfer process of these intensive tasks involves a large amount of data transmission, which will result in high latency of IoT applications.

[0003] The emergence of mobile edge computing (MEC) can effectively alleviate this challenge. Mobile edge computing can migrate complex tasks from computationally intensive tasks in Internet of Things (IoT) devices to edge servers, thereby providing computing services for Internet of Things (IoT) devices. By utilizing the computing and decision-making capabilities of edge servers and cloud servers, the computing delay and energy can be reduced, thereby improving the quality of experience of users.

[0004] However, the computing resources of edge node servers are limited, so in the edge computing system, the task offloading strategy of terminal devices, the location deployment and resource allocation of edge nodes are problems to be solved. Using traditional intelligent optimization algorithms or deep reinforcement learning algorithms to solve the exploration action space dimension is long, which leads to the problems of long iteration time or easy to fall into local optimal solution, which does not meet the requirements of low latency and high service quality of edge computing. SUMMARY

[0005] The purpose of the present application is to provide a deep deterministic gradient policy calculation offloading method based on alternating iterative optimization, which aims to solve the problem of traditional deep reinforcement learning method falling into local optimal solution due to long exploration action space dimension, and to realize fast calculation task offloading decision in dynamic calculation task environment.

[0006] To achieve the above purpose, the present application provides a deep deterministic gradient policy calculation offloading method based on alternating iterative optimization, comprising the following steps:

[0007] Step 1: modeling the terminal device offloading strategy, edge node resource allocation and location deployment variables in the application scenario as three agents, and designing corresponding Markov processes;

[0008] Step 2: initializing the observation environment state space of each agent;

[0009] Step 3: the agent alternately performs actions according to the state space;

[0010] Step 4: merging all agent actions to input the environment to obtain the reward value and the next environment state under the current state;

[0011] Step 5: Process the agent's experience pool;

[0012] Step 6: Extract N experience points to update the agent's neural network.

[0013] Optionally, in step 1, the terminal device unloading decision, UAV resource allocation, and location deployment are modeled as three intelligent agents for alternating decision optimization; the state spaces of the three intelligent agents... for:

[0014]

[0015] in, Represents intelligent agents In the time slot The environmental state of the constructed model was observed, where For terminal devices in time slots The resulting set of computational tasks; Indicates that agent 1 is in the time slot Actions at that time; Indicates that agent 1 is in the time slot Actions at that time; Indicates that agent 2 is in the time slot Actions at that time; Indicates that agent 2 is in the time slot Actions at that time; This indicates that agent 3 is in the time slot. Actions at that time; This indicates that agent 3 is in the time slot. Actions at that time.

[0016] Optional, action space for three agents The set of unloading decisions for the terminal device, the set of drone main frequency allocation coefficients, and the location coordinates of the drone are represented as follows:

[0017]

[0018] in, For terminal devices binary unloading strategy variable, A value of 1 indicates that the terminal device offloads the task to the edge node for computation; otherwise, the task will be computed locally. This represents the proportion of CPU clock speed coefficients allocated to terminal devices from edge nodes, i.e., the resource allocation strategy. This indicates the location coordinates of the edge nodes, i.e., the location deployment strategy.

[0019] Optionally, the reward function policy after merging the agent's actions in step 1 is as follows:

[0020]

[0021] wherein, represents the expected reward value obtained by the agent in state performing action . is the system energy consumption of time slot , is the system delay of time slot , is the punishment value constant obtained when the UAV flies out of the environment region;

[0022] Correspondingly, and the calculation formula is as follows:

[0023]

[0024]

[0025] wherein, represents the task transmission rate of the terminal device in time slot , represents the edge node CPU frequency, represents the number of cycles of the edge node processing each 1 bit size task, represents the CPU capacitance coefficient of the UAV.

[0026] Optionally, the execution process of step 3 comprises the following steps:

[0027] Step 3.1: merging the actions of agent 2 and agent 3 in time slot and the task size in the current environment to obtain the state of agent 1 in time slot and inputting it into the Actor network to obtain the action of agent 1 in time slot ;

[0028] Step 3.2: merging the actions of agent 1 and agent 3 in time slot and the task size in the current environment to obtain the state of agent 2 in time slot and inputting it into the Actor network to obtain the action of agent 2 in time slot ;

[0029] Step 3.3: merging the actions of agent 1 and agent 2 in time slot and the task size in the current environment to obtain the state of agent 2 in time slot And input it into its Actor network to obtain agent 3 in Actions in time slots .

[0030] Optionally, step 4 involves merging all agent actions. and merge action Enter the environment and receive a reward. and the next environmental state .

[0031] Optionally, the execution process of step 5 includes the following steps:

[0032] Step 5.1: Transfer the tuples Input agent 1's experience pool ;

[0033] Step 5.2: Transfer the tuples Input agent 2's experience pool ;

[0034] Step 5.3: Transfer the tuples Input agent 2's experience pool .

[0035] Optionally, each updating agent in step 6 contains four neural networks: an Actor network with parameters as follows: ; Critic network, parameters are ;Target Actor network, parameters are ; Target Critic network, parameters are ;

[0036] Actor networks are used to explore environmental states. And generate predicted actions Further evaluation of actions through the Critic network. Value and output Calculate the agent's state Next action The cumulative expected reward is shown in the following formula:

[0037]

[0038] in, Represents the state transition probability Get the next state , Indicates the state Execute action The obtained action value, Indicates the state Performing an action The reward value obtained;

[0039] The parameter The target network The value can be represented as:

[0040]

[0041] By calculating the mean square error of the estimated value and the actual value, the loss function of the Critic network is represented as:

[0042]

[0043] The target network adopts a soft update strategy to update the parameters, that is, the parameters of the target network will gradually approach the parameters of the online network to a certain extent during each update to reduce the mutation caused by the update; Specifically, the parameters in the target Actor network and the target Critic network are updated by iteration And :

[0044]

[0045] .

[0046] The application provides a deep deterministic gradient policy computing unloading method based on alternating iterative optimization, establishes a task unloading decision model in a collaborative application scenario of Internet of Things devices and edge computing server nodes, and models terminal device unloading strategies, edge node resource allocation and position deployment variables coupled in the model as three agents, designs corresponding Markov processes, and solves by alternating training and learning of the agents. The optimization goal of the application is to reduce the delay and energy consumption weighted sum of the system. Compared with the prior art, the problem of falling into a local optimal solution due to long exploration action space dimension in the traditional deep reinforcement learning algorithm is solved, fast unloading decision in a dynamic computing task unloading environment is realized, and user experience is improved. BRIEF DESCRIPTION OF DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0048] Figure 1 is a step flowchart of a deep deterministic gradient policy computing unloading method based on alternating iterative optimization of the application.

[0049] Figure 2 is an agent action flow diagram of a deep deterministic gradient policy calculation offloading method based on alternating iterative optimization of the present application.

[0050] Figure 3 is an environmental situation diagram of an embodiment of the present application.

[0051] Figure 4 is a relationship diagram of the reward value changing with the number of training in an embodiment of the present application.

[0052] Figure 5 is a time delay and energy consumption weighted sum cost comparison diagram of IDDPG and other three benchmark algorithms under different time slots in an embodiment of the present application. DETAILED DESCRIPTION

[0053] Embodiments of the present application are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below are exemplary and are intended to explain the present application, and cannot be understood as limiting the present application.

[0054] Referring to Figure 1 and Figure 2 , the present application provides a deep deterministic gradient policy calculation offloading method based on alternating iterative optimization, comprising the following steps:

[0055] S1: modeling the terminal device offloading strategy, edge node resource allocation and location deployment variables in the application scenario as three agents, and designing corresponding Markov processes;

[0056] S2: initializing the observation environment state space of each agent;

[0057] S3: the agent alternately performs actions according to the state space;

[0058] S4: merging all agent actions to input the environment to obtain the reward value and the next environment state under the current state;

[0059] S5: processing the experience pool of the agent;

[0060] S6: extracting N experiences to update the neural network of the agent.

[0061] The Interactive Deep Deterministic Policy Gradient (IDDPG) method for computational offloading in alternating iterative optimization described in this invention is characterized by decomposing the coupled problems of task offloading decision-making, resource allocation, and UAV location deployment in collaborative application scenarios between IoT devices and edge nodes into three agents that are alternately trained and solved. The specific steps are further explained below:

[0062] In step S1, the terminal device unloading decision, UAV resource allocation, and location deployment are modeled as three intelligent agents for alternating decision optimization; the state spaces of the three intelligent agents... for:

[0063]

[0064] in, Represents intelligent agents In the time slot The environmental state of the constructed model was observed, where For terminal devices in time slots The resulting set of computational tasks; Indicates that agent 1 is in the time slot Actions at that time; Indicates that agent 1 is in the time slot Actions at that time; Indicates that agent 2 is in the time slot Actions at that time; Indicates that agent 2 is in the time slot Actions at that time; This indicates that agent 3 is in the time slot. Actions at that time; This indicates that agent 3 is in the time slot. Actions at that time.

[0065] Action space of three agents The set of unloading decisions for the terminal device, the set of drone main frequency allocation coefficients, and the location coordinates of the drone are represented as follows:

[0066]

[0067] in, For terminal devices binary unloading strategy variable, A value of 1 indicates that the terminal device offloads the task to the edge node for computation; otherwise, the task will be computed locally. This represents the proportion of CPU clock speed coefficients allocated to terminal devices from edge nodes, i.e., the resource allocation strategy. This indicates the location coordinates of the edge nodes, i.e., the location deployment strategy.

[0068] Furthermore, the reward function policy after merging the agent's actions in step S1 is as follows:

[0069]

[0070] in, Indicates the agent's state Next action The expected reward value obtained; For time slots The system energy consumption, For time slots System latency, This is a constant representing the penalty value obtained when the drone flies out of the environmental area;

[0071] Correspondingly, and The calculation formula is as follows:

[0072]

[0073]

[0074] in, Indicates the terminal device in the time slot The task transmission rate, Indicates the CPU clock speed of the edge node. This represents the number of cycles an edge node processes for each 1-bit task. This represents the CPU capacitance coefficient of the drone.

[0075] Step S2: Initialize the environment and the agent's observation state space. and action space .

[0076] In step S3, the agent alternately executes actions according to the state space, specifically including the following steps:

[0077] S301: Will The actions of time-slot agents 2 and 3, combined with the task size in the current environment, are used to obtain the action of agent 1 in the time slot. The state of the time slot And input it into its Actor network to obtain agent 1 in Actions in time slots ;

[0078] S302: Will The actions of agent 1 and agent 3 in the time slot are combined with the task size in the current environment to obtain the action of agent 2. The state of the time slot And input it into its Actor network to obtain agent 2 in Actions in time slots ;

[0079] S303: Will The actions of agent 1 and agent 2 in a time slot are combined with the task size in the current environment to obtain the action of agent 2 in the time slot. The state of the time slot And input it into its Actor network to obtain agent 3 in Actions in time slots ;

[0080] In step S4, all agent actions are merged. and merge action Enter the environment and receive a reward. and the next environmental state .

[0081] Step S5, which processes the agent's experience pool, includes the following steps:

[0082] S501: tuple Input agent 1's experience pool ;

[0083] S502: tuple Input agent 2's experience pool ;

[0084] S503: tuple Input agent 2's experience pool ;

[0085] Step S6, the process of extracting N experience points to update the agent's neural network is as follows:

[0086] S601: Using the formula Calculate the loss function for Critic and update the parameters. ;

[0087] S602: Calculate the policy gradient of the Actor network using the following formula and update the parameters. ;

[0088]

[0089] S603: Soft update of the target Actor network and the target Critic network.

[0090] Furthermore, in step S6, each agent contains four neural networks: an Actor network (with parameters of...). ), Critic network (parameters are) ), target Actor network (parameters are) ), target Critic network (parameters are) ).

[0091] Actor networks are used to explore environmental states. And generate predicted actions Further evaluation of actions through the Critic network. Value and output Calculate the agent's state Next action The cumulative expected reward is shown in the following formula. This method helps optimize the Actor network, enabling the agent to better adapt to the environment and obtain higher rewards.

[0092]

[0093] in, Represents the state transition probability Get the next state , Indicates the state Execute action The obtained action value, Indicates the state Execute action The reward value obtained. Then the parameter Target network The value can be represented as:

[0094]

[0095] By calculating the mean square error between the estimated and actual values, the loss function of the Critic network is expressed as:

[0096]

[0097] The target network employs a soft update strategy to update its parameters. This means that with each update, the target network's parameters gradually approach the online network's parameters to minimize abrupt changes caused by the update. Specifically, the parameters in the target Actor network and the target Critic network are updated iteratively. and :

[0098]

[0099] .

[0100] Please see Figures 3 to 5Furthermore, the advantages of the present invention are illustrated through specific embodiments and comparisons with other methods.

[0101] like Figure 3 As shown, this technology is specifically implemented in monitoring scenarios such as forest fire prevention and firefighting, with the following environmental conditions:

[0102] (1) Calculation environment: Assume a fire has occurred in a forest area, and rescue personnel are needed to carry out firefighting and rescue work. The firefighters are unevenly distributed in this area. An IoT monitoring device and a drone equipped with a miniature edge computing server, serving as an aerial edge node.

[0103] (2) Computational task: Assume the service cycle of the entire system is divided into... Each time slot generates a computationally intensive task for the terminal device. The data that needs to be analyzed is transmitted wirelessly to the drone for calculation. The drone then transmits the analysis results back to the emergency command center for fire situation assessment and to provide information support to rescue personnel.

[0104] (3) Task unloading strategy: The unloading strategy of the terminal device is a binary unloading strategy, that is, the task cannot be divided and can only be unloaded to the drone computing or local computing.

[0105] (4) Resource allocation strategy: A UAV needs to handle the computational tasks of offloading multiple terminal devices simultaneously. Therefore, the UAV CPU needs to allocate resources reasonably for computation. The offloading strategy is as follows: .

[0106] (5) UAV location deployment strategy: UAVs need to receive computing tasks from multiple terminal devices at the same time. Different UAV location deployments will affect the communication quality between the terminal devices and the UAVs.

[0107] The execution process of the specific embodiment can be referred to steps S1 to S6.

[0108] Figure 4 The graph illustrates the relationship between the reward value of the IDPG method of this invention and the number of training iterations. As shown in the figure, in the early stages of training, the agent continuously explores, tries and fails, and accumulates experience based on changes in the environmental state, resulting in some fluctuations in the early curve. Furthermore, as the number of training iterations increases, the weight parameters of the Actor and Critic networks are gradually optimized, thus the reward gradually stabilizes in the later stages. The IDPG method converges after approximately 20 training iterations.

[0109] Figure 5 The paper presents a comparison of the latency, energy consumption weighted sum, and cost of the IDPG method of this invention with three other benchmark algorithms under different time slots. The corresponding baseline algorithm names are as follows:

[0110] (1) All local: All computing tasks generated by the terminal device are computed locally, and the drone is deployed in the center of the environment area.

[0111] (2) Greed: Greedy strategy. The terminal device selects the strategy with the lower cost based on the cost of local computation and offload computation, and deploys the drone in the center of the environment area.

[0112] (3) DDPG: Traditional deep deterministic gradient strategy algorithm, with the same parameters and environment as IDPG for Actor and Critic network layers.

[0113] Depend on Figure 5 It is evident that the All local algorithm has the highest cost, while the IDPG method of this invention has the lowest cost. Compared to the All local, Greed, and DDPG algorithms, the IDPG method reduces the average system cost by approximately 14.5%, 2%, and 2.2%, respectively. This is because the All local algorithm places all tasks on a local computed scale; however, the limited computing power of the terminal device leads to increased overall computational latency and energy consumption. The DDPG algorithm, due to its excessively large action space dimension, cannot be effectively fitted by the neural network, thus leading to local optima in the unloading decision. The Greed algorithm, by employing local exploration and lacking comprehensive observation of environmental information, also falls into local optima. The IDPG method, by constructing three agents, reduces the dimensionality of the traditional DDPG action space, thereby decreasing the probability of falling into local optima and resulting in more stable performance.

[0114] The above description discloses only one preferred embodiment of the present invention, and should not be construed as limiting the scope of the present invention. Those skilled in the art will understand that all or part of the processes of the above embodiments can be implemented, and equivalent changes made in accordance with the claims of the present invention are still within the scope of the invention.

Claims

1. A method for computing offloading based on alternating iterative optimization of deep deterministic gradient policy, characterized in that, The method comprises the following steps: Step 1: modeling the terminal device unloading strategy, edge node resource allocation and location deployment variables in the application scene into three agents, and designing corresponding Markov processes; In step 1, the terminal device offloading decision, the UAV resource allocation and the location deployment are respectively modeled as three agents for alternately decision optimization; the state space of the three agents is is: ； wherein, represents an agent at a time slot an environment state of the constructed model is observed, wherein is a terminal device at a time slot a set of computing tasks generated; represents an action of the agent 1 at a time slot ; represents an action of the agent 2 at a time slot ; represents an action of the agent 3 at a time slot ; Action space of three agents The unloading decision set of the terminal device, the main frequency allocation coefficient set of the unmanned aerial vehicle, and the position coordinates of the unmanned aerial vehicle are respectively represented as follows: ； wherein, is a binary offloading policy variable for the terminal device is a binary offloading policy variable for the terminal device equals 1 if the terminal device offloads the task to the edge node for computation, otherwise the task is computed locally; represents a CPU frequency coefficient proportion allocated by the edge node to the terminal device, i.e., a resource allocation policy; represents a location coordinate of the edge node, i.e., a location deployment policy; Step 2: initializing the observation environment state space of each agent; Step 3: the agents alternately execute actions according to the state space; The execution process of step 3 comprises the following steps: Step 3.1: Compute the state of agent 1 at time slot t + 1 from the state of agent 1 at time slot t and the actions of agents 2 and 3 at time slot t. The action of agent 1 at time slot t + 1 and the task size in the current environment of agent 1 at time slot t + 1 are obtained by merging the action of agent 2 and agent 3 at time slot t and the task size in the current environment of agent 3 at time slot t. The state of agent 1 at time slot t + 1 and input its Actor network to obtain the action of agent 1 at time slot t + 1 ;​ Step 3.2: [The text appears to be incomplete and contains several grammatical errors. A more accurate translation would require the full context.] The actions of agent 1 and agent 3 in the time slot are combined with the task size in the current environment to obtain the action of agent 2. The state of the time slot And input it into its Actor network to obtain agent 2 in Actions in time slots ; Step 3.3: [The text appears to be incomplete and contains several grammatical errors. A more accurate translation would require the full context.] The actions of agent 1 and agent 2 are combined with the task size in the current environment to obtain agent 3. The state of the time slot And input it into its Actor network to obtain agent 3 in Actions in time slots ; Step 4: inputting all agent actions into the environment to obtain a reward value and a next environment state under the current state; Step 5: processing the experience pool of the agent; Step 6: extracting N experiences to update the neural network of the agent.

2. The deep deterministic gradient policy calculation unloading method based on alternating iterative optimization according to claim 1, wherein a reward function strategy after the action merging of the agent is as follows:

3. The deep deterministic gradient policy calculation unloading method based on alternating iterative optimization according to claim 2, wherein ； wherein, represents the agent performing an action in state and obtaining a reward value is the system energy consumption of time slot , is the system latency of time slot , is a constant punishment value obtained when the UAV flies out of the environment region. Correspondingly, and The calculation formula is as follows: ； ； in, Indicates the terminal device in the time slot The task transmission rate, Indicates the CPU clock speed of the edge node. This represents the number of cycles an edge node processes for each 1-bit task. This represents the CPU capacitance coefficient of the drone.

4. The deep deterministic gradient policy calculation unloading method based on alternating iterative optimization according to claim 3, wherein The execution process of Step 4, specifically merging all agent actions and inputting the environment to get the reward and the next state of the environment . The execution process of step 5 comprises the following steps:

5. The deep deterministic gradient policy calculation unloading method based on alternating iterative optimization according to claim 4, wherein Step 5.1: Create a tuple Input the experience pool of agent 1 ; Step 5.2: The tuple Input the experience pool of the agent 2 ; Step 5.3: The tuple Input the experience pool of the agent 3 . By calculating the mean square error of the estimated value and the actual value, the loss function of the Critic network is represented as: Each updated agent of Step 6 contains 4 neural networks: the Actor network with parameters ; the Critic network with parameters ; the target Actor network with parameters ; Target Critic network, parameterized by ; Actor network is used to explore the environment state and generate predicted actions The value of the action is further evaluated by the Critic network and output The cumulative expected reward of the agent performing the action in the state is calculated as follows: ； wherein, denotes the state transition probability get the next state , denotes the action performed in state perform the action the resulting action value, denotes the action performed in state perform the action the resulting reward value; Parameters Target network The value can be expressed as: ； ​ ； The target network adopts a soft update strategy to update the parameters, that is, each time the parameters of the target network are gradually close to the parameters of the online network to a certain extent to reduce the mutation caused by the update; specifically, the parameters in the target Actor network and the target Critic network are updated in an iterative manner and : ； 。

Images