Depth deterministic gradient strategy calculation unloading method based on alternate iterative optimization

Abstract

The invention relates to the technical field of deep reinforcement learning, in particular to a depth deterministic gradient strategy calculation unloading method based on alternate iterative optimization, and the method comprises the steps: building a task unloading decision model in an Internet of Things equipment and edge calculation server node cooperative application scene; a terminal equipment unloading strategy, edge node resource allocation and position deployment variables coupled in the model are modeled into three agents, a corresponding Markov process is designed, and the agents are used for alternately training, learning and solving, the optimization objective of the method is to reduce the time delay and energy consumption weighted sum of the system, and compared with the prior art, the method has the advantages that the optimization efficiency is improved, and the cost is reduced. The problem that a traditional deep reinforcement learning algorithm falls into a local optimal solution due to long space dimension of exploration actions is solved, and rapid unloading decision in a dynamic computing task unloading environment is realized, so that user experience is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of deep reinforcement learning, and in particular to a deep deterministic gradient strategy calculation offloading method based on alternating iterative optimization. Background Art

[0002] With the rapid growth and application of IoT devices, such as smartphones, sensors, and wearable devices, a large number of computationally intensive tasks need to be transferred from IoT devices to cloud servers for execution. However, the transfer process of these intensive tasks involves a large amount of data transmission, which will lead to high latency in IoT applications.

[0003] The emergence of mobile edge computing (MEC) can effectively alleviate this challenge. Mobile edge computing can migrate complex tasks from computationally intensive tasks in IoT devices to edge servers, thereby providing computing services for IoT devices. By utilizing the computing and decision-making capabilities of edge servers and cloud servers to reduce computing latency and energy, the user experience quality can be improved.

[0004] However, the computing resources of edge node servers are limited. Therefore, in edge computing systems, the task offloading strategy of terminal devices, the location deployment of edge nodes, and resource allocation are all issues that need to be solved. The use of traditional intelligent optimization algorithms or deep reinforcement learning algorithms to solve and explore the action space dimension is long, resulting in long iteration time or easy to fall into the local optimal solution problem, which does not meet the requirements of low latency and high service quality of edge computing. Summary of the invention

[0005] The purpose of the present invention is to provide a deep deterministic gradient strategy computation offloading method based on alternating iterative optimization, aiming to solve the problem that traditional deep reinforcement learning methods fall into local optimal solutions due to the long dimension of exploring action space, and to realize fast computational task offloading decision in a dynamic computational task environment.

[0006] To achieve the above object, the present invention provides a deep deterministic gradient strategy calculation unloading method based on alternating iterative optimization, comprising the following steps:

[0007] Step 1: Model the terminal device offloading strategy, edge node resource allocation and location deployment variables in the application scenario as three intelligent agents and design the corresponding Markov process;

[0008] Step 2: Initialize the observation environment state space of each agent;

[0009] Step 3: The agent performs actions alternately according to the state space;

[0010] Step 4: Combine all agent actions and input them into the environment to obtain the reward value in the current state and the next environment state.

[0011] Step 5: Process the agent’s experience pool;

[0012] Step 6: Extract N experiences to update the agent’s neural network.

[0013] Optionally, in step 1, the terminal device unloading decision, drone resource allocation and location deployment are modeled as three agents for alternating decision optimization; the state space S of the three agents is:

[0014]

[0015] Among them, S i [t] represents the environment state of the constructed model observed by agent i at time slot t, where D t A is the set of computing tasks generated by the terminal device in time slot t; 1 (t) represents the action of agent 1 at time slot t; A 1 (t-1) represents the action of agent 1 at time slot t-1; A 2 (t) represents the action of agent 2 at time slot t; A 2 (t-1) represents the action of agent 2 at time slot t-1; A 3 (t) represents the action of agent 3 at time slot t; A 3 (t-1) represents the action of agent 3 at time slot t-1.

[0016] Optionally, the action spaces A[t] of the three agents are the unloading decision set of the terminal device, the main frequency allocation coefficient set of the drone, and the location coordinates of the drone, respectively, and are expressed as follows:

[0017]

[0018] Wherein, b is the binary offloading policy variable of terminal device i. b equals 1, indicating that the terminal device offloads the task to the edge node for calculation, otherwise the task is calculated locally; λ represents the ratio of the CPU main frequency coefficient allocated by the edge node to the terminal device, i.e., the resource allocation strategy; q represents the location coordinates of the edge node, i.e., the location deployment strategy.

[0019] Optionally, the reward function strategy after the agent's actions in step 1 are combined as follows:

[0020]

[0021] Where r[t] represents the expected reward value obtained by the agent when performing action A[t] in state S[t]; E t is the system energy consumption in time slot t, T tis the system delay of time slot t, and punish is the penalty value constant obtained when the UAV flies out of the environmental area;

[0022] Correspondingly, T t and E t The calculation formula is as follows:

[0023]

[0024] Among them, R t represents the task transmission rate of the terminal device in time slot t, f uav Indicates the CPU main frequency of the edge node, C uav Indicates the number of cycles that the edge node processes for each 1-bit task, K uav Indicates the CPU capacitance coefficient of the drone.

[0025] Optionally, the execution process of step 3 includes the following steps:

[0026] Step 3.1: Combine the actions of Agent 2 and Agent 3 in time slot t-1 with the task size in the current environment to obtain the state S of Agent 1 in time slot t 1 (t)=[D t ,A 2 (t-1),A 3 (t-1)] and input its Actor network to obtain the action of agent 1 at time slot t

[0027] Step 3.2: Combine the actions of Agent 1 and Agent 3 in time slot t-1 with the task size in the current environment to obtain the state S of Agent 2 in time slot t 2 (t)=[D t ,A 1 (t),A 3 (t-1)] and input its Actor network to obtain the action of Agent 2 at time slot t

[0028] Step 3.3: Combine the actions of Agent 1 and Agent 2 in time slot t-1 and the task size in the current environment to obtain the state S of Agent 2 in time slot t 3 (t)=[D t ,A 1 (t),A 2 (t-1)] and input its Actor network to obtain the action of agent 3 at time slot t

[0029] Optionally, the execution process of step 4 is to merge all agent actions into A(t) = [A 1 (t),A 2 (t),A3 (t)] and input the merged action A(t) into the environment to obtain the reward r and the next environment state S'.

[0030] Optionally, the execution process of step 5 includes the following steps:

[0031] Step 5.1: Substitute the tuple (S 1 (t),A 1 (t),r,S') input the experience pool D of agent 1 1 ;

[0032] Step 5.2: Substitute the tuple (S 2 (t),A 2 (t),r,S') input into the experience pool D of agent 2 2 ;

[0033] Step 5.3: Substitute the tuple (S 3 (t),A 3 (t),r,S') input into the experience pool D of agent 2 3 .

[0034] Optionally, each update agent in step 6 contains 4 neural networks: Actor network with parameters θ μ ; Critic network, parameter is θ Q ; Target Actor network, parameter is θ μ' ; Target Critic network, parameter is θ Q' ;

[0035] The Actor network is used to explore the environment state s and generate a predicted action a. The Critic network further evaluates the value of action a and outputs Q(s,a|θ Q ), the cumulative expected reward of the agent performing action a in state s is calculated as follows:

[0036]

[0037] Among them, s'~P represents the state transition probability P to get the next state s', Q'(s',a') represents the action value obtained by performing action a' in state s', and r represents the reward value obtained by performing action a' in state s';

[0038] Then the parameter θ Q' The target network Q value can be expressed as:

[0039] y=r+γQ'(s',μ'(s'|θ μ' )|θ Q' )

[0040] By calculating the mean square error between the estimated value and the actual value, the loss function of the Critic network is expressed as:

[0041] L(θ Q )=E μ' [(Q(s,a|θ Q )-y 2 )]

[0042] The target network uses a soft update strategy to update parameters, that is, each time the update is performed, the parameters of the target network will gradually approach the parameters of the online network to a certain extent to reduce the mutation caused by the update; specifically, the parameters θ in the target Actor network and the target Critic network are updated iteratively. μ' and θ Q' :

[0043] θ μ' ←ω 1 θ μ +(1-ω 1 )θ μ'

[0044] θ Q' ←ω 2 θ Q +(1-ω 2 )θ Q' .

[0045] The present invention provides a deep deterministic gradient strategy computing offloading method based on alternating iterative optimization, establishes a task offloading decision model in a collaborative application scenario of Internet of Things devices and edge computing server nodes, and models the terminal device offloading strategy, edge node resource allocation and location deployment variables coupled in the model as three intelligent agents, designs a corresponding Markov process, and uses intelligent agents to alternately train and learn to solve. The optimization goal of the present invention is to reduce the system's latency and weighted sum of energy consumption. Compared with the prior art, it solves the problem that traditional deep reinforcement learning algorithms fall into local optimal solutions due to the long dimension of exploring action space, and realizes fast offloading decisions in a dynamic computing task offloading environment, thereby improving user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0047] Figure 1 It is a flowchart of the steps of a deep deterministic gradient strategy calculation unloading method based on alternating iterative optimization of the present invention.

[0048] Figure 2 It is a schematic diagram of the intelligent agent action flow of the deep deterministic gradient strategy calculation unloading method based on alternating iterative optimization of the present invention.

[0049] Figure 3 It is a schematic diagram of the environmental conditions of a specific embodiment of the present invention.

[0050] Figure 4 It is a schematic diagram of the relationship between the reward value and the number of training times in a specific embodiment of the present invention.

[0051] Figure 5 It is a schematic diagram comparing the weighted sum of delay and energy consumption of IDDPG and other three benchmark algorithms at different time slots according to a specific embodiment of the present invention. DETAILED DESCRIPTION

[0052] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0053] See also Figure 1 and Figure 2 The present invention provides a deep deterministic gradient strategy calculation unloading method based on alternating iterative optimization, comprising the following steps:

[0054] S1: Model the terminal device offloading strategy, edge node resource allocation and location deployment variables in the application scenario as three intelligent agents and design the corresponding Markov process;

[0055] S2: Initialize the observation environment state space of each agent;

[0056] S3: The agent performs actions alternately according to the state space;

[0057] S4: Combine all agent actions and input them into the environment to obtain the reward value in the current state and the next environment state;

[0058] S5: Processing agent’s experience pool;

[0059] S6: Extract N experiences to update the agent’s neural network.

[0060] The interactive deep deterministic policy gradient calculation offloading (IDDPG) method for alternating iterative optimization described in the present invention is characterized in that the task offloading decision, resource allocation and drone location deployment coupling problems in the collaborative application scenario of IoT devices and edge nodes are decomposed into three intelligent agents for alternating training and solving, which are further explained in combination with specific steps below:

[0061] In step S1, the terminal device unloading decision, drone resource allocation and location deployment are modeled as three agents for alternating decision optimization; the state space S of the three agents is:

[0062]

[0063] Among them, S i [t] represents the environment state of the constructed model observed by agent i at time slot t, where D t A is the set of computing tasks generated by the terminal device in time slot t; 1 (t) represents the action of agent 1 at time slot t; A 1 (t-1) represents the action of agent 1 at time slot t-1; A 2 (t) represents the action of agent 2 at time slot t; A 2 (t-1) represents the action of agent 2 at time slot t-1; A 3 (t) represents the action of agent 3 at time slot t; A 3 (t-1) represents the action of agent 3 at time slot t-1.

[0064] The action spaces A[t] of the three agents are the unloading decision set of the terminal equipment, the main frequency allocation coefficient set of the drone, and the location coordinates of the drone, respectively, which are expressed as follows:

[0065]

[0066] Wherein, b is the binary offloading policy variable of terminal device i. b equals 1, indicating that the terminal device offloads the task to the edge node for calculation, otherwise the task is calculated locally; λ represents the ratio of the CPU main frequency coefficient allocated by the edge node to the terminal device, i.e., the resource allocation strategy; q represents the location coordinates of the edge node, i.e., the location deployment strategy.

[0067] Furthermore, the reward function strategy after the action combination of the agent in step S1 is as follows:

[0068]

[0069] Where r[t] represents the expected reward value obtained by the agent when performing action A[t] in state S[t]; E t is the system energy consumption in time slot t, T t is the system delay of time slot t, and punish is the penalty value constant obtained when the UAV flies out of the environmental area;

[0070] Correspondingly, T t and E t The calculation formula is as follows:

[0071]

[0072] Among them, R t represents the task transmission rate of the terminal device in time slot t, f uav Indicates the CPU main frequency of the edge node, C uav Indicates the number of cycles that the edge node processes for each 1-bit task, K uav Indicates the CPU capacitance coefficient of the drone.

[0073] Step S2: Initialize the environment, the agent's observation state space S and action space A.

[0074] In step S3, the agent performs actions alternately according to the state space, which specifically includes the following steps:

[0075] S301: Combine the actions of Agent 2 and Agent 3 in time slot t-1 and the task size in the current environment to obtain the state S of Agent 1 in time slot t 1 (t)=[D t ,A 2 (t-1),A 3 (t-1)] and input its Actor network to obtain the action of agent 1 at time slot t

[0076] S302: Combine the actions of Agent 1 and Agent 3 in time slot t-1 with the task size in the current environment to obtain the state of Agent 2 in time slot t And input its Actor network to get the action of agent 2 in time slot t

[0077] S303: Combine the actions of Agent 1 and Agent 2 in time slot t-1 and the task size in the current environment to obtain the state S of Agent 2 in time slot t 3 (t)=[D t ,A 1 (t),A 2 (t-1)] and input its Actor network to obtain the action of agent 3 at time slot t

[0078] In step S4, all agent actions are combined into A(t) = [A 1 (t),A 2 (t),A 3 (t)] and input the merged action A(t) into the environment to obtain the reward r and the next environment state S'.

[0079] Step S5 processes the agent's experience pool, including the following steps:

[0080] S501: The tuple (S 1 (t),A 1 (t),r,S') input the experience pool D of agent 1 1 ;

[0081] S502: The tuple (S 2 (t),A 2 (t),r,S') input into the experience pool D of agent 2 2 ;

[0082] S503: The tuple (S 3 (t),A 3 (t),r,S') input into the experience pool D of agent 2 3 ;

[0083] Step S6: The process of extracting N experiences to update the neural network of the agent is as follows:

[0084] S601: Using formula Calculate the Critic's loss function and update the parameter θ Q ;

[0085] S602: Use the following formula to calculate the policy gradient of the Actor network and update the parameter θ μ ;

[0086]

[0087] S603: Soft update the target Actor network and the target Critic network.

[0088] Furthermore, in step S6, each agent contains four neural networks: Actor network (with parameters θ μ ), Critic network (parameter is θ Q ), the target Actor network (with parameters θ μ' ), target critic network (parameter is θ Q' ).

[0089] The Actor network is used to explore the environment state s and generate a predicted action a. The Critic network further evaluates the value of action a and outputs Q(s,a|θ Q ), the cumulative expected reward of the agent performing action a in state s is calculated as shown in the following formula. This method helps to optimize the Actor network, enabling the agent to better adapt to the environment and obtain higher rewards.

[0090]

[0091] Among them, s'~P represents the state transition probability P to get the next state s', Q'(s',a') represents the action value obtained by performing action a' in state s', and r represents the reward value obtained by performing action a' in state s'. Then the parameter θ Q' The target network Q value can be expressed as:

[0092] y=r+γQ'(s',μ'(s'|θ μ' )|θ Q' )

[0093] By calculating the mean square error between the estimated value and the actual value, the loss function of the Critic network is expressed as:

[0094] L(θ Q )=E μ' [(Q(s,a|θ Q )-y 2 )]

[0095] The target network uses a soft update strategy to update parameters, that is, each time the update is performed, the parameters of the target network will gradually approach the parameters of the online network to a certain extent to reduce the mutation caused by the update. Specifically, the parameters θ in the target actor network and the target critic network are updated iteratively. μ' and θ Q' :

[0096] θ μ' ←ω 1 θ μ +(1-ω 1 )θ μ'

[0097] θ Q' ←ω 2 θ Q +(1-ω 2 )θ Q' .

[0098] See also Figures 3 to 5 The present invention also illustrates the advantages of the present invention through specific embodiments and comparison with other methods.

[0099] like Figure 3 As shown, it is specifically implemented in monitoring scenarios such as forest fire prevention and firefighting, and the environmental conditions are as follows:

[0100] (1) Computing environment: Assume that a fire breaks out in a forest area and rescue workers are needed to carry out firefighting and rescue work. N IoT monitoring devices and a drone are unevenly distributed in the area. The drone is equipped with a micro edge computing server as an aerial edge node.

[0101] (2) Computational tasks: Assuming that the service cycle of the entire system is divided into T time slots, the terminal device will generate a computationally intensive task D (the monitored data that needs to be analyzed) in each time slot, which will be sent wirelessly to the drone for calculation. The drone will then transmit the analysis results back to the emergency command center for fire disaster assessment and provide information support to rescue personnel.

[0102] (3) Task offloading strategy: The offloading strategy of the terminal device is a binary offloading strategy, that is, the task is indivisible and can only be offloaded to drone computing or local computing.

[0103] (4) Resource allocation strategy: A drone needs to handle computing tasks offloaded by multiple terminal devices at the same time. Since the drone CPU needs to reasonably allocate resources for computing, the offloading strategy is

[0104] (5) Drone location deployment strategy: Drones need to receive computing tasks from multiple terminal devices at the same time. Different drone location deployments will affect the communication quality between the terminal devices and the drones.

[0105] The execution process of the specific embodiment may refer to step S1 to step S6.

[0106] Figure 4 The relationship between the reward value of the IDDPG method of the present invention and the number of training times is shown. As can be seen from the figure, in the early stage of training, the agent continuously explores, tries and errors, and accumulates experience according to the changes in the environment state, so the early curve fluctuates. As the number of training times increases, the weight parameters of the Actor and Critic networks are also gradually optimized, so the later rewards gradually stabilize. After about 20 training times, the IDDPG method converges.

[0107] Figure 5 The delay and energy consumption weighted and cost comparison of the IDDPG method of the present invention and other three benchmark algorithms under different time slots are shown. The names of the corresponding baseline algorithms shown are as follows:

[0108] (1) All local: All computing tasks generated by the terminal devices are calculated locally, and the drone is deployed in the center of the environmental area.

[0109] (2) Greed: Greedy strategy: The terminal device selects the strategy with the lowest cost and deploys the drone in the center of the environment area based on the cost of local calculation and offload calculation.

[0110] (3) DDPG: Traditional deep deterministic gradient strategy algorithm. The Actor and Critic network layer parameters and environment are the same as IDDPG.

[0111] Depend on Figure 5 It can be seen that the cost of the All local algorithm is the highest, and the cost of the IDDPG method of the present invention is the lowest. Compared with the All local, Greed and DDPG algorithms, the average system cost of the IDDPG method is reduced by about 14.5%, 2%, and 2.2%, respectively. The reason is that the All local algorithm puts all tasks into local calculation, but the terminal device has poor computing power, which leads to increased overall computing delay and energy consumption. The DDPG algorithm cannot effectively fit the neural network due to the large dimension of the action space, and the offloading decision falls into the local optimal solution. The Greed algorithm also falls into the local optimal solution due to the use of local exploration and incomplete observation of environmental information. The IDDPG method reduces the dimensionality of the action space of the traditional DDPG by constructing three intelligent agents, thereby reducing the probability of falling into the local optimum, and the performance is relatively stable.

[0112] What is disclosed above is only a preferred embodiment of the present invention, and it certainly cannot be used to limit the scope of rights of the present invention. Ordinary technicians in this field can understand that all or part of the processes of the above embodiment and equivalent changes made according to the claims of the present invention still fall within the scope of the invention.

Claims

1. A deep deterministic gradient strategy calculation offloading method based on alternating iterative optimization, characterized in that: The following steps are involved: Step 1: Model the terminal device offloading strategy, edge node resource allocation and location deployment variables in the application scenario as three intelligent agents and design the corresponding Markov process; Step 2: Initialize the observation environment state space of each agent; Step 3: The agent performs actions alternately according to the state space; Step 4: Combine all agent actions and input them into the environment to obtain the reward value in the current state and the next environment state. Step 5: Process the agent’s experience pool; Step 6: Extract N experiences to update the agent’s neural network.

2. The deep deterministic gradient strategy calculation unloading method based on alternating iterative optimization according to claim 1, characterized in that: In step 1, the terminal device offloading decision, drone resource allocation and location deployment are modeled as three agents for alternating decision optimization; the state space S of the three agents is: Among them, S i [t] represents the environment state of the constructed model observed by agent i at time slot t, where D t is the set of computing tasks generated by the terminal device at time slot t; A1(t) represents the action of agent 1 at time slot t; A1(t-1) represents the action of agent 1 at time slot t-1; A2(t) represents the action of agent 2 at time slot t; A2(t-1) represents the action of agent 2 at time slot t-1; A3(t) represents the action of agent 3 at time slot t; A3(t-1) represents the action of agent 3 at time slot t-1.

3. The deep deterministic gradient strategy calculation unloading method based on alternating iterative optimization according to claim 2, characterized in that: The action spaces A[t] of the three agents are the unloading decision set of the terminal equipment, the main frequency allocation coefficient set of the drone, and the location coordinates of the drone, respectively, which are expressed as follows: Wherein, b is the binary offloading policy variable of terminal device i. b equals 1, indicating that the terminal device offloads the task to the edge node for calculation, otherwise the task is calculated locally; λ represents the ratio of the CPU main frequency coefficient allocated by the edge node to the terminal device, i.e., the resource allocation strategy; q represents the location coordinates of the edge node, i.e., the location deployment strategy.

4. The deep deterministic gradient strategy calculation unloading method based on alternating iterative optimization according to claim 3, characterized in that: The reward function strategy after the action combination of the agent in step 1 is as follows: Where r[t] represents the expected reward value obtained by the agent when performing action A[t] in state S[t]; E t is the system energy consumption in time slot t, T t is the system delay of time slot t, and punish is the penalty value constant obtained when the UAV flies out of the environmental area; Correspondingly, T t and E t The calculation formula is as follows: Among them, R t represents the task transmission rate of the terminal device in time slot t, f uav Indicates the CPU main frequency of the edge node, C uav Indicates the number of cycles that the edge node processes for each 1-bit task, K uav Indicates the CPU capacitance coefficient of the drone.

5. The deep deterministic gradient strategy calculation unloading method based on alternating iterative optimization according to claim 4, characterized in that: The execution process of step 3 includes the following steps: Step 3.1: Combine the actions of Agent 2 and Agent 3 in time slot t-1 with the task size in the current environment to obtain the state of Agent 1 in time slot t S1(t) = [D t ,A2(t-1),A3(t-1)] and input its Actor network to obtain the action of agent 1 in time slot t Step 3.2: Combine the actions of Agent 1 and Agent 3 in time slot t-1 with the task size in the current environment to obtain the state of Agent 2 in time slot t S2(t) = [D t ,A1(t),A3(t-1)] and input its Actor network to obtain the action of agent 2 in time slot t Step 3.3: Combine the actions of Agent 1 and Agent 2 in time slot t-1 and the task size in the current environment to obtain the state of Agent 2 in time slot t S3(t) = [D t ,A1(t),A2(t-1)] and input its Actor network to obtain the action of agent 3 in time slot t 6. The deep deterministic gradient strategy calculation unloading method based on alternating iterative optimization according to claim 5, characterized in that: The execution process of step 4 is to merge all agent actions A(t) = [A1(t), A2(t), A3(t)], and input the merged action A(t) into the environment to obtain the reward r and the next environment state S'.

7. The deep deterministic gradient strategy calculation unloading method based on alternating iterative optimization according to claim 6, characterized in that: The execution process of step 5 includes the following steps: Step 5.1: Input the tuple (S1(t), A1(t), r, S') into the experience pool D1 of agent 1; Step 5.2: Input the tuple (S2(t), A2(t), r, S') into the experience pool D2 of agent 2; Step 5.3: Input the tuple (S3(t), A3(t), r, S') into the experience pool D3 of agent 2.

8. The deep deterministic gradient strategy calculation unloading method based on alternating iterative optimization according to claim 7, characterized in that: Each update agent in step 6 contains 4 neural networks: Actor network with parameters θ μ ; Critic network, parameter is θ Q ; Target Actor network, parameter is θ μ '; Target Critic network, parameter is θ Q '; The Actor network is used to explore the environment state s and generate a predicted action a. The Critic network further evaluates the value of action a and outputs Q(s,a|θ Q ), the cumulative expected reward of the agent performing action a in state s is calculated as follows: Among them, s'~P represents the state transition probability P to get the next state s', Q'(s',a') represents the action value obtained by performing action a' in state s', and r represents the reward value obtained by performing action a' in state s'; Then the parameter θ Q The target network Q value can be expressed as: y=r+γQ'(s',μ'(s'|θ μ ')|θ Q ') By calculating the mean square error between the estimated value and the actual value, the loss function of the Critic network is expressed as: L(θ Q )=E μ '[(Q(s,a|θ Q )-y 2 )] The target network uses a soft update strategy to update parameters, that is, each time the update is performed, the parameters of the target network will gradually approach the parameters of the online network to a certain extent to reduce the mutation caused by the update; specifically, the parameters θ in the target Actor network and the target Critic network are updated iteratively. μ ' and θ Q ': i μ '←ω1θ μ +(1-ω1)θ μ 'th Q '←ω2θ Q +(1-ω2)θ Q '。

Images