Power internet of things intelligent scheduling method based on exploration enhanced depth deterministic strategy gradient algorithm

Abstract

The invention provides a power Internet of Things intelligent scheduling method based on an exploration enhanced depth deterministic strategy gradient algorithm, and the method comprises the steps: 1, constructing a multi-objective optimization model of a power Internet of Things system, and enabling the model to be used for minimizing the transportation cost and maximizing the customer satisfaction; and step 2, providing an exploration enhanced depth deterministic strategy gradient method. According to the method, the exploration capability enhanced network is introduced into the depth deterministic strategy gradient algorithm, so that the optimization performance of the algorithm is effectively improved, and the problem that the traditional algorithm is easy to fall into local optimum is successfully avoided. According to the method, the model established in the step 1 is optimized and solved by using the method in the step 2, so that the simultaneous optimization of the transportation cost and the user satisfaction of the power Internet of Things dispatching system is realized. According to the method, a new thought and technical support are provided for the development of the power Internet of Things, and meanwhile, a wide prospect is opened up for the application of the deep reinforcement learning algorithm in the field of intelligent scheduling of the power Internet of Things.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent Internet of Things, and in particular relates to an intelligent scheduling method for the power Internet of Things based on an exploration-enhanced deep deterministic policy gradient algorithm. Background Art

[0002] Against the backdrop of the rapid development of artificial intelligence, the Power Internet of Things (PoT), a core pillar of smart power management, is undergoing tremendous transformation. By integrating cutting-edge technologies such as next-generation information technology, the Internet of Everything (IoT), and artificial intelligence (AI), PoT systems have achieved a leap from traditional manual operations to digital, intelligent management. However, faced with increasingly complex logistics networks and diverse customer demands, PoT intelligent scheduling still faces significant challenges. First, modern logistics networks exhibit a highly complex topology, encompassing multiple levels of transportation nodes, multiple modes of transport, and multidimensional decision variables. The intertwining of dynamic factors such as transportation distance, cargo characteristics, carrying capacity, and timeliness requirements transforms logistics scheduling into a typical multi-objective optimization problem, requiring the optimal balance between reducing transportation costs and improving customer satisfaction. Second, traditional scheduling algorithms often suffer from poor adaptability to dynamic environments, low computational efficiency, and insufficient data utilization when handling such complex problems. Existing advanced deep reinforcement learning methods, such as the deep deterministic policy gradient algorithm, also suffer from insufficient multi-objective optimization capabilities and are prone to falling into local optima. Summary of the Invention

[0003] Purpose of the invention: The technical problem to be solved by the present invention is to address the deficiencies of the existing technology and provide a method for intelligent scheduling of the power Internet of Things based on an exploratory enhanced deep deterministic policy gradient algorithm, comprising the following steps:

[0004] Step 1: Establish a multi-objective optimization model for the power Internet of Things system;

[0005] Step 2: Establish an exploration-enhanced deep deterministic policy gradient algorithm to optimize and solve the model established in step 1.

[0006] Step 1 includes the following steps:

[0007] Step 1.1, establish a transportation cost model;

[0008] Step 1.2, establish a customer satisfaction model;

[0009] Step 1.3, establish a multi-objective optimization model for the power Internet of Things system.

[0010] Step 1.1 includes designing the following transportation cost model:

[0011] TC=TC fuel (d,w)+TCmaintent (d,w)+TC penalty (t) (1),

[0012] Where TC represents the transportation cost, TC fuel is the fuel cost, TC fuel =k1·d·w, where d is the transport distance, w is the vehicle load, and k1 is the fuel cost per unit distance and unit weight, which is set at 2 yuan / km·ton; TC maintent is the vehicle maintenance cost, TC maintent =k2·d·w, where k2 is the maintenance cost per unit distance, which is 0.4 yuan / km·ton;

[0013] TC penalty is the penalty cost, which depends on whether the transportation time t exceeds the stipulated time. If it exceeds, TC penalty =k3max(0,tt max ), where k3 is the penalty cost per unit time, which is 100 yuan / day, t max It is the maximum transportation time specified, which is 3 days.

[0014] Step 1.2 includes designing the following customer satisfaction model:

[0015] CS=w1·CS punctuality (t)+w2·CS integrity (d,w) (2),

[0016] Among them, CS represents customer satisfaction, w1 and w2 are weight coefficients; CS punctuality is the punctuality evaluation score, expressed as Among them, t expected is the customer's expected shipping time, which is 2 days. tolerance It is the shipping time deviation that customers can accept, and its value is 1 day; CS integrity is the integrity evaluation score, expressed as e is a natural constant.

[0017] Step 1.3 includes: The multi-objective optimization model is expressed as:

[0018]

[0019] Where st represents the constraint, F represents the objective function vector, t max represents the maximum transportation time, w max Indicates the maximum vehicle load.

[0020] Step 2 includes the following steps:

[0021] Step 2.1, construct the Markov decision process;

[0022] Step 2.2, design state, action and reward functions;

[0023] Step 2.3, establish an exploration-enhanced deep deterministic policy gradient algorithm.

[0024] Step 2.1 includes: In the path optimization of an intelligent vehicle, its position change is affected only by its previous position, remaining battery charge, and load, which conforms to the Markov property. That is, given the vehicle's current position and speed, as well as the next action to be taken (such as steering angle, acceleration, etc.), the vehicle's next movement can be guided without considering its past position and action. Therefore, the optimization problem in Equation (3) is modeled as a Markov decision process.

[0025] Step 2.2 includes:

[0026] Step 2.2.1, Design State: In the power Internet of Things system, the smart vehicle interacts with the environment as an intelligent agent and makes the best decision based on the acquired state information, which includes the location information, remaining power and load of the smart vehicle. t Defined as:

[0027] s t =(x t ,y t ,pr,w) (4),

[0028] Among them, x t and y t They represent the horizontal and vertical coordinates of the current position of the smart vehicle, and pr represents the remaining power of the vehicle;

[0029] Step 2.2.2: The intelligent vehicle’s action decision takes into account the vehicle’s steering angle and speed to ensure efficient route planning. Design action a t :

[0030] a t =(θ t ,v t ) (5),

[0031] Among them, θ t and v t They represent the steering angle and speed of the smart vehicle at time slot t respectively;

[0032] Step 2.2.3: The reward function is mainly used to guide the agent to make the best decision action based on the current state. Design the reward function r t :

[0033]

[0034] Among them, CS t is the customer satisfaction of the vehicle at time slot t, TC t is the transportation cost of the vehicle in time slot t.

[0035] Step 2.3 includes:

[0036] Step 2.3.1, the agent is set to in represents a parameterized exploration-enhanced policy network, express The collection of Q μ (s t ,a t θ Q ) represents the parameterized value network; meanwhile, the target strategy network Directly copied from the exploration enhancement strategy network, the target value network Q' μ (s t ,a t ;θ' Q ) is directly copied to the value network;

[0037] Step 2.3.2, explore the enhanced strategy network, the input state is s t , get the output action a t :

[0038]

[0039] σ t =σ0e -0.2t / T (8),

[0040] in, It means the mean is zero and the variance is Gaussian distribution function, T represents the total number of time slots for intelligent learning, σ0 represents the initial variance value; σ t As the time slot t increases, it continues to decrease. This setting helps the agent enhance exploration in the early stages of learning (generating a wider range of random action attempts through larger variance), and continuously reduces variance perturbations in the later stages of learning to enhance convergence (making action selection more stable through smaller variance);

[0041] When exploring the enhanced policy network update, randomly sample N from the experience replay pool batch Experience, the Monte Carlo method is used to calculate the gradient, assuming for The probability distribution of , the update process is as follows:

[0042]

[0043] in, represents the gradient of the exploration enhancement policy network, η π is the policy network learning rate, θ Q is the value network parameter;

[0044] Step 2.3.3, the value network minimizes the loss function L(θ Q ) to update:

[0045]

[0046] Among them, y t is the approximate target Q value generated by the target network based on random sampling experience, γ is the discount factor, is the predicted Q value of the target value network, is the target policy network, θ' Q is the target value network parameter;

[0047] The value network parameters are updated by gradient descent, and the loss function L(θ Q ) For the value network parameter θ Q The gradient of is calculated as follows:

[0048]

[0049] in, is the gradient of the value network;

[0050] Step 2.3.4: Use soft update to update.

[0051] In step 2.3.4, let τ represent the learning rate of the target network update, and the update process of the target policy network and the target value network are expressed as:

[0052]

[0053] θ' Q =τθ Q +(1-τ)θ' Q (15),

[0054] in, are the target policy network parameters.

[0055] This paper innovatively proposes an intelligent dispatching method for the power Internet of Things (IoT) based on an exploration-enhanced deep deterministic policy gradient algorithm. By introducing an exploration-enhanced network, this method effectively improves the algorithm's global search capabilities in complex solution spaces. This not only avoids local optima but also achieves global optimization of the IoT dispatching solution, ultimately providing users with higher-quality and more efficient logistics services.

[0056] Beneficial effects: The present invention breaks through the limitations of the traditional single-objective optimization framework by constructing a dynamic coupling model that minimizes transportation costs and maximizes customer satisfaction. Combined with the original exploration-enhanced strategy network, it effectively solves the problem of low exploration efficiency and easy to fall into local optimality caused by the deterministic actions output by the strategy network in the DDPG algorithm, and improves the convergence performance of the algorithm by 23%. At the same time, the method of the present invention can reduce the overall operating cost of the system by more than 25% and improve customer satisfaction by more than 22%. While improving the performance of the power Internet of Things system, its dynamic optimization framework can be expanded to other application fields such as power network optimization and smart logistics scheduling, providing relevant technical solutions for the industry's digital transformation and green and low-carbon development. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more apparent.

[0058] Figure 1 It is a flow chart of the method of the present invention.

[0059] Figure 2 Schematic diagram of the results of the embodiment of the present invention.

[0060] Figure 3 Schematic diagram of the results of the embodiment of the present invention.

[0061] Figure 4 Schematic diagram of the results of the embodiment of the present invention. DETAILED DESCRIPTION

[0062] like Figure 1 As shown, an embodiment of the present invention provides a method for intelligent scheduling of the power Internet of Things based on an exploratory enhanced deep deterministic policy gradient algorithm, comprising the following steps:

[0063] Step 1: Establish a multi-objective optimization model for the power Internet of Things system;

[0064] Step 2: Establish an exploration-enhanced deep deterministic policy gradient algorithm to optimize and solve the model established in step 1.

[0065] Step 1 includes the following steps:

[0066] Step 1.1, establish a transportation cost model;

[0067] Step 1.2, establish a customer satisfaction model;

[0068] Step 1.3, establish a multi-objective optimization model for the power Internet of Things system.

[0069] Step 1.1 includes designing the following transportation cost model:

[0070] TC=TC fuel (d,w)+TC maintent (d,w)+TC penalty (t) (1),

[0071] Where TC represents the transportation cost, TC fuel is the fuel cost, TC fuel =k1·d·w, where d is the transport distance, w is the vehicle load, and k1 is the fuel cost per unit distance and unit weight, which is set at 2 yuan / km·ton; TC maintent is the vehicle maintenance cost, TC maintent =k2·d·w, where k2 is the maintenance cost per unit distance, which is 0.4 yuan / km·ton;

[0072] TC penalty is the penalty cost, which depends on whether the transportation time t exceeds the stipulated time. If it exceeds, TC penalty =k3max(0,tt max ), where k3 is the penalty cost per unit time, which is 100 yuan / day, t max It is the maximum transportation time specified, which is 3 days.

[0073] Step 1.2 includes designing the following customer satisfaction model:

[0074] CS=w1·CS punctuality (t)+w2·CS integrity (d,w) (2),

[0075] Among them, CS represents customer satisfaction, w1 and w2 are weight coefficients; CS punctuality is the punctuality evaluation score, expressed as Among them, t expected is the customer's expected shipping time, which is 2 days. tolerance It is the shipping time deviation that customers can accept, and its value is 1 day; CS integrity is the integrity evaluation score, expressed as e is a natural constant.

[0076] Step 1.3 includes: The multi-objective optimization model is expressed as:

[0077]

[0078] Where st represents the constraint, F represents the objective function vector, t max represents the maximum transportation time, w max Indicates the maximum vehicle load.

[0079] Step 2 includes the following steps:

[0080] Step 2.1, construct the Markov decision process;

[0081] Step 2.2, design state, action and reward functions;

[0082] Step 2.3, establish an exploration-enhanced deep deterministic policy gradient algorithm.

[0083] Step 2.1 includes: In the path optimization of an intelligent vehicle, its position change is affected only by its previous position, remaining battery charge, and load, which conforms to the Markov property. That is, given the vehicle's current position and speed, as well as the next action to be taken (such as steering angle, acceleration, etc.), the vehicle's next movement can be guided without considering its past position and action. Therefore, the optimization problem in Equation (3) is modeled as a Markov decision process.

[0084] Step 2.2 includes:

[0085] Step 2.2.1, Design State: In the power Internet of Things system, the smart vehicle interacts with the environment as an intelligent agent and makes the best decision based on the acquired state information, which includes the location information, remaining power and load of the smart vehicle. t Defined as:

[0086] s t =(x t ,y t ,pr,w) (4),

[0087] Among them, x t and y t They represent the horizontal and vertical coordinates of the current position of the smart vehicle, and pr represents the remaining power of the vehicle;

[0088] Step 2.2.2: The intelligent vehicle’s action decision takes into account the vehicle’s steering angle and speed to ensure efficient route planning. Design action a t :

[0089] a t =(θ t ,v t ) (5),

[0090] Among them, θ t and v t They represent the steering angle and speed of the smart vehicle at time slot t respectively;

[0091] Step 2.2.3: The reward function is mainly used to guide the agent to make the best decision action based on the current state. Design the reward function r t :

[0092]

[0093] Among them, CS t is the customer satisfaction of the vehicle at time slot t, TC t is the transportation cost of the vehicle in time slot t.

[0094] Step 2.3 includes:

[0095] Step 2.3.1, the agent is set to in represents a parameterized exploration-enhanced policy network, express The collection of Q μ (s t ,a t θ Q ) represents the parameterized value network; meanwhile, the target strategy network Directly copied from the exploration enhancement strategy network, the target value network Q' μ (s t ,a t ;θ' Q ) is directly copied to the value network;

[0096] Step 2.3.2, explore the enhanced strategy network, the input state is s t , get the output action a t :

[0097]

[0098] σ t =σ0e -0.2t / T (8),

[0099] in, It means the mean is zero and the variance is Gaussian distribution function, T represents the total number of time slots for intelligent learning, σ0 represents the initial variance value; σ t As the time slot t increases, it continues to decrease. This setting helps the agent enhance exploration in the early stages of learning (generating a wider range of random action attempts through larger variance), and continuously reduces variance perturbations in the later stages of learning to enhance convergence (making action selection more stable through smaller variance);

[0100] When exploring the enhanced policy network update, randomly sample N from the experience replay pool batch Experience, the Monte Carlo method is used to calculate the gradient, assuming for The probability distribution of , the update process is as follows:

[0101]

[0102] in, represents the gradient of the exploration enhancement policy network, η π is the policy network learning rate, θ Q is the value network parameter;

[0103] Step 2.3.3, the value network minimizes the loss function L(θ Q ) to update:

[0104]

[0105] Among them, y t is the approximate target Q value generated by the target network based on random sampling experience, γ is the discount factor, is the predicted Q value of the target value network, is the target policy network, θ' Q is the target value network parameter;

[0106] The value network parameters are updated by gradient descent, and the loss function L(θ Q ) For the value network parameter θ Q The gradient of is calculated as follows:

[0107]

[0108] in, is the gradient of the value network;

[0109] Step 2.3.4: Use soft update to update.

[0110] In step 2.3.4, let τ represent the learning rate of the target network update, and the update process of the target policy network and the target value network are expressed as:

[0111]

[0112] θ' Q =τθ Q +(1-τ)θ' Q (15),

[0113] in, are the target policy network parameters.

[0114] In another specific embodiment of the present invention, a method for intelligent scheduling of the power Internet of Things based on an exploratory enhanced deep deterministic policy gradient algorithm is provided, comprising the following steps:

[0115] Step 1: Establish a multi-objective optimization model for the power Internet of Things system;

[0116] Step 2: Establish an exploration-enhanced deep deterministic policy gradient algorithm to optimize and solve the model established in step 1.

[0117] Step 1 includes the following steps:

[0118] Step 1.1, establish a transportation cost model;

[0119] For a certain task (d = 50 km, w = 2 tons, t = 3 days), the transportation cost TC = 240 yuan;

[0120] Step 1.2, establish a customer satisfaction model;

[0121] For a certain task (d=50 km, t=3 days, w=2 tons), the customer satisfaction is 95.7%.

[0122] Step 1.3, establish a multi-objective optimization model for the power Internet of Things system.

[0123] Step 2 includes the following steps:

[0124] Step 2.1, construction of Markov decision process;

[0125] Step 2.2, design of state, action, and reward functions, specifically includes: Based on the multi-objective optimization model in step 1.3, define the agent's state (vehicle position, vehicle battery level, and load), action (vehicle steering angle and speed), and reward function (the ratio of customer satisfaction to transportation cost);

[0126] Step 2.3, explore the construction of the enhanced deep deterministic policy gradient algorithm. (1) Parameter setting: the number of learning rounds of the agent is 1000, the learning rate is 0.0005, the number of randomly selected experiences is 20, the discount factor is 0.99, the range of the maximum steering angle of the vehicle is set to 30 degrees, the maximum speed of the vehicle is set to 30km / h, and the maximum number of time slots is 200. The learning ability of the agent is enhanced by the method of the present invention, so that it gradually converges to the optimal solution. (2) Training process: After 300 rounds of training, the algorithm gradually converges and realizes the optimal planning of vehicle paths in the power Internet of Things system. (3) Performance evaluation: The reward value of the agent increases with the increase of the number of training rounds ( Figure 2 , horizontal axis: number of training rounds, vertical axis: reward value), indicating that it has learned a better route, effectively reduced transportation costs and improved user satisfaction.

[0127] For the case of 10 tasks, the reward value from the first round to the 1000th round increased by 725 from -11, indicating that the agent has continuously learned better actions, thereby optimizing the transportation path. Compared with the random method, the method of the present invention reduces the vehicle transportation cost by 34% (e.g. Figure 3As shown, the horizontal axis is the number of training rounds, and the vertical axis is the transportation cost).

[0128] In addition, for the case of 10 tasks, compared with the random method, the method of the present invention can search for a better transportation path and reduce the total transportation distance of the vehicle from 516.64 km to 356.92 km, which greatly reduces the transportation distance. Figure 4 As shown, horizontal coordinate: X-axis coordinate, vertical coordinate: Y-axis coordinate).

[0129] The present invention provides an intelligent scheduling method for the power Internet of Things based on an exploration-enhanced deep deterministic policy gradient algorithm. First, a multi-objective optimization model for the power Internet of Things system is constructed, which aims to minimize transportation costs and simultaneously maximize user satisfaction to ensure the improvement of the comprehensive performance of the power Internet of Things system. Secondly, in view of the challenge of insufficient exploration capability of the traditional deep deterministic policy gradient algorithm, an exploration-enhanced policy network is innovatively proposed to significantly increase the diversity of actions. Finally, based on the deep fusion of the exploration-enhanced policy network and the deep deterministic policy gradient algorithm, a novel exploration-enhanced deep deterministic policy gradient algorithm is developed. This algorithm can dynamically and in real time optimize the overall performance of the power Internet of Things system. By implementing the method proposed in the present invention, the transportation cost and user satisfaction of the power Internet of Things system will be greatly improved, laying a solid technical foundation for the intelligent scheduling of the power Internet of Things.

[0130] The present invention provides a method for intelligent scheduling of the power Internet of Things based on an exploratory enhanced deep deterministic policy gradient algorithm. There are many methods and approaches to implement this technical solution. The above is only a preferred embodiment of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and such improvements and modifications should also be considered within the scope of protection of the present invention. Any components not specified in this embodiment may be implemented using existing technologies.

Claims

1. A method for intelligent dispatching of power Internet of Things based on an exploration-enhanced deep deterministic policy gradient algorithm, characterized in that: The following steps are involved: Step 1: Establish a multi-objective optimization model for the power Internet of Things system; Step 2: Establish an exploration-enhanced deep deterministic policy gradient algorithm to optimize and solve the model established in step 1.

2. The method according to claim 1, characterized in that Step 1 includes the following steps: Step 1.1, establish a transportation cost model; Step 1.2, establish a customer satisfaction model; Step 1.3, establish a multi-objective optimization model for the power Internet of Things system.

3. The method according to claim 2, characterized in that Step 1.1 includes designing the following transportation cost model: TC=TC fuel (d,w)+TC maintent (d,w)+TC penalty (t) (1), Where TC represents the transportation cost, TC fuel is the fuel cost, TC fuel = k1·d·w, where d is the transport distance, w is the vehicle load, and k1 is the fuel cost per unit distance and unit weight; TC maintent is the vehicle maintenance cost, TC maintent =k2·d·w, where k2 is the maintenance cost per unit distance; TC penalty is the penalty cost, which depends on whether the transportation time t exceeds the stipulated time. If it exceeds, TC penalty =k3 max(0,tt max ), where k3 is the penalty cost per unit time, t max is the maximum shipping time specified.

4. The method according to claim 3, characterized in that Step 1.2 includes designing the following customer satisfaction model: CS=w1·CS punctuality (t)+w2·CS integrity (d,w) (2), Among them, CS represents customer satisfaction, w1 and w2 are weight coefficients; CS punctuality is the punctuality evaluation score, expressed as Among them, t expected is the customer's expected shipping time, t tolerance It is the shipping time deviation that customers can accept; CS integrity is the integrity evaluation score, expressed as e is a natural constant.

5. The method according to claim 4, characterized in that Step 1.3 includes: The multi-objective optimization model is expressed as: Where st represents the constraint, F represents the objective function vector, t max represents the maximum transportation time, w max Indicates the maximum vehicle load.

6. The method according to claim 5, characterized in that Step 2 includes the following steps: Step 2.1, construct the Markov decision process; Step 2.2, design state, action and reward functions; Step 2.3, establish an exploration-enhanced deep deterministic policy gradient algorithm.

7. The method according to claim 6, characterized in that Step 2.1 includes modeling the optimization problem in equation (3) as a Markov decision process.

8. The method according to claim 7, characterized in that Step 2.2 includes: Step 2.2.1, Design State: In the power Internet of Things system, the smart vehicle interacts with the environment as an intelligent agent and makes the best decision based on the acquired state information, which includes the location information, remaining power and load of the smart vehicle. t Defined as: s t =(x t ,y t ,pr,w) (4), Among them, x t and y t They represent the horizontal and vertical coordinates of the current position of the smart vehicle, and pr represents the remaining power of the vehicle; Step 2.2.2, design action a t : a t =(θ t ,v t ) (5), Among them, θ t and v t They represent the steering angle and speed of the smart vehicle at time slot t respectively; Step 2.2.3: Design the reward function r t : Among them, CS t is the customer satisfaction of the vehicle at time slot t, TC t is the transportation cost of the vehicle in time slot t.

9. The method according to claim 8, characterized in that Step 2.3 includes: Step 2.3.1, the agent is set to in represents a parameterized exploration-enhanced policy network, express The collection of Q μ (s t ,a t θ Q ) represents the parameterized value network; meanwhile, the target strategy network Directly copied from the exploration enhancement strategy network, the target value network Q' μ (s t ,a t ;θ' Q ) is directly copied to the value network; Step 2.3.2, explore the enhanced strategy network, the input state is s t , get the output action a t : s t =σ0e -0.2t / T (8), in, It means the mean is zero and the variance is Gaussian distribution function, T represents the total number of time slots for intelligent learning, and σ0 represents the initial variance value; When exploring the enhanced policy network update, randomly sample N from the experience replay pool batch Experience, the Monte Carlo method is used to calculate the gradient, assuming for The probability distribution of , the update process is as follows: in, represents the gradient of the exploration enhancement policy network, η π is the policy network learning rate, θ Q is the value network parameter; Step 2.3.3, the value network minimizes the loss function L(θ Q ) to update: Among them, y t is the approximate target Q value generated by the target network based on random sampling experience, γ is the discount factor, is the predicted Q value of the target value network, is the target policy network, θ' Q is the target value network parameter; The value network parameters are updated by gradient descent, and the loss function L(θ Q ) For the value network parameter θ Q The gradient of is calculated as follows: in, is the gradient of the value network; Step 2.3.4: Use soft update to update.

10. The method according to claim 9, characterized in that In step 2.3.4, let τ represent the learning rate of the target network update, and the update process of the target policy network and the target value network are expressed as: in, are the target policy network parameters.

Images