A robot path planning method based on deep reinforcement learning

Abstract

The application discloses a robot path planning method based on deep reinforcement learning, and relates to the fields of intelligent agriculture, path planning, robots and the like. Firstly, a farm environment is perceived, a robot state space and an action space are defined, and a multi-modal weighted combination reward mechanism and an experience replay buffer are set. A learnable weight coefficient is introduced into a Critic network loss function in a traditional DDPG algorithm, and an entropy regularization term is added to a target function of an Actor network. Then, a differential game is generated through an adaptive decay greedy strategy to select a control strategy or a DDPG algorithm to generate an action. Finally, the action or the control strategy is executed, network parameters and target network parameters are updated, and the experience replay buffer is dynamically updated. Compared with other path planning methods, the application improves the adaptability of path planning to a dynamic environment, and also has good efficiency and safety in a complex agricultural environment.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of smart agriculture, path planning and robots, and particularly relates to a robot path planning method based on deep reinforcement learning. BACKGROUND

[0002] With the development of global agriculture towards intelligence and automation, robots are increasingly applied in agricultural operations. Robots can efficiently perform tasks such as sowing, weeding, fertilizing, spraying pesticides, and picking, greatly improving production efficiency and reducing labor costs. In complex agricultural environments, such as gardens, greenhouses and farms, robots need to have efficient path planning capabilities to adapt to dynamic, multi-obstacle, unstructured and uneven ground environments, and improve work efficiency.

[0003] The agricultural environment is more complex and variable than other environments, and the robot's operation process involves decision-making under uncertain, dynamic and complex conditions. Path planning faces many challenges. Nowadays, multi-robot systems are widely used in the agricultural field. In agricultural environments such as farms and orchards, not only are there static obstacles and uneven road surfaces, but there are also dynamic obstacle avoidance problems and local path planning problems between robots.

[0004] Traditional local obstacle avoidance algorithms usually assume that obstacles are static or that the motion of obstacles is very slow, which limits their ability to deal with dynamic obstacles. In multi-robot agricultural operation systems, dynamic obstacles are mainly other robots that are currently engaged in agricultural operations. Traditional methods often cannot effectively predict or avoid the motion of these obstacles. For A*, Dijkstra and other algorithms commonly used in path planning, frequent re-planning is required when facing dynamic changes in the environment, which requires a large amount of computation. If the robot needs to avoid obstacles in real time and spends a lot of computing resources each time the path is recalculated, this will affect the real-time performance of the robot in actual application, especially in environments such as agriculture that require efficient operation, which may cause response delays. Path planning methods based on artificial potential field are prone to local optimal solutions, which not only wastes time and computing resources, but also may cause the robot to fail to reach the target or repeatedly loop around and lose direction when performing tasks. Traditional dynamic window methods rely on local information and are prone to local optimization. Dynamic window algorithms only consider the feasible speed in a short period of time, but do not consider the global optimal path, which may cause the robot to fall into a local optimal solution, resulting in redundant paths or unnecessary stagnation. SUMMARY

[0005] In view of the above problems and actual needs, the application provides a robot path planning method based on deep reinforcement learning. An improved deep deterministic policy gradient (DDPG) algorithm is used as the core, and a differential game strategy is combined to improve the obstacle avoidance capability of robots in a dynamic environment in an agricultural environment. The method balances the differential game control strategy and the DDPG algorithm generated action through an adaptive decay epsilon-greedy strategy to improve the path planning efficiency, ensure that the robot effectively avoids obstacles in a dynamic environment, and safely and efficiently reaches the target point. At the same time, an optimization method based on a multi-modal weighted combination reward mechanism is used to ensure that the robot preferentially selects the optimal path.

[0006] The application provides a robot path planning method based on deep reinforcement learning, which is realized through the following steps:

[0007] Step 1: The robot R0 performs environment perception in the farm through a laser radar, defines the current state according to the perception information, and defines the action space.

[0008] Step 1.1: The current state s of the robot R0 is defined t Vector:

[0009]

[0010] In the state vector, d T is the distance between the robot and the target, is the included angle with the target, v t is the linear velocity, ω t is the angular velocity, d Oi is the distance from the robot to the obstacle O i , is the included angle with the obstacle O i , and i is the obstacle number;

[0011] Step 1.2: Based on the current state of the robot, the position coordinates (x t , y t ) are defined, and kinematics modeling is performed;

[0012] Step 1.3: The action space of the robot is defined, and the action space is a set of actions that the robot can select:

[0013] a t =[v t , ω t ].

[0014] Step 2: A multi-modal weighted combination reward mechanism is designed, and the dynamic weight adjustment is performed through the hyperbolic tangent function and the obstacle density function based on the reward mechanism, and an experience replay buffer is set;

[0015] Step 2.1: The reward mechanism contains three sub-reward items, i.e., path guidance reward r motion , obstacle avoidance constraint reward r safety and smoothness constraint reward r stable . The priority of the sub-reward is adjusted by dynamic weights a(t) and β, and the comprehensive reward function r total is:

[0016] r total = a(t) r motion + (1 - a(t)) r safety + β r stable

[0017] Step 2.2: The path guidance reward function r motion is:

[0018]

[0019] where d T is the distance between the robot R0 and the target point T, A1 is the reward maximum value of the path guidance reward function, A2 is the growth rate constant, and d max is the estimated maximum distance from the start point to the end point.

[0020] The obstacle avoidance constraint reward function r safety is:

[0021]

[0022] where d Oi is the distance between the robot R0 and the obstacle O i , B1 is the reward maximum value of the collision constraint reward function, and B2 is the growth rate constant.

[0023] The smoothness constraint reward function r stable is:

[0024] r stable = - (C1 |Δκ| + C2 |a z |),

[0025] where C1 and C2 are reward weight coefficients, Δκ is the curvature change rate, and a z is the axial acceleration.

[0026] The adjustment methods of the weights a(t) and β in the dynamic weight adaptive mechanism are as follows:

[0027]

[0028] β = β0 + χ · d obs

[0029] where a min ​​is the initial weight, a max is the final weight, β is the time scaling factor, β0is the base time factor, χ is the obstacle impact factor, d obs is the obstacle density function in the farm environment, tanh(β·t) is the hyperbolic tangent function;

[0030] Step 2.3: Set up the experience replay buffer D, the robot selects an action a t according to the current state s t , calculates the reward R t according to the pre-set reward mechanism, enters the next state s t+1 , and stores the information into the experience replay buffer D.

[0031] Step 3: Initialize the Actor-Critic network, introduce a learnable weight coefficient m in the loss function of the Critic network to dynamically adjust the contribution of different states and actions to the loss, add an entropy regularization term in the objective function of the Actor network, and initialize the target network.

[0032] Step 3.1: The objective of the Critic network is to learn the Q value function, i.e., given the current state s t and action a t , calculate the long-term cumulative reward, expressed as:

[0033]

[0034] where Q(s t ,a t ; θ Critic ) is the expected cumulative reward after performing action a t in state s t , is the average expected value, R t is the reward obtained immediately after performing action a t in state s t , γ is the discount factor that balances the weight of immediate reward and cumulative reward, is the cumulative reward after performing action a t+1 in the next state s t+1 , t+1 is the action generated by the Actor network in the next state, is the maximum Q value estimate among all actions in the next state, θ Critic is the Critic network parameter, is the Critic target network parameter;

[0035] The loss function L(θ Critic ) of the Critic network is as follows:

[0036]

[0037] where m is a learnable weight coefficient, which dynamically adjusts the contribution of the loss in the current state to the overall optimization according to the importance of the sample, y t is the target Q value, and the parameters of the Critic network are optimized by minimizing the loss function, thereby reducing the TD error, represents the expectation calculated for the state s sampled from the experience replay buffer D.

[0038] The goal of the Actor network is to optimize the policy by maximizing the Q value given by the Critic network, and entropy regularization is added to the objective function to improve exploration, and the objective function J Actor is:

[0039]

[0040] where is the entropy of action a t , and a is the regularization coefficient;

[0041] Step 3.2: To determine whether to use differential game to generate control strategy or use deep deterministic policy gradient algorithm to generate action in the current state, an adaptive decay ε-greedy strategy is introduced, and the expression is:

[0042]

[0043] ε current = max(ε min , ε initial · e -λ·g(s) ),

[0044] where g(s) is the environmental complexity function of state s, λ is the decay adjustment parameter, ε current is the exploration rate of the current state, ε min is the minimum exploration rate, ε initial is the initial exploration rate, and ε switch is the preset policy switching threshold, μ is a random number generated by uniform sampling, argmax Q(s, a) is the action with the maximum Q value, and DG(a) is the control strategy generated by differential game.

[0045] Step 4: If the current state is selected by the adaptive decay ε-greedy strategy to use differential game for dynamic obstacle avoidance, a differential game model is established to generate the control strategy.

[0046] Step 4.1: Establish a differential game model, define robot 1 as R0 and robot 2 as R1.

[0047] The definitions of R0and R1control strategies are u0= [v0, ω0] and u1= [v1, ω1] respectively;

[0048] Step 4.2: Set the cost function J0of R0as:

[0049]

[0050] where w1, w2, w3 are weight parameters, balancing the importance of different terms of the cost function, w1d -1 making the two robots as far away as possible, w2‖u0‖ 2 balancing energy consumption, making the robot R0close to the target point, d is the distance between the two robots;

[0051] Set the cost function J1of robot R1as:

[0052]

[0053] where w4, w5 are also weight parameters, w4d -1 making the two robots as far away as possible, w5‖u1‖ 2 controlling energy consumption;

[0054] Step 4.3: Feedback Nash equilibrium ensures that robots R0and R1optimize their strategies in the differential game, and any unilateral strategy change will not make the cost function of both smaller, construct Hamiltonians H0, H1:

[0055]

[0056] where f0, f1 are the kinematic equations of the robots, is the co-state variable of the robot;

[0057] Step 4.4: Derive the constructed Hamiltonian with respect to the control strategy, set the derivative to zero, and then solve the Hamiltonian equation to obtain the feedback control strategy;

[0058] Step 4.5: The control strategies of robots R0and R1are updated by gradient descent method:

[0059]

[0060] until the objective function meets the convergence condition, the feedback Nash equilibrium convergence condition is as follows:

[0061]

[0062] where is the final control strategy, and η is the update parameter.

[0063] Step 5: The robot performs an action, obtains a reward and enters the next state, stores the data to the experience replay buffer; calculates the target Q value, generates the next action through the Actor network, and updates the Actor-Critic network and target network parameters, and repeatedly cycles until the robot reaches the target point.

[0064] Step 5.1: Select and perform an action according to the current policy, obtain a reward according to the current state and reward mechanism after performing the action, and enter the next state;

[0065] Step 5.2: Perform action a t , and interact with the environment to obtain a reward R t , and observe the new state s t+1 ;

[0066] Step 5.3: Store the experience of the current interaction in the experience pool for subsequent training, and extract a small batch of data samples from the experience pool, and update the Critic network parameters using the sample, and calculate the probability of each experience sample being sampled:

[0067]

[0068] where, is a normalization term to ensure that the sample probability sums to 1, and σ is a hyperparameter, δ t is the TD error.

[0069] Step 5.4: Calculate the target Q value y t :

[0070] y t = r t + γQ'(s t+1 , a t+1 ; θ - Critic );

[0071] The goal is to make the Q value of the current state as close to the target Q value as possible, so the Critic network needs to minimize the following mean square error loss function:

[0072]

[0073] Step 5.5: Update the Actor-Critic parameters, and the Critic network parameters are updated through gradient descent:

[0074]

[0075] where α0 is the learning rate, controlling the step size of each parameter update, is the loss function with respect to the parameter θ CriticThe gradient;

[0076] The parameters of the Actor network are updated using the gradient of the Critic network. The update of the Actor network parameters uses the gradient ascent method, as shown in the following expression:

[0077]

[0078] in, The objective function with respect to parameter θ Actor The gradient;

[0079] The target network uses a soft update method, expressed as follows:

[0080]

[0081] Where, θ Critic With θ Actor For the network parameters of the Actor-Critic network, and τ represents the target network parameters, and τ is the network parameter update coefficient.

[0082] Step 5.6: Repeat the training until the termination condition is met. Attached Figure Description

[0083] Figure 1 This is an overall flowchart of an embodiment of the present invention. Detailed Implementation

[0084] To more clearly illustrate the purpose, technical solution, and advantages of this invention, the invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0085] Appendix Figure 1 This is an overall flowchart of an embodiment of the present invention. This embodiment provides a robot path planning method based on deep reinforcement learning, which specifically includes the following steps: farm environment perception and definition of robot state space and action space; setting up a multimodal weighted combination reward mechanism and experience replay buffer; initializing the network parameters and target network parameters of the improved DDPG algorithm; selecting differential game or DDPG algorithm through an adaptive decay ε-greedy strategy; generating a control strategy through differential game or generating an action through the DDPG algorithm according to the selection of the greedy strategy; executing the action or control strategy; updating the network parameters and target network parameters; and updating the experience replay buffer.

[0086] Step 1: Robot R0 uses LiDAR to perceive its environment on the farm, defines its current state based on the perceived information, and defines its action space.

[0087] Step 1.1: Define the current state s of robot R0. t vector:

[0088]

[0089] In the state vector, d T It is the distance between the robot and the target. It is the angle between the object and the target, v t It is linear velocity, ω t It is angular velocity, d Oi It is with obstacle O i distance, It is with obstacle O i The included angle, where i is the obstacle number;

[0090] Step 1.2: Based on the robot's current state, define the position coordinates as (x... t ,y t ), and perform kinematic modeling;

[0091] Step 1.3: Define the robot's motion space, which is represented by the set of actions the robot can choose.

[0092] a t =[v t ,ω t ].

[0093] Step 2: Design a multimodal weighted combination reward mechanism. Based on this reward mechanism, adjust the dynamic weights using the hyperbolic tangent function and obstacle density function, and set up an experience replay buffer.

[0094] Step 2.1: The reward mechanism includes three sub-reward items, namely path guidance reward r motion Obstacle avoidance constraint reward r safety And the stable constraint reward r stable By dynamically adjusting the priority of sub-rewards using weights α(t) and β, the overall reward function r is... total for:

[0095] r total =α(t)·r motion +(1-α(t))·r safety +β·r stable ,

[0096] Step 2.2: Path-guided reward function r motion for:

[0097]

[0098] Where d T A1 is the distance between robot R0 and target point T, A2 is the maximum reward of the path-guided reward function, and d is the growth rate constant.max is the estimated maximum distance from the start point to the end point;

[0099] Obstacle avoidance constraint reward function r safety is:

[0100]

[0101] where d Oi is the distance between robot R0 and obstacle O i , B1 is the reward maximum value of collision constraint reward function, and B2 is the growth rate constant;

[0102] Stationary constraint reward function r stable is:

[0103] r stable =-(C1|Δκ|+C2|a z |),

[0104] where C1 and C2 are reward weight coefficients, Δκ is the curvature change rate, and a z is the axial acceleration;

[0105] The adjustment methods of weights α(t) and β in the dynamic weight adaptive mechanism are as follows:

[0106]

[0107] β=β0+χ·d obs ,

[0108] where α min is the initial weight, α max is the maximum weight, β is the time scaling coefficient, β0 is the basic time coefficient, χ is the obstacle influence factor, d obs is the obstacle density function in the farm environment, and tanh(β·t) is the hyperbolic tangent function;

[0109] Step 2.3: Set the experience replay buffer D, and the robot selects the action a t according to the current state s t , calculates the reward R t according to the preset reward mechanism, enters the next state s t+1 , and stores the information into the experience replay buffer D.

[0110] Step 3: Initialize the Actor-Critic network, introduce the learnable weight coefficient m in the loss function of the Critic network to dynamically adjust the contribution of different states and actions to the loss, add the entropy regularization term in the objective function of the Actor network, and initialize the target network.

[0111] Step 3.1: The goal of the Critic network is to learn the Q-value function, that is, given the current state s... t and action a t When calculating the long-term cumulative reward, the expression is:

[0112]

[0113] Where Q(s) t ,a t ;θ Critic ) is in state s t Next, execute action a t The expected cumulative reward after that, R is the average expected value. t In state s t Next, execute action a t The reward received immediately afterwards, γ is a discount factor that balances the weight of immediate and cumulative rewards. In the next state s t+1 Next, execute action a t+1 The cumulative reward after that, a t+1 The action generated by the Actor network in the next state. It is the maximum Q-value estimate among all actions in the next state, θ Critic These are the parameters of the Critic network. These are the Critic target network parameters;

[0114] The loss function L(θ) of the Critic network Critic )as follows:

[0115]

[0116] Where m is a learnable weight coefficient, dynamically adjusted according to the importance of the sample to determine the contribution of the current state's loss to the overall optimization, and y t The objective is the Q-value. The parameters of the Critic network are optimized by minimizing the loss function, thereby reducing the TD error. This represents the expectation calculated for the state s sampled from the experience playback buffer D.

[0117] The goal of the Actor network is to optimize the policy by maximizing the Q-value given by the Critic network. Entropy regularization is added to the objective function to improve exploratory behavior. The objective function J... Actor for:

[0118]

[0119] in It is action a t The entropy, where α is the regularization coefficient;

[0120] Step 3.2: To determine whether to generate a control strategy using differential game or to generate an action using deep deterministic policy gradient algorithm, an adaptive decay ε-greedy strategy is introduced, expressed as:

[0121]

[0122] ε current = max(ε min , ε initial · e -λ·g(s) ),

[0123] where g(s) is the environmental complexity function of state s, λ is the decay adjustment parameter, ε current is the exploration rate of the current state, ε min is the minimum exploration rate, ε initial is the initial exploration rate, ε switch is the preset policy switching threshold, μ is a random number generated by uniform sampling, argmax Q(s, a) is the action with the maximum Q value, and DG(a) is the control strategy generated by differential game.

[0124] Step 4: If the current state selects differential game for dynamic obstacle avoidance through adaptive decay ε-greedy strategy, a differential game model is established to generate a control strategy.

[0125] Step 4.1: Establish a differential game model, define robot 1 as R0 and robot 2 as R1.

[0126] Define the control strategies of R0 and R1 as u0 = [v0, ω0] and u1 = [v1, ω1], respectively.

[0127] Step 4.2: Set the cost function J0 of R0 as:

[0128]

[0129] where w1, w2, and w3 are weight parameters that balance the importance of different terms in the cost function, w1d -1 to keep the two robots as far apart as possible, w2‖u0‖ 2 to balance energy consumption, to keep robot R0 close to the target point, and d is the distance between the two robots.

[0130] Set the cost function J1 of robot R1 as:

[0131]

[0132] where w4 and w5 are also weight parameters, w4d -1 to keep the two robots as far apart as possible, and w5‖u1‖2 controlling energy consumption;

[0133] Step 4.3: Feedback Nash equilibrium ensures that robots R0 and R1 optimize their strategies in the differential game, and any unilateral strategy change will not make the cost function of both smaller. The Hamiltonian H0, H1 is constructed:

[0134]

[0135] where f0, f1 are the kinematic equations of the robots, is the co-state variable of the robot;

[0136] Step 4.4: Derive the constructed Hamiltonian with respect to the control strategy, set the derivative to zero, and then solve the Hamiltonian equation to obtain the feedback control strategy;

[0137] Step 4.5: The control strategies of robots R0 and R1 are updated by gradient descent method:

[0138]

[0139] until the objective function satisfies the convergence condition. The feedback Nash equilibrium convergence condition is as follows:

[0140]

[0141] where is the final control strategy, and η is the update parameter.

[0142] Step 5: The robot performs actions, obtains rewards and enters the next state, and stores data to the experience replay buffer. The target Q value is calculated, and the next action is generated by the Actor network, and the Actor-Critic network and target network parameters are updated. Repeat the cycle until the robot reaches the target point.

[0143] Step 5.1: Select and execute actions according to the current strategy. After executing the action, obtain the reward according to the current state and reward mechanism, and enter the next state;

[0144] Step 5.2: Execute action a t and interact with the environment to obtain reward R t and observe the new state s t+1 ;

[0145] Step 5.3: Store the experience of the current interaction in the experience pool for subsequent training. At the same time, extract a small batch of data samples from the experience pool, and update the Critic network parameters using the sample. Calculate the probability of each experience sample being sampled:

[0146]

[0147] where, is a normalization term that ensures the samples are sampled with probability and sum to 1, σ is a hyperparameter, δ t is the TD error.

[0148] Step 5.4: Compute the target Q value y t as follows:

[0149] y t = r t + γQ'(s t+1 , a t+1 ; θ - Critic );

[0150] The goal is to make the Q value of the current state as close to the target Q value as possible, so the Critic network needs to be trained to minimize the following mean squared error loss function:

[0151]

[0152] Step 5.5: Update the Actor-Critic parameters, the Critic network parameters are updated by gradient descent:

[0153]

[0154] where α0 is the learning rate, controlling the step size of each parameter update, is the gradient of the loss function with respect to the parameter θ Critic ;

[0155] The Actor network parameters are updated using the Critic network's gradient, and the Actor network parameters are updated using the gradient ascent method, expressed as follows:

[0156]

[0157] where, is the gradient of the target function with respect to the parameter θ Actor ;

[0158] The target network uses a soft update method, expressed as follows:

[0159]

[0160] where θ Critic and θ Actor are the network parameters of the Actor-Critic network, and are the target network parameters, and τ is the network parameter update coefficient;

[0161] Step 5.6: The training is repeated until the termination condition is reached.

Claims

1. A robot path planning method based on deep reinforcement learning, characterized in that, Includes the following steps: Step 1: Robot R0 uses LiDAR to perceive its environment on the farm, defines its current state based on the perceived information, and defines its action space; Step 2: Design a multimodal weighted combination reward mechanism. Based on this reward mechanism, adjust the dynamic weights using the hyperbolic tangent function and obstacle density function, and set up an experience replay buffer. Step 2.1: The reward mechanism includes three sub-reward items, namely path guidance reward r motion Obstacle avoidance constraint reward r safety And the stable constraint reward r stable By dynamically adjusting the priority of sub-rewards using weights α(t) and β, the overall reward function r is... total for: r total =α(t)·r motion +(1-α(t))·r safety +β·r stable , Step 2.2: Path-guided reward function r motion for: Where d T A1 is the distance between robot R0 and target point T, A2 is the maximum reward of the path-guided reward function, and d is the growth rate constant. max It is the estimated maximum distance from the starting point to the end point; Obstacle avoidance constraint reward function r safety for: Where d Oi It is robot R0 and obstacle O i The distance, B1 is the maximum reward of the collision constraint reward function, and B2 is the growth rate constant; stationary constrained reward function r stable for: r stable =-(C1|Δκ|+C2|a z |), Where C1 and C2 are reward weight coefficients, Δκ is the rate of change of curvature, and a z It is the axial acceleration; The adjustment methods for weights α(t) and β in the dynamic weight adaptive mechanism are as follows: β=β0+χ·d obs , Where α min It is the initial weight, α max The final weights are β, the time scaling factor is β0, the base time factor is χ, and the obstacle influence factor is d. obs It is the obstacle density function in the farm environment, and tanh(β·t) is the hyperbolic tangent function; Step 2.3: Set the experience replay buffer D, and the robot will replay the experience based on the current state s. t Select action a t Calculate the reward R according to the preset reward mechanism. t Enter the next state s t+1 And store the information in the experience playback buffer D; Step 3: Initialize the Actor-Critic network, introduce learnable weight coefficients m into the loss function of the Critic network to dynamically adjust the contribution of different states and actions to the loss, add an entropy regularization term to the objective function of the Actor network, and initialize the target network. Step 3.1: The goal of the Critic network is to learn the Q-value function, that is, given the current state s... t and action a t When calculating the long-term cumulative reward, the expression is: Where Q(s) t ,a t ;θ Critic ) is in state s t Next, execute action a t The expected cumulative reward after that, R is the mean expected value. t In state s t Next, execute action a t The reward received immediately afterwards, γ is a discount factor that balances the weight of immediate and cumulative rewards. In the next state s t+1 Next, execute action a t+1 The cumulative reward after that, a t+1 The action generated by the Actor network in the next state. It is the maximum Q-value estimate among all actions in the next state, θ Critic These are the parameters of the Critic network. These are the Critic target network parameters; The loss function L(θ) of the Critic network Critic )as follows: Where m is a learnable weight coefficient, dynamically adjusted according to the importance of the sample to determine the contribution of the current state's loss to the overall optimization, and y t The objective is the Q-value. The parameters of the Critic network are optimized by minimizing the loss function, thereby reducing the TD error. This represents the expectation calculated for the state s sampled from the experience playback buffer D; The goal of the Actor network is to optimize the policy by maximizing the Q-value given by the Critic network. Entropy regularization is added to the objective function to improve exploratory behavior. The objective function J... Actor for: in It is action a t The entropy, where α is the regularization coefficient; Step 3.2: To determine whether to use a differential game theory generation control strategy or a deep deterministic gradient algorithm to generate actions in the current state, an adaptive decay ε-greedy strategy is introduced, expressed as: e current =max(e min ,he initial ·e -λ·g(s) ), Where g(s) is the environmental complexity function of state s, λ is the decay adjustment parameter, and ε current It is the exploration rate of the current state, ε min It is the minimum exploration rate, ε initial It is the initial exploration rate, ε switch is the preset policy switching threshold, μ is a random number generated by uniform sampling, argmaxQ(s,a) is the action with the largest Q value, and DG(a) is the control policy generated by differential game. Step 4: If the current state selects differential game theory for dynamic obstacle avoidance through an adaptive decay ε-greedy strategy, a control strategy is generated by establishing a differential game model. Step 4.1: Establish a differential game model, define robot 1 as R0, and robot 2 as R1; Define the control strategies for R0 and R1 as u0 = [v0, ω0] and u1 = [v1, ω1], respectively. Step 4.2: Set the cost function J0 of R0 as follows: Where w1, w2, and w3 are weight parameters that balance the importance of different terms in the cost function, w1d -1 Keep the two robots as far apart as possible, w2‖u0‖ 2 Balance energy consumption, Move robot R0 closer to the target point, where d is the distance between the two robots; The cost function J1 for robot R1 is defined as follows: Where w4 and w5 are also weight parameters, w4d -1 Keep the two as far apart as possible, w5‖u1‖ 2 Control energy consumption; Step 4.3: Feedback Nash equilibrium ensures that robots R0 and R1 optimize their strategies in the differential game, and any unilateral policy change will not reduce their cost functions. Construct Hamiltonians H0 and H1: Where f0 and f1 represent the kinematic equations of the robot. These are the robot's common-state variables; Step 4.4: Differentiate the constructed Hamiltonian with respect to the control strategy, set the derivative function to zero, and then solve the Hamiltonian equation to obtain the feedback control strategy. Step 4.5: The control strategies for robot R0 and R1 are updated using gradient descent. Until the objective function satisfies the convergence condition, the convergence condition of the feedback Nash equilibrium is as follows: in For the final control strategy, η is the update parameter; Step 5: The robot performs an action, receives a reward, and enters the next state. The data is stored in the experience replay buffer. The target Q value is calculated, and the next action is generated through the Actor network. The parameters of the Actor-Critic network and the target network are updated. This process is repeated until the robot reaches the target point.

Images