Continuous space path planning method based on deep reinforcement learning in unknown environment

Abstract

The invention relates to a continuous space path planning method based on deep reinforcement learning in an unknown environment, and belongs to the technical field of path planning, and the method comprises the steps: obtaining a target direction, a target distance, a moving speed and an obstacle distance of a mobile robot, generating an obstacle feature vector according to the obstacle distance, and defining a state vector; constructing a path planning model comprising a strategy network and a target strategy network; initializing a hidden feature vector, and inputting the hidden feature vector and the state vector of the mobile robot into the strategy network to obtain an updated hidden feature vector and an updated action vector; the mobile robot executes corresponding actions according to the action vectors, dynamic path planning of the robot in an unknown environment is achieved by defining the state vectors and combining the strategy network, time sequence information is transmitted through hidden feature vectors, the robot can generate the action vectors in combination with the state vectors, and therefore the continuity and adaptability of decision making are improved. The path can be adjusted more efficiently, and the obstacle avoidance success rate and the navigation efficiency are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of path planning, and particularly relates to a continuous space path planning method based on deep reinforcement learning in an unknown environment. Background Art

[0002] With the upsurge of artificial intelligence, autonomous navigation of robots has received extensive attention and research. Path planning is one of the key technologies for realizing autonomous navigation of robots, and its purpose is to plan a path that meets the constraints of the robot itself according to the task goal. In recent decades, with the rapid development of artificial intelligence, especially reinforcement learning, new possibilities have been provided for the path planning problem.

[0003] However, how to achieve efficient and reliable path planning in an unknown environment remains the core challenge in the field of autonomous mobile robots. Existing path planning methods based on reinforcement learning (such as DDPG policy network, TD3 policy network) have limitations in dealing with continuous-time dynamic problems due to the lack of continuous-time modeling ability when facing complex and dynamic unknown environments. In practical applications, the environment faced by the robot is continuously changing and uninterrupted, and discrete processing will cause the policy to be unable to update decisions in a timely manner in a rapidly changing scenario, thereby reducing the flexible response ability to environmental changes, resulting in the robot being difficult to adjust the path in a timely manner in a rapidly changing environment, and thus reducing the success rate of obstacle avoidance and path planning. Summary of the Invention

[0004] Aiming at the deficiencies in the related technologies, the purpose of the present invention is to provide a continuous space path planning method based on deep reinforcement learning in an unknown environment, so as to solve the technical problems that existing technologies have limitations in dealing with continuous-time dynamic problems due to the lack of continuous-time modeling ability when facing complex and dynamic unknown environments, discrete processing will cause the policy to be unable to update decisions in a timely manner in a rapidly changing scenario, thereby reducing the flexible response ability to environmental changes, resulting in the robot being difficult to adjust the path in a timely manner in a rapidly changing environment, and thus reducing the success rate of obstacle avoidance and path planning.

[0005] The present invention provides a continuous space path planning method based on deep reinforcement learning in an unknown environment, including the following steps: Step of defining a state vector: Obtain the target direction, target distance, moving speed and obstacle distance of the mobile robot, generate an obstacle feature vector according to the obstacle distance, and define a state vector based on the target direction, target distance, moving speed and obstacle feature vector; Step of building a model: Build a path planning model, and the path planning model includes a policy network; Step of generating an action vector: Initialize a hidden feature vector and combine it with the state vector of the mobile robot input the above into the policy network to obtain a hidden feature vector and an action vector ; Action execution step: The mobile robot executes corresponding actions according to the action vector

[0006] In the embodiment of the present invention, by defining the state vector and combining with the policy network, dynamic path planning of the robot in an unknown environment is realized. By transmitting temporal information through the hidden feature vector, the robot can generate an action vector in combination with the state vector and execute corresponding actions according to the action vector, thereby improving the continuity and adaptability of decision-making, being able to adjust the path more efficiently, and significantly improving the obstacle avoidance success rate and navigation efficiency.

[0007] In some embodiments of the present invention, the policy network includes: An input layer for inputting the state vector and the hidden feature vector; A state evolution layer including multiple CFC units, and the CFC unit is used to perform dynamic response modeling on the input state vector and the hidden feature vector through an analytical approximation method, and output an action vector and an updated hidden feature vector; An output layer for restricting the range of the action vector through an activation function.

[0008] The embodiment of the present invention realizes the temporal modeling ability of the motion state of the mobile robot, ensures that the network can capture long-term dependencies, enhances the adaptability of the path planning model to continuous spatio-temporal changes, enables the robot to quickly adjust the strategy when facing sudden obstacles or target changes, thereby improving the robustness and real-time performance of path planning.

[0009] In some embodiments of the present invention, the path planning model further includes a target policy network, an evaluation network, and a target evaluation network; The target policy network obtains the target action vector of the mobile robot at the next moment according to the state vector of the mobile robot at the next moment and the hidden feature vector ; ; The evaluation network generates an evaluation value and an evaluation value according to the state vector and the action vector ; The target evaluation network generates an evaluation value and an evaluation value according to the state vector and the action vector .​

[0010] In the embodiments of the present invention, an evaluation network and a target evaluation network are used to generate evaluation values, reducing the deviation of policy optimization, enabling the mobile robot to more accurately evaluate the action value in a complex environment, and thus improving the success rate and safety of path planning.

[0011] In some embodiments of the present invention, the model construction step further includes: Calculating an obstacle avoidance reward value according to the action vector and the state vector storing the obstacle avoidance reward value, the state vector , the hidden feature vector , the action vector , the state vector , the hidden feature vector in a six-tuple into an experience replay area, and training the path planning model according to the experience replay area.

[0012] In the embodiments of the present invention, by calculating the obstacle avoidance reward value and storing and using the obstacle avoidance reward value, the state vector, the action vector, etc. in a six-tuple for training, the overfitting problem caused by data correlation can be avoided, the stability of training and the sample utilization rate are improved, and the path planning model can learn the generalization ability from diverse environmental interactions, so as to show stronger adaptability and reliability in an unknown environment.

[0013] In some embodiments of the present invention, the method further includes: Evaluation network update step: calculating a target value according to the evaluation value , the evaluation value and the obstacle avoidance reward value, calculating a loss function based on the evaluation value , the evaluation value , the evaluation value and the target value , and updating the weight parameters of the evaluation network according to the loss function .

[0014] In the embodiments of the present invention, by combining the target value and the obstacle avoidance reward value to calculate the loss function, the accuracy of the target value estimation can be ensured. By dynamically adjusting the weight parameters, the path planning model can quickly adapt to environmental changes, significantly improving the efficiency of policy learning, and enabling the mobile robot to converge to the optimal path planning strategy faster in an unknown environment.

[0015] In some embodiments of the present invention, the method further includes: Policy network update step: updating the weight parameters of the policy network through the gradient descent algorithm Update the policy network according to the number of delayed steps, and generate the action vector according to the policy network The target value calculated and output by the evaluation network Maximize it.

[0016] In the embodiments of the present invention, the parameters of the policy network are updated by using the gradient descent algorithm, and the policy oscillation is reduced by the delayed update mechanism, and the target value output by the evaluation network is maximized , drive the policy network to generate better actions, thereby improving the performance of path planning, further stabilizing the training process, avoiding the convergence difficulty caused by frequent changes of the policy, and making the mobile robot perform more stably in long-term tasks.

[0017] In some embodiments of the present invention, the calculation model of the obstacle avoidance reward value is as follows: ; ; Wherein, is the reward value for approaching the target point; is the reward value for approaching the obstacle; is the obstacle avoidance reward value in the th direction; is the sum of the obstacle avoidance reward values in the th direction; is the target distance and target direction at the current moment; is the target distance and target direction at the previous moment; is the target distance; is the threshold of the target distance; is the obstacle distance; is the threshold of the obstacle distance; is the amplification factor; is the weight coefficient.

[0018] In the embodiments of the present invention, by combining the reward value for approaching the target point, the reward value for approaching the obstacle and the obstacle avoidance reward value of the direction, the behavior of the robot is comprehensively guided. By dynamically adjusting the reward weight, the path planning model can achieve a balance between exploration and obstacle avoidance, significantly improving the decision-making quality of the mobile robot in a complex environment, so as to balance safety and efficiency.

[0019] In some embodiments of the present invention, the calculation model of the target value is as follows: ; Wherein, is the discount factor; is the policy noise.

[0020] In the embodiments of the present invention, by introducing a discount factor and policy noise into the calculation model of the target value , it is possible to balance immediate rewards and long-term benefits. By taking the minimum value of the evaluation value and the evaluation value , the problem of overestimation of the target value is further suppressed, enabling the mobile robot to pay more attention to long-term cumulative rewards when planning paths, avoiding short-sighted behaviors, and thus achieving global optimal path planning in a dynamic environment.

[0021] In some embodiments of the present invention, the calculation model of the loss function is: ; ; where is the minimum number of samples sampled from the experience replay area; is the th sample sampled from the experience replay area.

[0022] In the embodiments of the present invention, the evaluation network is optimized through the loss function, and the average error is calculated using batch samples in the experience replay area to improve the stability of training, ensure that the path planning model learns from diverse interaction data, avoid overfitting to a single scenario, and significantly improve the accuracy of the target value estimation, enabling the mobile robot to more reliably evaluate the action value in an unknown environment.

[0023] In some embodiments of the present invention, the specific action execution steps are as follows: Based on the action vector , the linear velocity and angular velocity of the mobile robot are obtained, and the driving wheel speed of the mobile robot is calculated based on a two-wheel differential model. The mobile robot executes corresponding actions according to the linear velocity, angular velocity, and driving wheel speed.

[0024] In the embodiments of the present invention, by converting the action vector into a linear velocity and angular velocity and controlling the driving wheels based on the differential model, it is possible to ensure the smooth and continuous movement of the mobile robot, avoid mechanical losses or trajectory jitters caused by sudden changes, and enable the mobile robot to stably execute the planned path in complex terrains by precisely controlling the driving wheel speed, improving the practicality and reliability of the system. Description of the Drawings

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will describe the specific embodiments of the present invention in detail with reference to the drawings. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings without creative efforts. Figure 1A CFC network layer structure diagram provided by an embodiment of the present invention; Figure 2 A flowchart of a continuous space path planning method based on deep reinforcement learning in an unknown environment provided by an embodiment of the present invention; Figure 3 A schematic diagram of a mobile robot application provided by an embodiment of the present invention; Figure 4 A schematic diagram of a two-wheel differential model of a mobile robot provided by an embodiment of the present invention; Figure 5 A laser radar scanning diagram of a mobile robot provided by an embodiment of the present invention; Figure 6 A path planning model structure diagram provided by an embodiment of the present invention; Figure 7 A flowchart of a training environment for path planning provided by an embodiment of the present invention; Figure 8 A flowchart of an unknown test environment for path planning provided by an embodiment of the present invention; Figure 9 A test result diagram of a DDPG policy network provided by an embodiment of the present invention; Figure 10 A test result diagram of a TD3 policy network provided by an embodiment of the present invention; Figure 11 A test result diagram of a path planning model provided by an embodiment of the present invention. Detailed implementation manners

[0026] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be described and explained below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments provided by the present application without creative efforts belong to the scope of protection of the present application. It should be noted that the terms used herein are only for describing specific implementation manners and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless otherwise clearly specified in the context, the singular form is also intended to include the plural form. In addition, it should be understood that the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices. Achieving efficient and reliable path planning in unknown environments is a core challenge in the field of autonomous mobile robots. Existing path searches based on global maps or traditional reinforcement learning networks have achieved certain results in specific scenarios. However, when facing environmental dynamic changes and uncertainties, their adaptability and obstacle avoidance success rate are still insufficient. Especially in complex environments, it is often difficult to effectively handle rapidly changing states, resulting in a lag in the decision-making process and affecting the accuracy and efficiency of path planning.

[0027] Meanwhile, deep reinforcement learning has attracted much attention for its ability to handle high-dimensional state spaces and continuous action control problems, and it updates the policy in real time by using sensor data. However, existing deep reinforcement learning methods still face problems such as large model scale, low decision-making efficiency, and poor environmental adaptability. How to reduce the model scale while maintaining a high success rate and significantly improve the decision-making efficiency has become an important research direction in this field.

[0028] In addition, existing reinforcement learning-based path planning methods (such as DDPG policy network, TD3 policy network) have problems of poor adaptability, low computational efficiency, and insufficient obstacle avoidance accuracy when facing complex and dynamic unknown environments. Especially, traditional multi-layer perceptrons have limitations in dealing with continuous-time dynamic problems, resulting in difficulties for robots to adjust paths in a timely manner in rapidly changing environments, thereby reducing the success rate of obstacle avoidance and path planning.

[0029] Among them, the DDPG policy network is a deep reinforcement learning algorithm based on the Actor-Critic framework, suitable for continuous action spaces. It can directly output continuous actions through deterministic policy gradients, and combines an experience replay buffer and a target network to improve stability.

[0030] The TD3 policy network is an improvement of the DDPG policy network. It can use a dual-Critic network to take the minimum value of two Q-values as the target to reduce overestimation. Through delayed updates, the policy network update frequency is lower than that of the Critic network, and by adding noise to the target action, the policy optimization is smoothed.

[0031] However, the DDPG policy network or TD3 policy network uses a multi-layer perceptron as the policy network, and its input and output are based on discrete time steps. Since the multi-layer perceptron is a discrete-time model and cannot describe continuous state evolution through differential equations, in a continuously changing environment, discrete processing will lead to decision-making lags and inability to respond to dynamic obstacles or target movements in real time. In practical applications, the environment faced by robots is continuously changing and uninterrupted. Discrete processing will cause the policy to be unable to update decisions in a timely manner in rapidly changing scenarios, thereby reducing the flexible response ability to environmental changes.

[0032] At the same time, the policy network with a multi-layer perceptron structure achieves nonlinear mapping by stacking fully connected layers, which requires a large number of parameters to fit complex functions. However, redundant parameters may reduce the generalization ability, making it difficult to process perception data quickly and efficiently in real-time decision-making scenarios, resulting in decision delays and reducing overall operating efficiency.

[0033] In addition, in the existing deep reinforcement learning framework, the experience replay area usually uses a four-tuple (current state, action, reward, next state) to store experience data. This four-tuple structure lacks effective capture of time series and dynamic changes, resulting in the policy network being unable to fully utilize continuous time information during training. In complex environments, the algorithm lacks sufficient historical information during the learning process, which in turn limits the policy's ability to adapt to environmental changes, especially in tasks with strong temporal dependencies.

[0034] In addition, the existing reward function design often only considers approaching the target and simple obstacle avoidance behaviors. The reward design does not subdivide the dynamic threat level of multi-directional obstacles, and lacks fine perception and dynamic response to the distribution of multi-directional obstacles, resulting in inaccurate obstacle avoidance in complex obstacle distributions, which in turn causes the mobile robot to be unable to maintain stable and efficient obstacle avoidance performance in complex environments.

[0035] The present invention adds a CFC network layer in the TD3 strategy network and the target strategy network to replace the multi-layer perceptron architecture, which can accurately model the continuous-time dynamic system.

[0036] like Figure 1 As shown in the figure, the CFC network layer is a closed continuous-time network. The CFC network layer includes an input layer, multiple CFC units and an output layer. The input data is input to the main network of the CFC unit through the input layer for feature extraction, and then sent to three head networks in parallel: the g branch generates the new information that should be injected at present, the f branch gives the "liquid time constant" that changes with the time encoding vector, and the h branch generates the historical content that needs to be retained. The output of the f branch is multiplied by the time vector and then formed into a continuous-time gate σ by an activation function. Subsequently, the network performs a weighted sum of g and h element by element in the Hadamard structure with σ to retain the old information, inject new information, and forget outdated information, and obtain the updated hidden representation y(t). Since the timestamp is only explicitly involved at the gate, this mechanism enables the CFC network layer to continuously and smoothly track environmental dynamics in irregular sampling or decision hysteresis scenarios.

[0037] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other. The technical solution of the present invention is described in detail below in conjunction with specific embodiments and the accompanying drawings.

[0038] As attachedFigure 2 As shown in Figure 2 , the present invention provides a continuous space path planning method based on deep reinforcement learning in an unknown environment, including the following steps: State vector definition step S1: Obtain the target direction, target distance, moving speed, and obstacle distance of the mobile robot, generate an obstacle feature vector according to the obstacle distance, and define a state vector based on the target direction, target distance, moving speed, and obstacle feature vector.

[0039] Model construction step S2: Construct a path planning model, and the path planning model includes a target policy network.

[0040] In an optional embodiment, the path planning model is a CCTD3 model.

[0041] Action vector generation step S3: Initialize a hidden feature vector and input it together with the state vector of the mobile robot into the policy network to obtain a hidden feature vector and an action vector ; In some embodiments of the present invention, the hidden feature vector is the hidden state of the CFC unit, which is used to store the historical state, actions, and information of environmental feedback of the mobile robot evolving over time in the environment, so that the policy network and the target policy network can capture the continuous change of the state over time.

[0042] Action execution step S4: The mobile robot executes corresponding actions according to the action vector ;

[0043] Based on the above method, by defining the state vector and combining the policy network, the dynamic path planning of the robot in an unknown environment is realized. By transmitting temporal information through the hidden feature vector, the robot can generate an action vector in combination with the state vector and execute corresponding actions according to the action vector, thereby improving the continuity and adaptability of decision-making, being able to adjust the path more efficiently, and significantly improving the obstacle avoidance success rate and navigation efficiency.

[0044] In some embodiments of the present invention, the linear velocity and angular velocity of the mobile robot are obtained according to the action vector and the driving wheel speed of the mobile robot is calculated based on a two-wheel differential model. The mobile robot executes corresponding actions according to the linear velocity, angular velocity, and driving wheel speed.

[0045] In some embodiments of the present invention, as Figure 3 shown, the mobile robot can adopt a Turtlebot 2 mobile robot based on a two-wheel differential model.

[0046] Combined with Figure 4As shown, at the current moment, according to the motion vector the linear velocity and the angular velocity are obtained. Combining with the wheelbase of the mobile robot, the speeds of the left and right drive wheels of the mobile robot are calculated respectively through the two-wheel differential model: the speed of the left drive wheel : ; ; If the mobile robot is turning, its turning radius is related to the angular velocity and the linear velocity as follows: ; If the speeds of the left drive wheel and the right drive wheel of the mobile robot are different and the mobile robot moves along a curved trajectory, its turning radius is related to the angular velocity , the linear velocity and the wheelbase as follows: ; According to the above relationships and the speed of the left drive wheel , the speed of the right drive wheel , the wheelbase and the turning radius and the angular velocity and the linear velocity the calculation formula of the two-wheel differential model is obtained as: .

[0047] In some embodiments of the present invention, in the state vector definition step S1, the moving speed of the mobile robot is the linear velocity of the mobile robot at the previous moment and the angular velocity of the mobile robot at the previous moment; Combined with Figure 5 shown, the mobile robot can obtain the obstacle distances within 180° in the front direction through a single-line lidar, divide the obstacle distances within 180° into 18 directions, and use the data of these 18 directions as lidar features to obtain an obstacle feature vector: ; where is the direction, is the distance of the lidar in the th direction.

[0048] Define a 22-dimensional state vector according to the target direction, target distance, linear velocity of the mobile robot at the previous moment, angular velocity of the mobile robot at the previous moment, and the obstacle feature vector: ; Among them, is the target direction; is the target distance; is the linear velocity of the mobile robot at the previous moment; is the angular velocity of the mobile robot at the previous moment.

[0049] In some embodiments of the present invention, the mobile robot controls its operation according to the action vector to obtain the linear velocity, angular velocity, and drive wheel speed, and continues to obtain the state vector based on the linear velocity, angular velocity, and drive wheel speed, and then continuously generates the action vector and dynamically adjusts the path until it finally reaches the target point.

[0050] By converting the action vector into the linear velocity and angular velocity and controlling the drive wheels based on the differential model, it can ensure the smooth and continuous movement of the mobile robot, avoid mechanical losses or trajectory jitters caused by sudden changes, and enable the mobile robot to stably execute the planned path in complex terrains by precisely controlling the drive wheel speed, improving the practicability and reliability of the system.

[0051] In some embodiments of the present invention, as shown in combination with Figure 6 , the policy network includes: An input layer for inputting the state vector and the hidden feature vector; A state evolution layer including multiple CFC units, and the CFC units are used to perform dynamic response modeling on the input state vector and hidden feature vector through an analytical approximation method, and output the action vector and the updated hidden feature vector; An output layer for restricting the range of the action vector through an activation function.

[0052] In some embodiments of the present invention, the policy network specifically includes: An input layer for inputting the state vector and the hidden feature vector ; A CFC network layer including multiple CFC units, and each CFC unit simulates the state evolution of a continuous-time dynamic system, performs dynamic response modeling on the input state vector and the hidden feature vector , and outputs the action vector and the updated hidden feature vector , and can efficiently process the changes in sensing data in the continuous-time domain.

[0053] The output layer is used to limit the action vector through an activation function ; In an optional embodiment, the action vector can be limited within the range of [0, 1] through the Sigmoid activation function.

[0054] In some embodiments of the present invention, the path planning model further includes a target policy network, and the target policy network obtains the target action vector of the mobile robot at the next moment according to the state vector of the mobile robot at the next moment and the hidden feature vector ; Among them, the target policy network includes: An input layer for inputting the state vector at the next moment and the updated hidden feature vector ; The CFC network layer includes multiple CFC units, and the CFC units perform dynamic response modeling on the input state vector and the hidden feature vector and output an intermediate action representation.

[0055] An output layer for mapping the intermediate action representation output by the CFC unit to the target action vector through a fully connected layer .

[0056] Through the ability to model the time series of the motion state of the mobile robot, it is ensured that the network can capture long-term dependencies, enhance the adaptability of the path planning model to continuous spatio-temporal changes, enable the robot to quickly adjust its strategy when facing sudden obstacles or target changes, and thus improve the robustness and real-time performance of path planning.

[0057] In some embodiments of the present invention, the path planning model further includes an evaluation network and a target evaluation network; The evaluation network generates an evaluation value and an evaluation value according to the state vector and the action vector ; The target evaluation network generates an evaluation value and an evaluation value according to the state vector and the action vector .

[0058] Specifically, the evaluation network includes: An input layer for inputting the state vector and the action vector ; The hidden layer includes three fully connected layers, and each fully connected layer includes 512 neurons. Through the fully connected layer, the state vector​ and the action vector Perform feature extraction and vector mapping, and use the ReLU activation function to make the evaluation network fit the state vector and the action vector of the non-linear relationship; Output layer, used to output the evaluation value through a fully connected layer and the evaluation value .

[0059] The target evaluation network includes: Input layer, used to input the state vector of the next moment and the target action vector ; Hidden layer, including three fully connected layers, through the fully connected layer to perform feature extraction and vector mapping on the state vector and the action vector Perform feature extraction and vector mapping, and use the ReLU activation function to make the evaluation network fit the state vector and the action vector of the non-linear relationship; Output layer: used to output the evaluation value through a fully connected layer and the evaluation value .

[0060] Generating the evaluation value through the evaluation network and the target evaluation network reduces the deviation of policy optimization, enabling the mobile robot to more accurately evaluate the action value in a complex environment, thereby improving the success rate and safety of path planning.

[0061] In some embodiments of the present invention, the model construction step S2 further includes: (1) Initialization phase: Initialize the policy network, evaluation network, target policy network, and target evaluation network, and initialize the parameters of the target policy network and target evaluation network to be the same as those of the policy network and evaluation network through hard update; Initialize the experience replay area , and the experience replay area is used to store the experience data of the interaction between the mobile robot and the environment; Adopt - the greedy exploration algorithm to initialize the exploration parameters.

[0062] (2) Training phase: Reset the environment, obtain the initial state vector , and initialize the hidden feature vector ; The state vector of the current moment and the hidden feature vector Input into the policy network to generate an action vector and the updated hidden feature vector , and transmit the generated action vector to the control execution module of the mobile robot to drive the mobile robot to execute corresponding actions; After the mobile robot executes corresponding actions according to the action vector , update the state vector , and calculate the obstacle avoidance reward value according to the action vector and the state vector ; ; The calculation model of the obstacle avoidance reward value is: ; ; ; Among them, is the reward value for approaching the target point; is the reward value for approaching the obstacle; is the target distance; is the threshold of the target distance; is the obstacle distance; is the threshold of the obstacle distance; is the amplification coefficient of the distance and direction difference between the mobile robot and the target point at the previous moment and the current moment; is the weight coefficient of the multi-directional obstacle avoidance reward; In an alternative embodiment, the reward value encourages the mobile robot to approach the target point during path planning, and the reward value encourages the mobile robot to avoid obstacles during path planning; is the target distance and target direction at the current moment; is the target distance and target direction at the previous moment; is the target distance at the current moment; is the included angle towards the target point at the current moment; ; ; ; ; Among them, is the obstacle avoidance reward value in the th direction; is the sum of the obstacle avoidance reward values in the th direction; is the The weight coefficient of the obstacle avoidance reward in each direction; is the lidar distance in the th direction after normalization, when it tends to 0, it means the mobile robot is moving away from the obstacle, and when it tends to 1, it means the mobile robot is approaching the obstacle; is the lidar distance in the th direction without normalization; is the predicted safe distance; is the maximum moving speed of the mobile robot; is the motion time interval of the mobile robot; is the predicted safe distance after normalization. When is less than the predicted safe distance after normalization , the mobile robot obtains the corresponding obstacle avoidance reward , and when is greater than the predicted safe distance after normalization , no reward is given; by combining the reward value for approaching the target point, the reward value for approaching the obstacle, and the obstacle avoidance reward value for direction, the robot's behavior is comprehensively guided. By dynamically adjusting the reward weights, the path planning model can achieve a balance between exploration and obstacle avoidance, significantly improving the decision-making quality of the mobile robot in complex environments, thus taking into account both safety and efficiency.

[0063] Store the obstacle avoidance reward value , the state vector , the hidden feature vector , the action vector , the state vector , the hidden feature vector in a six-tuple and store it in the experience replay buffer ; Judge whether the experience data in the experience replay buffer meets or exceeds the number of mini-batch samples used in each training. If it meets, train the path planning model and randomly extract a mini-batch of six-tuple samples from the buffer; by calculating the obstacle avoidance reward value and storing and using the obstacle avoidance reward value, state vector, action vector, etc. in a six-tuple for training, it can avoid the overfitting problem caused by data correlation, improve the stability of training and the sample utilization rate, enable the path planning model to learn generalization ability from diverse environmental interactions, and thus show stronger adaptability and reliability in unknown environments.

[0064] Input the state vector into the target policy network to generate the target action vector , and according to - the greedy exploration algorithm, add policy noise to the target action vector ​ , generate a smooth target action, limit it within the action space range, and calculate the target value .

[0065] Among them, the calculation model of the target value is as follows: ; Among them, is the discount factor.

[0066] By introducing the discount factor and policy noise into the calculation model of the target value , the immediate reward and long-term benefit can be balanced. By taking the minimum value of the evaluation value and the evaluation value , the problem of overestimating the target value is further suppressed, enabling the mobile robot to pay more attention to the long-term cumulative reward when planning the path, avoiding short-sighted behavior, and thus achieving global optimal path planning in a dynamic environment.

[0067] In some embodiments of the present invention, the above method further includes: Evaluation network update step S5: Calculate the target value based on the evaluation value , the evaluation value and the obstacle avoidance reward value . Calculate the loss function based on the evaluation value , the evaluation value and the target value , and update the weight parameters of the evaluation network according to the loss function .

[0068] By combining the target value and the obstacle avoidance reward value to calculate the loss function, the accuracy of estimating the target value can be ensured. By dynamically adjusting the weight parameters, the path planning model can quickly adapt to environmental changes, significantly improving the efficiency of policy learning, and enabling the mobile robot to converge to the optimal path planning strategy faster in an unknown environment.

[0069] Among them, the calculation model of the loss function is: ; ; Among them, is the minimum number of samples sampled from the experience replay area; is the th sample sampled from the experience replay area.

[0070] Specifically, the state vector and the action vector Input into the evaluation network to obtain evaluation values and evaluation value respectively. Then, through the calculation model of the loss function, obtain loss function and loss function . Through minimizing the total loss, update the weight parameters of the evaluation network .

[0071] Optimize the evaluation network through the loss function, calculate the average error using the batch samples in the experience replay area, improve the stability of training, ensure that the path planning model learns from diverse interaction data, avoid overfitting to a single scenario, and significantly improve the accuracy of the target value estimation, enabling the mobile robot to more reliably evaluate the action value in an unknown environment.

[0072] In some embodiments of the present invention, the above method further includes: Policy network update step S6: Calculate the loss function through the gradient descent algorithm to update the weight parameters of the policy network , update the policy network according to the delay steps, and maximize the target value calculated and output by the evaluation network according to the action vector generated by the policy network.

[0073] Among them, the calculation model of the loss function is: ; Among them, is the output of the policy network according to the state vector .

[0074] The gradient calculation model in the policy network update process is:

[0075] Among them, is the gradient of the evaluation network with respect to the action vector ; is the gradient of the action vector output by the policy network with respect to the weight parameters of the policy network.

[0076] By adopting the gradient descent algorithm to update the policy network parameters and reducing policy oscillation through the delayed update mechanism, maximize the target value output by the evaluation network, drive the policy network to generate better actions, thereby improving the performance of path planning, further stabilizing the training process, avoiding convergence difficulties caused by frequent policy changes, and enabling the mobile robot to perform more stably in long-term tasks.

[0077] In some embodiments of the present invention, the target network updates the weight parameters of the target policy network by means of soft update and the weight parameters of the target evaluation network ; The weight parameter and the weight parameter The update model is: ; ; wherein, is the soft update rate of the target network.

[0078] In order to verify the effectiveness of the above path planning method, a training environment for path planning is set as Figure 7 shown, and the DDPG policy network, TD3 policy network and path planning model are trained respectively. After the training is completed, as Figure 8 shown, a test experiment is carried out in an unknown test environment.

[0079] The test results of the DDPG policy network, TD3 policy network and path planning model in the unknown test environment are respectively as Figure 9 , Figure 10 and Figure 11 shown, wherein Figures 9 - 11 the abscissa in represents the horizontal distance of the unknown test environment, the ordinate represents the vertical distance of the unknown test environment, the blue dot is the starting point, the red dot is the target point, the green dot is the timeout, the red "×" is the collision between the mobile robot and the obstacle, the blue solid line is the path of the mobile robot, the red dotted line is the target straight path, and the line connecting the points is the trajectory of the mobile robot from the starting point to the target point.

[0080] It can be seen from Figure 9 that the test success rate of the DDPG policy network in the unknown test environment is 80.5%; it can be seen from Figure 10 that the test success rate of the TD3 policy network in the unknown test environment is 88.5%; it can be seen from Figure 11 that the test success rate of the path planning model in the unknown test environment is 99%.

[0081] Through the path planning method, a high obstacle avoidance success rate can be achieved in the path planning of an unknown environment, and the number of neural network parameters of the path planning model is 3,803, which is reduced by more than 30 times compared with 137,986 of the DDPG policy network model and TD3 policy network.

[0082] It should be noted that the above is a reference manner of the continuous space path planning method based on deep reinforcement learning, and the present invention is not limited thereto.

[0083] The embodiments of the present invention achieve dynamic path planning of a robot in an unknown environment by defining a state vector and combining a policy network with a target policy network. By transmitting temporal information through a hidden feature vector, the robot can generate an action vector in combination with the state vector. At the same time, the target policy network predicts the action vector at the next moment, thereby enhancing the continuity and adaptability of decision-making, enabling more efficient path adjustment, significantly improving the success rate of obstacle avoidance and navigation efficiency, and solving the technical problems of the prior art. In the face of complex and dynamic unknown environments, due to the lack of continuous time modeling ability, there are limitations in dealing with continuous time dynamic problems. Discrete processing will cause the policy to be unable to update decisions in a timely manner in rapidly changing scenarios, thereby reducing the flexible response ability to environmental changes, resulting in difficulties for the robot to adjust the path in a timely manner in a rapidly changing environment, and further reducing the success rate of obstacle avoidance and path planning.

[0084] Finally, it should be noted that the embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The same or similar parts among the embodiments can be referred to each other. The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that it is still possible to modify the specific implementation manners of the present invention or perform equivalent replacements on some technical features. Without departing from the spirit of the technical solutions of the present invention, they should all be covered within the scope of the technical solutions claimed by the present invention.

Claims

1. A continuous space path planning method based on deep reinforcement learning in an unknown environment, characterized in that: include: The state vector definition step includes obtaining a target direction, a target distance, a moving speed and an obstacle distance of the mobile robot, generating an obstacle feature vector according to the obstacle distance, and defining a state vector based on the target direction, the target distance, the moving speed and the obstacle feature vector; Model building step: building a path planning model, wherein the path planning model includes a strategy network; Action vector generation step: Initialize a hidden feature vector and compare it with the state vector of the mobile robot Input the policy network to obtain the hidden feature vector and motion vector ; Action execution step: the mobile robot performs the action according to the action vector Perform the corresponding action.

2. The continuous space path planning method based on deep reinforcement learning in an unknown environment according to claim 1, characterized in that: The policy network includes: Input layer, used to input state vector and hidden feature vector; A state evolution layer, comprising a plurality of CFC units, wherein the CFC units are used to perform dynamic response modeling on the input state vector and the hidden feature vector by an analytical approximation method, and output an action vector and an updated hidden feature vector; The output layer is used to limit the range of the action vector through an activation function.

3. The continuous space path planning method based on deep reinforcement learning in an unknown environment according to claim 1, characterized in that: The path planning model also includes a target strategy network, an evaluation network and a target evaluation network; The target strategy network moves the robot's state vector at the next moment. and the hidden feature vector Get the target action vector of the mobile robot at the next moment ; The evaluation network is based on the state vector and the action vector Generate evaluation value and evaluation value ; The target evaluation network is based on the state vector and the action vector Generate evaluation value and evaluation value .

4. The continuous space path planning method based on deep reinforcement learning in an unknown environment according to claim 3 is characterized in that: The model building step also includes: According to the motion vector and the state vector Calculate the obstacle avoidance reward value, and transform the obstacle avoidance reward value and the state vector , the hidden feature vector , the action vector , the state vector , the hidden feature vector The six-tuple is stored in the experience playback area, and the path planning model is trained according to the experience playback area.

5. The continuous space path planning method based on deep reinforcement learning in an unknown environment according to claim 4, characterized in that: The method further comprises: Evaluation network update step: based on the evaluation value , evaluation value and the obstacle avoidance reward value to calculate the target value , based on the evaluation value , evaluation value and the target value Calculate the loss function and update the weight parameters of the evaluation network according to the loss function .

6. The continuous space path planning method based on deep reinforcement learning in an unknown environment according to claim 5, characterized in that: The method further comprises: Policy network update step: Update the weight parameters of the policy network through the gradient descent algorithm , update the policy network according to the delay step number, and generate the action vector according to the policy network The target value output by the evaluation network is calculated maximize.

7. The continuous space path planning method based on deep reinforcement learning in an unknown environment according to claim 5, characterized in that: The calculation model of the obstacle avoidance reward value is: ； ； in, is the reward value for approaching the target point; is the reward value for approaching an obstacle; For the Obstacle avoidance reward value in each direction; for The sum of the obstacle avoidance reward values ​​in each direction; is the target distance and target direction at the current moment; is the target distance and target direction at the previous moment; is the target distance; is the threshold of target distance; is the obstacle distance; is the obstacle distance threshold; is the magnification factor; is the weight coefficient.

8. The continuous space path planning method based on deep reinforcement learning in an unknown environment according to claim 7, characterized in that: The target value The calculation model is: ； in, is the discount factor; is the strategic noise.

9. The continuous space path planning method based on deep reinforcement learning in an unknown environment according to claim 8, characterized in that: The calculation model of the loss function is: ； ； in, is the minimum number of samples sampled from the experience playback area; is the first sampled from the experience playback area samples.

10. The continuous space path planning method based on deep reinforcement learning in an unknown environment according to any one of claims 1 to 9, characterized in that: The specific steps of executing the action are: According to the motion vector The linear velocity and angular velocity of the mobile robot are obtained, the driving wheel speed of the mobile robot is calculated based on a one-two wheel differential model, and the mobile robot performs corresponding actions according to the linear velocity, angular velocity and driving wheel speed.

Images