Path planning method based on neuron elastic learning and multi-Q-value strategy
By introducing neuronal elastic learning and multi-Q value strategies in deep reinforcement learning path planning, the problems of slow repair of drift environments and overestimation of Q value in complex environments are solved, and more efficient and stable path planning performance is achieved.
Patent Information
- Application Number
- CN202411858103.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-17
- Publication Date
- 2025-04-01
AI Technical Summary
The existing deep reinforcement learning path planning algorithm is difficult to quickly repair the drifting environment when facing complex environments, resulting in data failure and decision-making errors. At the same time, there is a problem of Q value overestimation, which hinders the improvement of path planning performance.
The path planning method based on neuronal elastic learning and multi-Q value strategy is adopted, and strategy learning is performed through the Actor-Critic network. The neuronal elastic learning mechanism is designed to reduce the drift environment repair time, retain the main behavioral neurons and randomly initialize the secondary neuron weights, and a multi-Q value strategy is introduced to integrate action value and strategy action value by taking the average of the two smallest Q values.
It significantly enhances the adaptation and repair ability of mobile robots to drift environments caused by complex environment changes and uncertainties, reduces the problem of Q value overestimation, improves the stability and performance of path planning, and significantly reduces the number of collisions of mobile robots in navigation.
Smart Images
Figure CN119737968A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of path planning, and in particular to a path planning method based on neuron elastic learning and multi-Q value strategy. Background Art
[0002] With the rapid development of artificial intelligence technology, mobile robots have been widely used in many fields such as automated production, service robots and space exploration. Realizing autonomous navigation has become the focus of current research.
[0003] Mobile robots must plan efficient and reasonable collision-free paths under their own constraints to avoid obstacles and reach the target location smoothly. Traditional path planning algorithms, such as artificial potential field method and A * Pathfinding algorithms,rely on fixed prior knowledge of the environment and therefore face challenges in,dealing with unknown complex environments.
[0004] As an emerging artificial intelligence technology, deep reinforcement learning has achieved remarkable results in fields such as games and control with its powerful self-learning and generalization capabilities. Some researchers have begun to explore the combination of deep reinforcement learning and mobile robot path planning, and have preliminarily demonstrated its feasibility and potential in complex environments.
[0005] However, mainstream deep reinforcement learning methods often have the problem of slow repair of such drift environments when faced with complex environments, resulting in data invalidation and decision-making errors; in addition, deep neural network function approximation will lead to the problem of Q-value overestimation, and the slow repair of the drift environment aggravates the accumulation of errors, which seriously hinders the path planning performance; therefore, the development of deep reinforcement learning algorithms that can alleviate the drift environment problem while alleviating Q-value overestimation and improving learning stability, so that they can be better suitable for path planning of mobile robots under unknown complex environments, is the current research hotspot and development direction. Summary of the invention
[0006] In view of the shortcomings of the existing methods, the present invention proposes a mobile robot path planning method based on neuron elastic learning and multi-Q value strategy to solve the limitation that the existing deep reinforcement learning path planning algorithm cannot reduce the Q value overestimation problem while shortening the repair time for drift alleviation.
[0007] The technical solution adopted by the present invention is: a path planning method based on neuron elastic learning and multi-Q value strategy comprises the following steps:
[0008] Step 1: Build a path planning system for mobile robots in unknown environments and use the Actor-Critic network for strategy learning.
[0009] As a preferred embodiment of the present invention, step one specifically includes:
[0010] Step 11: Build a training environment, generate wall obstacles, column obstacles, dynamic obstacles, and target points, and visualize the environmental information perceived by the robot;
[0011] Step 12: Design a random dynamic environment generation module to control the generation probability and quantity range parameters of walls and obstacles of different shapes;
[0012] Step 13: Use a laser radar sensor to detect the environment;
[0013] Step 14: perform equidistant sampling on the original laser scanning data to obtain laser ranging information as state input;
[0014] Step 15: Reset the robot state and initialize the environment, and use the interaction between the robot and the environment to obtain training data;
[0015] Step 16: Store the training state transition data, execution actions, and immediate rewards of corresponding environment feedback in the experience replay pool for the Actor network and the Critic network.
[0016] Step 2: Initialize the parameters of the Actor Network and Critic Network. Distinguish the primary and secondary functions of neurons in each layer of the ActorNetwork, delete the secondary neurons and assign random values, and retain what the primary neurons have learned.
[0017] As a preferred embodiment of the present invention, step 2 specifically includes:
[0018] Let N = {n1, n2, n3, ...} be all neurons in a layer, and T = {t1, t2, t3, ...} be a set of test inputs;
[0019] By adding N i All neurons and percentage coefficients in Multiply, then perform linear offset and rounding operations, and finally get the area of the desired behavior at this level. The formula is:
[0020]
[0021] Among them, a is the adjustable coefficient, F r Represents the forgetting rate of previous information.
[0022] As a preferred embodiment of the present invention, the adjustable coefficient is 0.5.
[0023] As a preferred implementation of the present invention, step 2 further includes: updating the primary and secondary functional behaviors using a speed update factor, the formula is:
[0024]
[0025] Where L is the loss function; z [l] , W [l] and b [l] Represent the weighted sum output, weight and bias of the lth layer, z [l] =W [l] ·q [l-1] +b [l] ;q [l] represents the activation function g of the lth layer [l] Right [l] The derivative of ; η is the speed update factor.
[0026] As a preferred embodiment of the present invention, the rate update factor η is used to adjust the rate update of neurons.
[0027] As a preferred implementation of the present invention, step 3 is also included, using multiple Q values in CriticNetwork and selecting the average of the two smallest Q values as the strategy evaluation target, the formula is:
[0028]
[0029] Among them, Q1, Q2, ..., Q K represents K independent Q-value functions, and They represent the two smallest Q values among the K Q values.
[0030] As a preferred embodiment of the present invention, a path planning system based on neuronal elastic learning and multi-Q value strategy includes: a memory for storing instructions executable by a processor; a processor for executing instructions to implement a path planning method based on neuronal elastic learning and multi-Q value strategy.
[0031] As a preferred embodiment of the present invention, a computer readable medium stores a computer program code, and when the computer program code is executed by a processor, a path planning method based on neuron elastic learning and a multi-Q value strategy is implemented.
[0032] Beneficial effects of the present invention:
[0033] 1. The present invention designs a neuron elastic learning mechanism to reduce the repair time of the drift environment, retains the main behavior neurons and randomly initializes the weights of the secondary neurons, which greatly enhances the mobile robot's ability to adapt to and repair the drift environment caused by complex environmental changes and uncertainties;
[0034] 2. Introduce a multi-Q value strategy. By taking the average of the two smallest Q values, learn to integrate the action value and the strategy action value, reduce the Q value overestimation problem caused by the dynamic change of the neuron elastic learning weight, and effectively reduce the error expansion and accumulation caused by the dynamic change of the neuron elastic learning weight, making the path planning more reliable and stable; improve the stability of path planning, and build a dynamic obstacle production module in the gazebo simulation environment to change the environmental difficulty and enhance the generalization ability of the learning strategy;
[0035] 3. Setting up a dynamic obstacle generation module in the simulation environment can simulate the environmental complexity of various practical applications and significantly improve the complexity of the learning strategy;
[0036] 4. Compared with SAC, TD3, and DDPG, it achieves a more efficient strategy learning process and greatly reduces the number of collisions of mobile robots during navigation. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] Figure 1 It is a flow chart of the path planning method based on neuron elastic learning and multi-Q value strategy of the present invention;
[0038] Figure 2 It is the basic structure diagram of SAC algorithm;
[0039] Figure 3 is an example of a three-layer neural network;
[0040] Figure 4 It is a static environment map and static 3D point cloud modeling;
[0041] Figure 5 is the average reward graph of the four algorithms in the static environment of the present invention;
[0042] Figure 6 It is a comparison chart of the average rewards of the four algorithms in the static environment of the present invention;
[0043] Figure 7 is a comparison chart of the success rates of four algorithms in a static environment of the present invention;
[0044] Figure 8 It is the dynamic and static environment map and dynamic and static 3D point cloud modeling of the present invention;
[0045] Fig. 9 is the average reward graph of the four algorithms in the dynamic environment of the present invention;
[0046] Fig.10 It is a comparison chart of the average rewards of four algorithms in the dynamic environment of the present invention. DETAILED DESCRIPTION
[0047] The present invention is further described below in conjunction with the accompanying drawings and embodiments. This figure is a simplified schematic diagram, which only illustrates the basic structure of the present invention in a schematic manner, and therefore it only shows the components related to the present invention.
[0048] like Figure 1 As shown, the path planning method based on neuron elastic learning and multi-Q value strategy includes the following steps:
[0049] Step 1: Build a path planning system for mobile robots in unknown environments and use the Actor-Critic network for strategy learning.
[0050] Construct a dynamic obstacle generation module in the simulation environment to greatly improve the generalization ability of the final strategy, including:
[0051] Step 11: Build a training environment on the gazebo simulation platform. Generate wall obstacles, column obstacles, dynamic obstacles, and target points in the environment. Use the rviz module to visualize the environmental information perceived by the robot.
[0052] Step 12: Design a random dynamic environment generation module to construct parameterized training environments of different difficulty levels by controlling the generation probability and quantity range of walls and obstacles of different shapes;
[0053] Step 13: The mobile robot platform used in the experiment is a non-holonomic differential drive robot equipped with an LDS-01 laser radar sensor, which can detect the environment within a range of 0.12 meters to 3.5 meters in all directions at 360 degrees.
[0054] Step 14: perform equidistant sampling on the original laser scanning data to obtain 36-dimensional laser ranging information as state input;
[0055] Step 15: Before each experimental round, the robot state will be reset and the environment will be initialized immediately, and training data will be obtained through the interaction between the robot and the environment;
[0056] Step 16: Store the state transition data, execution actions, and immediate rewards of the corresponding environment feedback collected during training in the experience replay pool for training the Actor network and Critic network in the reinforcement learning model.
[0057] like Figure 2The existing SAC (Soft Actor-Critic) is an off-policy DRL algorithm based on maximum entropy. By combining off-policy with Actor-Critic and applying them to autonomous navigation of mobile robots, SAC can achieve better performance in continuous state and action space. In the SAC infrastructure, the Actor network accepts the input state and outputs the mean and variance of the probability distribution of the action space. The Critic network evaluates the value of the Agent's action or state in the current state. The action output by the Actor network affects the entropy of the probability distribution and the value evaluation of the Critic network, which will be used to judge the quality of the action taken, thereby affecting the decision of the Actor network.
[0058] Step 2: Initialize the parameters of the Actor Network and Critic Network. Distinguish the primary and secondary functions of neurons in each layer of the ActorNetwork, delete the secondary neurons and assign them values at random, and retain what the primary neurons have learned.
[0059] By designing a neuron elastic learning algorithm, neurons in each layer are divided into two categories: highly active neurons and low-active neurons. By deleting low-active neurons and reassigning them to distinguish primary and secondary functional behaviors, catastrophic forgetting caused by equal attention can be alleviated, and the model's adaptability to drift environments can be enhanced.
[0060] A deep neural network (DNN) is composed of neurons organized in multiple layers, including input layer, output layer and hidden layer; each neuron is a computing unit that calculates its output by applying an activation function; in a classic DNN, each neuron is fully connected to all neurons in the next layer, and each connection has a weight representing the strength of the connection; a DNN network can be viewed as a complex function that converts input into output, which is determined by the combined action of its computing units; neurons are basic computing units with specific perceptual preferences, and their synergy determines the functional performance of the entire deep neural network; Figure 3 (a) shows an example of a three-layer deep neural network.
[0061] Conceptually, the behavior of DNN can be divided into two categories: primary functional behavior and secondary functional behavior; the primary functional behavior encodes the ability to solve important problems, while the secondary functional behavior does not directly affect the solution of the problem, but is necessary to achieve the desired performance; in order to describe the primary and secondary functional behaviors, the present invention proposes a neuron classification method; the method divides neurons into two categories: highly active neurons and low active neurons; highly active neurons contribute almost all predictions, while low active neurons contribute indirectly to predictions; let N = {n1, n2, n3, ...} be a set of neurons of DNN; let T = {t1, t2, t3, ...} be a set of test inputs, and use ψ(t, n) to represent a function that returns the output of a neuron, which returns the output of n∈N under a given layer input t∈T; let DNN have l layers, l i Represents the neuron set on the i-th layer (1≤i≤l); for the i-th layer, use bottom k (t,i) represents the k neurons with the minimum output on this layer for a given t; Figure 3 In (a), assuming that ψ(t,n4) and ψ(t,n6) are smaller than ψ(t,n5) and ψ(t,n7), the first two neurons on the second layer are n5 and n7 (shown in blue).
[0062] The specific design of the neuron elastic learning function to distinguish between primary and secondary behaviors is as follows:
[0063] Let N = {n1, n2, n3, ...} be all neurons in a layer, T = {t1, t2, t3, ...} be a set of test inputs, and F r Indicates the forgetting rate of the previous information; in order to achieve a certain layer N i By dividing N i All neurons and percentage coefficients in Multiply, then perform linear offset and rounding operations, and finally get the area of the desired behavior at this level, which is specifically expressed as follows:
[0064]
[0065] Formula (1) can be used to screen highly active neurons. By adding a=0.5 to the calculation and then rounding down, the inconsistency behavior can be reduced, ensuring the accurate calculation of the number of highly active neurons in each layer and reducing potential calculation bias. Considering that the slow repair of this drift environment is due to the equal emphasis on primary and secondary functional behaviors, it leads to slow adaptation and catastrophic forgetting. Therefore, the update of primary and secondary functional behaviors is modified, that is, low-activity neurons will be "deleted" and highly active neurons will be retained. The specific performance is as follows:
[0066]
[0067] Where L is the loss function; z [l] , W [l] and b [l] Represent the weighted sum output, weight and bias of the lth layer, z [l] =W [l] ·q [l-1] +b [l] ;q [l] represents the activation function g of the lth layer [l] Right [l] The derivative of .
[0068] The present invention achieves different rate updates for two types of neurons by adjusting the speed update factor η; a larger η is given to highly active neurons to increase the amplitude of gradient updates and quickly adapt to environmental changes; a smaller η is used for low-activity neurons to reduce the learning speed by reducing the update amplitude; in this example, the larger η=0.1 and the smaller η=0.03.
[0069] Step 3: Use multiple Q values in the Critic Network and select the two smallest Q values to take the average as the strategy evaluation target; compare the average reward with the reward threshold at each evaluation, and call neuron elastic learning if the average reward obtained from training is lower than the reward threshold; combine the multi-Q value reinforcement learning method, select the average of the two smaller Q values among multiple Q values, reduce the impact of information loss caused by neuron elastic learning, and make the learned strategy more robust and reliable;
[0070] The specific design of the reinforcement learning method introduced by the multi-Q value strategy is as follows:
[0071] Experiments have found that while using neuron elastic learning to help improve the efficiency and generalization ability of the network, it also introduces instability; this instability is usually caused by information loss caused by changes in weight distribution, gradient flow, etc., and the gradual pruning of neurons may lead to a decrease in the network's representation ability; formally, let ζ i (s,a)=|Q i (s,a)-Q * (s,a)| represents the total error of the i-th iteration, let δ i (s,a)=|Q i (s,a)-TQ i-1 (s,a)| represents the current Bellman error, and the formula is:
[0072]
[0073] Formula 4 shows the error ζ of the current state-action pair (s, a) i (s,a) is affected by two parts: the current Bellman error δi The error ζ between (s,a) and the next state s0 and possible action a0 i-1 The expected value of (s0,a0) is accumulated through the discount factor γ; that is, the error of the current state action to (s,a) will be affected by the accumulated errors of other states and actions in previous iterations, not just the current Bellman error. Neuronal elastic learning alleviates the catastrophic forgetting caused by repairing the drift environment. However, it is necessary to consider the potential risks brought by the change in weight distribution and gradient flow in this method. These factors may iterate larger errors to the accumulated errors of the following states and actions; the information loss caused by using neuronal elastic learning will iterate larger errors to the accumulated errors of the following states and actions. Therefore, multiple Q strategies are used for improvement:
[0074]
[0075] Among them, Q1, Q2, ..., Q K represents K independent Q-value functions, and They represent the two smallest Q values among the K Q values.
[0076] Maximize a policy function by formula 5 To optimize a value function about state s; specifically It is a strategy function under state s, which selects two smaller Q values from multiple Q values and takes the average value to reduce the information loss caused by neuron elastic learning, and takes the maximum expected value to help the agent select the action with the highest value and achieve better decision-making. The present invention maximizes the strategy function To optimize the value function of state s, by selecting the two smaller Q values among multiple Q values and Taking the average reduces the model variance and improves sample efficiency.
[0077] Experimental environment configuration
[0078] The experimental environment is Windows 11+Pytorch 1.8, and the hardware is i9-13900HX@2.20GHz processor and 16GB memory; the simulation platform is the Gazebo simulation platform, which is a physical simulation platform model that can be used to simulate various robots, sensors and environments; the present invention conducts experiments in static and dynamic environments respectively; in all experiments, the model parameters used by the robot are consistent; at the same time, there is no need to understand or obtain any prior knowledge about the experimental environment in advance; experiments are conducted in static and dynamic environments respectively, and algorithms that have been tested and applied in many fields are selected, including SAC, TD3, DDPG and MQSF-AC, which are trained for 2000 rounds respectively, and evaluated for 100 rounds to analyze the reward size and obstacle avoidance success rate of the algorithm; details of the MQSF-AC model parameters are shown in Table 1.
[0079] Table 1MQSF-ACH Hyperparameters
[0080]
[0081] Static environment:
[0082] In order to verify the performance of the path planning of the Actor-Critic Algorithm in different scenarios, the reliability of the algorithm is first verified in a static environment; in a map environment, the target point will be randomly set, and then the performance of the mobile robot will be tested using SAC, TD3, DDPG and the MQSF-AC model of the present invention; whenever the robot encounters an obstacle or reaches the target point, a new starting position will be randomly generated on the map, which gives the robot the opportunity to contact more environments, thereby improving its exploration ability; the environment model and the 3D point cloud map are shown in Figure 2. Figure 4 As shown:
[0083] In a static environment, SAC, TD3 and the proposed model are used for path planning tasks, and their average reward values and collision rates are compared; each model is trained 2000 times and evaluated for 100 rounds; Figure 5The average reward of each round in 100 evaluations is recorded, where a round refers to a set of cycles containing ten evaluations; each point in the figure represents a round, and the curve shows the changing trend of the average reward in these rounds, which can more intuitively view the trend of the overall reward; as shown in the figure, in the TD3 algorithm, since the agent explores the environment in the initial stage, the robot is more likely to collide in the first 20 rounds of evaluation, resulting in most of the rewards being negative; the forgetting mechanism that deletes low-activity neurons is added to TD3 to retain highly active neurons, so it can converge faster than the TD3 algorithm; the SAC algorithm achieves stronger exploration capabilities and higher stability by using random strategies and the principle of maximum entropy; the present invention proposes a unique forgetting mechanism that combines the strategy of deleting low-activity neurons and multiple Q values to achieve rapid convergence while maintaining a certain stability; Figure 6 It is a comparison of four algorithms. The average reward of MQSF-AC proposed in this invention is about 80 after convergence, the average reward of SAC after convergence is about 45, and the average reward of TD3 and DDPG after convergence is about 50. The algorithm proposed in this invention can obtain a larger reward than the other three algorithms, which means that more target points can be reached within the same number of steps; Figure 7 It is a box plot of the success rates of the four algorithms; in the early stage of exploration, the average success rate of DDPG is 0.563, the average success rates of TD3 and SAC are 0.44 and 0.57, and the average success rate of MQSF-AC is 0.64; the average success rate of MQSF-AC is about 13%, 45% and 12.3% higher than that of the other three algorithms in the early stage; after convergence, the average success rates of DDPG, TD3 and SAC are 0.78, 0.742 and 0.7 respectively, and the average success rate of MQSF-AC is 0.91; Figure 7 The median and average success rates of MQSF-AC are higher than those of the other three algorithms.
[0084] Dynamic and static environment:
[0085] The effects of MQSF-AC, TD3, DDPG and SAC were tested in dynamic and static environments respectively. There are two more moving obstacles in the dynamic and static environments compared to the static environments. The environmental model is as follows: Figure 8 As shown, the objects framed by circles are dynamic obstacles, and the rest are static obstacles.
[0086] In the dynamic and static environments, MQSF-AC, SAC, TD3 and DDPG are also used for path planning tasks, and their average reward values and collision rates are compared; Fig. 9The average rewards of each round of the four algorithms were compared. MQSF-AC performed best in most rounds, with the highest rewards, and had smaller fluctuations in the first 50 rounds than the other three algorithms. The rewards of TD3 and DDPG algorithms fluctuated greatly, and the rewards of SAC algorithm were relatively stable, but lower than MQSF-AC. The convergence speed and overall average reward of MQSF-AC were significantly improved. Fig.10 The box plots of the success rates of the four algorithms are used to compare the 100-round synthesis success rates of the four algorithms in dynamic and static environments. In the evaluation, in the first 30 rounds, DDPG had an average success rate of 0.51 and an average reward of approximately 87.438; TD3 and SAC had average success rates of 0.57 and 0.58, and obtained average rewards of 69.107 and -412.185; MQSF-AC had an average success rate of approximately 0.63 and an average reward of approximately 116.454. During the exploration period, MQSF-AC increased the average reward of the best DDPG by about 33.18%, and increased it by about 128.28% compared with the worst performing SAC. After convergence, the average success rate of DDPG was about 0.74, and the average reward was 1017.788. The average success rates of TD3 and SAC were 0.75 and 0.821, and the average rewards were 777.09 and 907.134. The average success rate of MQSF-AC was about 0.85, and the average reward was about 1221.405. Compared with the other three algorithms, the success rate was increased by up to 14.86%, and the average reward was increased by up to 57.18%. Due to the introduction of dynamic obstacles, the exploration of the strategy will be more difficult than that of the static environment. In order to reduce the penalty caused by collision, the exploration of the environment becomes more negative, and the reward value obtained will also be reduced. It can be found that the neuron elastic learning of multiple Q values enables the algorithm to explore and learn faster in the early stage, and improves the average reward and success rate of the agent in the early exploration.
[0087] The obstacle avoidance path planning algorithm based on MQSF-AC proposed in the present invention enables the mobile robot to quickly learn and adapt to complex environments in the absence of prior knowledge; neuron elastic learning is introduced in the Actor-Critic network, and by comparing the size of the reward of each round with the set threshold reward, it is determined whether to call neuron elastic learning, that is, to randomly re-assign neurons with smaller feedback to input data to retain highly active neurons; at the same time, multiple Q value concepts are introduced in the SAC network to reduce the error accumulation caused by neuron elastic learning; in addition, the path planning and obstacle avoidance capabilities of the four algorithms are compared in static and dynamic and static environments respectively. The results show that the MQSF-AC algorithm proposed in the present invention can learn better strategies faster in the early exploration than SAC, TD3 and DDPG, and also shows good path planning capabilities after convergence.
[0088] Based on the above ideal embodiments of the present invention, the relevant staff can make various changes and modifications without departing from the technical concept of the present invention through the above description. The technical scope of the present invention is not limited to the contents of the specification, and its technical scope must be determined according to the scope of the claims.
Claims
1. A path planning method based on neuron elastic learning and multi-Q value strategy, characterized in that: The following steps are involved: Step 1: Build a path planning system for mobile robots in unknown environments and use the Actor-Critic network for strategy learning. Step 2: Initialize the parameters of the Actor and Critic networks; distinguish the primary and secondary functions of the neurons in each layer of the Actor network, delete the secondary neurons and assign random values, and retain what the primary neurons have learned.
2. The path planning method based on neuron elastic learning and multi-Q value strategy according to claim 1 is characterized in that: Step 2 specifically includes: Let N = {n1, n2, n3, ...} be all neurons in a layer, and T = {t1, t2, t3, ...} be the test input; By adding N i All neurons and percentage coefficients in Multiply, then perform linear offset and rounding operations, and finally get the area of the desired behavior at this level. The formula is: Among them, a is the adjustable coefficient, F r Represents the forgetting rate of previous information.
3. The path planning method based on neuron elastic learning and multi-Q value strategy according to claim 2 is characterized in that: Step 2 also includes: using the speed update factor to update the primary and secondary functional behaviors, the formula is: Where L is the loss function; z [l] , W [l] and b [l] Represent the weighted sum output, weight and bias of the lth layer, z [l] =W [l] ·q [l-1] +b [l] ;q [l] represents the activation function g of the lth layer [l] Right [l] The derivative of ; η is the speed update factor.
4. The path planning method based on neuron elastic learning and multi-Q value strategy according to claim 1 is characterized in that: It also includes step 3, using multiple Q values in the Critic network and selecting the average of the two smallest Q values as the strategy evaluation target. The formula is: Among them, Q1, Q2, ..., Q K represents K independent Q-value functions, and They represent the two smallest Q values among the K Q values.
5. The path planning method based on neuron elastic learning and multi-Q value strategy according to claim 2 is characterized in that: The adjustable coefficient is 0.
5.
6. The path planning method based on neuron elastic learning and multi-Q value strategy according to claim 3 is characterized in that: The speed update factor is used to adjust the update rate of the neuron.
7. The path planning method based on neuron elastic learning and multi-Q value strategy according to claim 1 is characterized in that: Step 1 specifically includes: Step 11: Build a training environment, generate wall obstacles, column obstacles, dynamic obstacles, and target points, and visualize the robot's perception of the environment; Step 12: Design a random dynamic environment generation module to control the generation probability and quantity range of walls and obstacles of different shapes; Step 13: Detect the environment; Step 14: perform equidistant sampling on the original laser scanning data to obtain laser ranging information as state input; Step 15: Reset the robot state and initialize the environment, and obtain training data through the interaction between the robot and the environment; Step 16: Store the training state transition data, execution actions, and immediate rewards of corresponding environment feedback in the experience replay pool for the Actor network and the Critic network.
8. A path planning system based on neuron elastic learning and multi-Q value strategy, characterized in that: include: a memory for storing instructions executable by a processor; A processor, used to execute instructions to implement the path planning method based on neuron elastic learning and multi-Q value strategy as described in any one of claims 1-7.
9. A computer readable medium storing a computer program code, characterized in that: When the computer program code is executed by a processor, the path planning method based on neuron elastic learning and multi-Q value strategy as described in any one of claims 1 to 7 is implemented.