Path planning method of noise D3QN algorithm with N-step updating
By introducing an N-step update-based noisy D3QN algorithm into path planning, and utilizing learnable noise perturbation and multi-step cumulative reward, the path planning problem of traditional deep reinforcement learning methods in dynamic environments is solved, achieving more efficient path planning results.
Patent Information
- Application Number
- CN202511528336.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-10-24
- Publication Date
- 2026-01-09
AI Technical Summary
Traditional deep reinforcement learning methods suffer from problems such as unstable Q-value estimation, insufficient exploration, and sparse rewards in the early stages of training, which makes it difficult to improve the convergence speed and accuracy of the objective function, and makes it difficult to achieve efficient path planning in dynamic environments.
An N-step update-based noisy D3QN algorithm is adopted. By introducing learnable noise perturbation into the neural network and combining it with multi-step cumulative rewards, an N-step update strategy is designed to improve the accuracy of exploration and path planning.
It improves the accuracy and stability of path planning, reduces short-sightedness, and enhances the efficiency and robustness of path planning.
Smart Images
Figure CN121297883A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the field of path planning algorithm, in particular to a path planning method of N-step updating noise D3QN algorithm. BACKGROUND
[0002] With the rapid development of artificial intelligence and intelligent manufacturing technology, mobile robots have shown great potential in many fields such as industry, logistics, and service. As one of its core functions, the path planning task requires the robot to automatically generate a safe and effective path from the starting point to the target point in an unknown or partially known environment, while meeting multiple constraints such as obstacle avoidance, shortest path, and energy optimization. Traditional graph search algorithms such as A* algorithm, Dijkstra algorithm, genetic algorithm, ant colony algorithm, and heuristic artificial potential field method can achieve high efficiency in path search. However, these algorithms generally rely on static maps and hand-designed heuristic functions, and perform poorly in dynamic environments, continuous state spaces, or sensor noise interference. Reinforcement learning, with its characteristics of not requiring pre-built models and being able to optimize strategies gradually in environmental interactions, has become an important means to solve complex path planning problems. By constructing a reward function and continuously updating the strategy, the robot can learn the navigation strategy autonomously without explicit maps or human intervention. Deep reinforcement learning further introduces deep neural networks as strategy function approximators, greatly improving processing capabilities in high-dimensional state spaces.
[0003] However, traditional deep reinforcement learning methods have problems such as unstable Q-value estimation in the early stages of algorithm training, insufficient exploration, and sparse rewards, which make it difficult to improve the convergence speed and accuracy of the target value function.
[0004] Therefore, there is a need for a path planning method of N-step updating noise D3QN algorithm to solve the above problems. SUMMARY
[0005] To solve the above technical problems, the technical solution adopted by the present application is:
[0006] A path planning method of N-step updating noise D3QN algorithm, comprising the following steps:
[0007] Step S101: Initialize the environment.
[0008] Initialize the decay factor , the learning rate , the maximum number of iterations , the number of steps per round , the action space , the parameter update interval , the experience buffer , the N-step update queue container , and the noise variable set , N-step update parameters , network parameters of the Q-network , set the parameters of the target Q-network to .
[0009] Step S102: The neural network injects learnable noise.
[0010] Specifically, by adding learnable noise to the weights and biases of the neural network, the reinforcement learning agent can maintain a certain degree of exploration during the training process; and introducing random disturbance in Q value calculation, so that each action selection not only depends on the Q value, but also is affected by the noise. In a neural network with p inputs and q outputs, the linear layer can be represented as:
[0011] (1)
[0012] In formula (1), is the input vector of the layer (R represents the real set), is the output vector of the layer, is the weight matrix, is the bias vector, w and b need to be learned and updated. In the noise network, replace with , where ⊙ represents element-wise multiplication; replace b with , and the equivalent noise linear layer expression is shown in formula (2):
[0013] (2)
[0014] In formula (2), and are learnable parameters, and are learnable noise scale parameters, both of which need to be learned and updated using gradient descent in the noisy network, and are random noise variables. In order to generate the matrix , a vector containing q Gaussian random variables needs to be multiplied by another vector containing p Gaussian random variables, which means a total of Gaussian random variables need to be generated. Subsequently, any element and in the noise matrix and can be generated by formula (3) and formula (4) respectively:
[0015] (3)
[0016] (4)
[0017] In formula (3), (4), the specific function of function f is , which is used to convert the generated Gaussian noise. And The values of the Gaussian random variables are all subject to standard normal distribution For the decomposed noise network, the weight matrix And the bias Is initialized by independent random samples of uniform distribution, and the range of uniform distribution is . The value of .
[0018] Through the above process, that is, introducing noise disturbance in the neural network, the output of the Q value in the neural network is affected, which is a fine-tuning of the current policy, making the exploration behavior in the early stage more relevant to the current policy. At this time, the action selection becomes formula (5):
[0019] (5)
[0020] Step S103: according to The optimal action is selected according to the policy.
[0021] In order to promote the learning of the agent and balance the relationship between exploration and utilization, exploration is usually given priority in the early stage of training to collect more information and identify the relationship between different states and actions; As the training progresses, the agent gradually transitions to the utilization stage of experience data. One of the classic strategies based on this idea is The strategy promotes exploration by assigning a greedy factor A constant initial value less than 1, and selects the optimal action in the current state with a probability of . The strategy can be expressed as:
[0022] (6)
[0023] In formula (6), Indicates the total number of actions of the action set .
[0024] Step S104: store the sample In the N-step update queue container .
[0025] Specifically, the cumulative reward Instead of single-step rewards, reduce the'short-sighted' problem, thereby improving the convergence speed of the algorithm and the accuracy of the target value function, The calculation method is as shown in formula (7):
[0026] (7)
[0027] In the formula, After introducing the N-step update strategy, the single-step experience generated by the interaction between the agent and the environment Is stored in the N-step update queue container , the maximum capacity is N, once the queue is filled with N experience samples, N continuous experiences are used to generate an N-step experience and stored in the experience buffer.
[0028] Step S105: a certain amount of samples are extracted to calculate the current Q value of the target network.
[0029] Specifically, a certain amount of samples are randomly extracted from the experience buffer to calculate the current Q value of the target network, and the specific calculation formula is as shown in formula (8):
[0030] (8)
[0031] When using the N-step update strategy, in order to maintain the length of N continuous experience samples in the single-step update queue, when adding a new sample from the top, the earliest sample will be deleted from the end of the queue. At the same time, when training the improved D3QN algorithm, the storage of experience samples in the experience pool is different from that in the original D3QN algorithm. In the original D3QN algorithm, the samples stored in the experience pool are , and in the improved D3QN algorithm, the data stored are .
[0032] Step S106: update the parameters of the Q network according to the loss function.
[0033] The loss function is calculated, and the calculation method is as shown in formula (9):
[0034] (9)
[0035] Wherein, y is the target Q value, The current predicted Q value. And update the parameters of the Q network according to the gradient descent method .
[0036] Determine whether it is the maximum episode, if yes, end, if not, return to execute the training step of the noise D3QN algorithm with N-step update.
[0037] Due to the adoption of the above technical solutions, the technical progress achieved by the present application relative to the prior art is:
[0038] In order to improve the accuracy of the target Q value in the training process of the D3QN algorithm, an N-step update strategy is designed, and the cumulative reward of multiple time steps is used instead of the single-step reward.
[0039] Secondly, the learning exploration noise is added to the neural network, so that the reinforcement learning agent can maintain a certain exploration in the training process, and the random disturbance is introduced in the Q value calculation, so that each action selection not only depends on the Q value, but also is affected by the noise, so as to improve the exploration ability of the mobile robot.
[0040] Finally, further design experiments to test the influence of different N values on the success rate of the algorithm, and then determine the value of the parameter N. The simulation experiment shows that in the complex obstacle environment, the path planning method of the N-step update noise D3QN algorithm proposed in the application has obvious advantages in path planning time, length and step number, and can effectively improve the accuracy, stability and overall robustness of the algorithm in path planning. BRIEF DESCRIPTION OF DRAWINGS
[0041] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or prior art description. Obviously, the drawings in the following description only some embodiments of the application, and for those skilled in the art, other drawings can be obtained according to the structures shown in these drawings without creative labor.
[0042] Fig. 1 The flowchart of the path planning method of the N-step update noise D3QN algorithm of the application;
[0043] Fig. 2 The reward value curve of different algorithms in the simulation training environment in the application;
[0044] Fig. 3 The step number change situation diagram of different algorithms in the simulation training environment in the application.
[0045] The following specific embodiments will further illustrate the application in conjunction with the above drawings DETAILED DESCRIPTION
[0046] For those skilled in the art to better understand the technical scheme of the present application, the technical scheme in the embodiments of the present application will be described clearly and completely below in combination with the drawings in the embodiments of the present application. Unless otherwise defined, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs. The terms used in the specification of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application. The terms "first", "second", and the like in the specification of the present application, the claims and the above drawings are used to distinguish different objects, and are not used to describe a specific order. In addition, the terms "comprise" and "have" and any variations thereof, for example, a process, method, system, product or device comprising a series of steps or units, are not limited to the listed steps or units, but optionally also include steps or units not listed, or optionally also include other steps or units inherent to the process, method, product or device.
[0047] Reference herein to "embodiments" means that the particular features, structures, or characteristics described in connection with the embodiments can be included in at least one embodiment of the present application. The appearance of the phrase in various places in the specification does not necessarily all refer to the same embodiment, nor is it necessarily independent or alternative embodiments to each other. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.
[0048] Please refer to Figs. 1-3 , the embodiment of the present application provides a path planning method of N-step updating noise D3QN algorithm, comprising the following steps:
[0049] Step S101: initializing the environment.
[0050] Initialize the decay factor , the learning rate , the maximum number of iterations , the number of steps per round , the action space , the parameter update interval , the experience buffer , the N-step update queue container , the set of noise variables , the N-step update parameter , the network parameters of the Q network , set the parameters of the target Q network to .
[0051] Step S102: injecting learnable noise into the neural network.
[0052] Specifically, by adding learnable noise to the weights and biases of the neural network, the reinforcement learning agent can maintain a certain level of exploration during the training process; and by introducing random perturbation in Q value calculation, each action selection not only depends on the Q value, but also is affected by noise. In a neural network with p inputs and q outputs, the linear layer can be represented as:
[0053] (1)
[0054] In formula (1), is the input vector of the layer (R represents the real set), is the output vector of the layer, is the weight matrix, is the bias vector, and w and b need to be learned and updated. In the noise network, replace with , where ⊙ represents element-wise multiplication; replace b with , and the equivalent noise linear layer expression is shown in formula (2):
[0055] (2)
[0056] In formula (2), and are learnable parameters, and are learnable noise scale parameters, which need to be learned and updated using gradient descent in the noisy network, and are random noise variables. In order to generate the matrix , a vector containing q Gaussian random variables needs to be multiplied by another vector containing p Gaussian random variables, which means a total of Gaussian random variables need to be generated. Subsequently, any element and in the noise matrix and can be generated by formula (3) and formula (4) respectively:
[0057] (3)
[0058] (4)
[0059] In formula (3), (4), the specific function of function f is , which is used to convert the generated Gaussian noise. And the values are Gaussian random variables, all subject to standard normal distribution For the decomposition noise network, the weight matrix and the bias are initialized by independent uniformly distributed random samples, with a uniform distribution range of . The value of is set to
[0060] Through the above process, that is, introducing noise disturbance in the neural network, the output of the Q value in the neural network is affected, which is a fine-tuning of the current policy, making the exploration behavior in the early stage more relevant to the current policy. At this time, the action selection becomes formula (5):
[0061] (5)
[0062] Step S103: selecting the optimal action according to the policy.
[0063] In order to promote the learning of the agent and balance the relationship between exploration and utilization, exploration is usually given priority in the early stage of training to collect more information and identify the relationship between different states and actions; as the training progresses, the agent gradually transitions to the utilization stage of experience data. One of the classic strategies based on this idea is the strategy, which promotes exploration by assigning a greedy factor a constant initial value less than 1, and selects the optimal action in the current state with a probability of . The strategy can be expressed as:
[0064] (6)
[0065] In formula (6), represents the total number of actions in the action set .
[0066] Step S104: storing the sample in the N-step update queue container .
[0067] Specifically, by replacing the single-step reward with the cumulative reward of multiple time steps, the'short-sighted' problem is reduced, thereby improving the convergence speed of the algorithm and the accuracy of the target value function, the calculation method of is shown in formula (7):
[0068] (7)
[0069] In the formula, . After introducing the N-step update strategy, the single-step experience generated by the interaction between the agent and the environment N-step update queue container The maximum capacity is N, and once the queue is filled with N experience samples, N consecutive experiences are used to generate an N-step experience and stored in the experience buffer.
[0070] Step S105: Extract a certain amount of samples to calculate the current Q value of the target network.
[0071] Specifically, a certain amount of samples are randomly extracted from the experience buffer to calculate the current Q value of the target network, and the specific calculation formula is shown in equation (8):
[0072] (8)
[0073] When the N-step update strategy is used, in order to maintain the length of N consecutive experience samples in the single-step update queue, when a new sample is added from the top, the oldest sample will be deleted from the end of the queue. At the same time, when training the improved D3QN algorithm, the storage of experience samples in the experience pool is different from that in the original D3QN algorithm. In the original D3QN algorithm, the samples stored in the experience pool are usually , while in the improved D3QN algorithm, the data stored are .
[0074] Step S106: Update the parameters of the Q network according to the loss function.
[0075] Calculate the loss function, and the calculation method is shown in equation (9):
[0076] (9)
[0077] Where y is the target Q value, is the current predicted Q value. And update the parameters of the Q network according to the gradient descent method .
[0078] Determine whether it is the maximum episode, if yes, end, if not, return to execute the training step of the N-step update noise D3QN algorithm.
[0079] Experimental example
[0080] To verify the effectiveness of the path planning method of the N-step update noise D3QN algorithm proposed in the present application, an environment with a size of The simulation training environment includes four large cube obstacles, a large wardrobe-type obstacle, and some randomly distributed conical roadblock-type obstacles, which do not coincide with any other obstacles. Comparative experiments are performed on four algorithms, i.e., a DDQN algorithm, a D3QN algorithm, a D3QN-PER algorithm, and an NS-ND3QN algorithm proposed in the present application. The four algorithms are trained in the built training environment, and each algorithm is trained for 2000 rounds. Meanwhile, total training time, total training steps, average reward value, and step number change in the training process of different algorithms are introduced as evaluation indexes for comparative analysis of different algorithms.
[0081] Referring to Fig. 2 , Fig. 3 and Table 1, Fig. 2 a reward value curve of different algorithms in the training environment; Fig. 3 a step number change graph of different algorithms in the training environment; and
[0082] As can be seen from Fig. 2 , Fig. 3 and Table 1, the path planning method of the noise D3QN algorithm with N-step update proposed in the present application has the highest average success rate, the shortest average path length, the least average step number and average time, and has higher accuracy and robustness.
[0083] Table 1 Test results of different algorithms in the training environment (average value of 10 times)
[0084]
[0085] The above has made a detailed description of the present application in general, but some modifications or improvements can be made on the basis of the present application, which is obvious to those skilled in the art. Therefore, the modifications or improvements without departing from the spirit of the present application are within the protection scope of the present application.
Claims
1. A path planning method of an N-step updated noise D3QN algorithm, characterized in that: Comprising the following steps: Step S101: initializing the environment, obtaining the initial state; Step S102: injecting learnable noise into the neural network; Step S103: According to selects the optimal action according to the policy Step S104: Storing the sample into the N-step update queue container ; Step S105: extracting a certain amount of samples to calculate the current Q value of the target network; Step S106: updating the parameters of the Q network according to the loss function.
2. The path planning method of claim 1, wherein: In step S101, an attenuation factor is initialized , a learning rate , a maximum number of iterations , a number of steps per episode , an action space , a parameter update interval , an experience buffer , an N-step update queue container , a set of noise variables , N-step update parameters , network parameters of a Q network , set the parameters of a target Q network to .
3. The path planning method of claim 2, wherein: In step S102, by adding learnable noise to the weights and biases of the neural network, and introducing random disturbance in Q value calculation, in a neural network with p inputs and q outputs, the linear layer can be represented as: (1) In formula (1), is an input vector of the layer (R denotes the real number set), is an output vector of the layer, is a weight matrix, is a bias vector, w and b need to be learned and updated; In a noisy network, we have replaced by , where denotes element-wise multiplication; and b is replaced by The expression of the equivalent noise linear layer is given by equation (2): (2) In formula (2), and are learnable parameters, and are learnable noise scale parameters, all of which need to be learned and updated using gradient descent in a noisy network; and are random noise variables; to generate the matrix , a vector containing q Gaussian random variables and another vector containing p Gaussian random variables are multiplied, a total of Gaussian random variables are needed to be generated; subsequently, any element and in the noise matrix and can be generated by formula (3) and formula (4) respectively: (3) (4) In formula (3), (4), the specific function of function f is , which is used to convert the generated Gaussian noise; The values are Gaussian random variables, all subject to standard normal distribution For the decomposition noise network, the weight matrix And the bias Is initialized by independent uniformly distributed random samples, and the uniform distribution range is ; The value of is set to ; Introducing noise disturbance in the neural network, thereby affecting the output of the Q value in the neural network, the action selection becomes formula (5): (5)。 4. The path planning method of the N-step updated noise D3QN algorithm according to claim 3, wherein: In step S103, The policy promotes exploration by assigning a greed factor of less than 1 to the initial value and a probability of selecting the optimal action in the current state; The policy can be expressed as: (6) In formula (6), representing the total number of actions of the action set representing the total number of actions of the action set 5. The path planning method of claim 4, wherein: In step S104, the cumulative reward from the interactions at multiple time steps Instead of single-step rewards, the'short-sighted' problem is reduced, thus improving the convergence speed of the algorithm and the accuracy of the target value function, The calculation method is shown in formula (7): (7) In the formula, ; after introducing the N-step update strategy, the single-step experience generated by the interaction between the agent and the environment is stored in the N-step update queue container , the maximum capacity is N, once the queue is filled with N experience samples, N continuous experiences are used to generate an N-step experience and stored in the experience buffer; when using the N-step update strategy, in order to maintain the length of N continuous experience samples in the single-step update queue, when adding a new sample from the top, the earliest sample will be deleted from the end of the queue; at the same time, when training the improved D3QN algorithm, the storage of experience samples in the experience pool is different from that in the original D3QN algorithm, in the original D3QN algorithm, the samples usually stored in the experience pool are , while in the improved D3QN algorithm, the data stored are 6. The path planning method of the N-step updating noise D3QN algorithm according to claim 5, wherein: In step S105, a certain amount of samples are randomly extracted from the experience buffer to calculate the current Q value of the target network, and the specific calculation formula is shown in formula (8): (8) 7. The path planning method of the N-step updating noise D3QN algorithm according to claim 6, wherein: In step S106, the loss function is calculated, and the calculation method is shown in formula (9): (9) wherein y is a target Q value, is a current predicted Q value; and parameters of the Q network are updated according to a gradient descent method ; Determine whether it is the maximum episode, if yes, end, if not, return to execute the training step of the N-step updating noise D3QN algorithm.