A path planning method based on deep reinforcement learning-fast exploration random tree

Abstract

The application discloses a path planning method based on deep reinforcement learning-fast exploration random tree, comprising the following steps: S1, obtaining a starting point and an ending point; S2, calculating candidate path points and selecting an action in an action state according to a Q value; S3, calculating a reward value and a new action state after the action is executed; S4, storing the action state, the action, the reward value and the new action state to an experience pool, in response to the number of stored experience values in the experience pool being greater than a batch size, randomly selecting experience values of the batch size, and updating the Q value and a time difference error through a policy network; S5, updating policy network parameters through a mean square error loss, and calculating a target network update step according to the time difference error; S6, judging whether a searched path reaches the ending point or satisfies a set maximum path point search number, if yes, outputting a current path, and if not, returning to S2. The application improves the search efficiency of the algorithm without increasing the search time of the algorithm.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of path planning of intelligent agents in a nuclear radiation environment, and particularly relates to a path planning method based on deep reinforcement learning and a rapidly-exploring random tree. BACKGROUND

[0002] Under the restriction of complex nuclear emergency external environmental conditions, the nuclear emergency robot itself system, equipment and components will be seriously damaged. Path planning in combination with its own performance, task requirements and environmental information helps to reduce the radiation damage to the nuclear emergency robot and ensure that the robot safely and efficiently completes the task target.

[0003] Under the current rapid development of artificial intelligence, the robot path planning method has developed from the initial traditional graph search-based method, random sampling-based method to the current swarm intelligence algorithm and reinforcement learning algorithm (RL). The representative algorithms of the graph search-based method are Dijkstra algorithm, A* algorithm and Bellman-Ford algorithm. These algorithms have achieved good results in solving path planning problems in different scenarios, but the search performance will decrease significantly when solving multi-objective and NP-hard problems. The representative algorithms of the random sampling-based method are Rapidly-exploring Random Tree (RRT) and Rapidly-exploring Random Tree Star (RRT*). These path planning algorithms have good local search ability and search efficiency, but the generated path is usually not optimal and may have more redundancy and twists and turns, and the search efficiency will decrease in a high-dimensional complex environment. The representative swarm intelligence-based path planning algorithms are ant colony algorithm and genetic algorithm. Although the swarm intelligence algorithm has good global optimization ability and is suitable for complex path planning scenarios, its computational complexity will increase with the change of the scene complexity, the algorithm search time will become longer, and the solution space of the problem is searched by repeated iteration, so the swarm intelligence algorithm has the defects of local optimum and low solution accuracy.

[0004] Although researchers have gradually used the above three types of path planning algorithms to solve the minimum radiation dose path planning problem in a radiation environment, due to the radiation and uncertainty of the nuclear radiation environment, factors such as radiation dose, distance, adaptability of different types of maps need to be considered in the solving process, therefore, the above algorithms have limitations in solving multi-objective path planning problems with different types of nuclear radiation maps.

[0005] In order to improve the interaction ability and adaptability of the path planning algorithm and the environment, a reinforcement learning algorithm mode Q-learning is developed and proposed by researchers, and Q-learning generates a strategy by establishing a table, that is, a Q-table, to map the relationship between the state and the behavior of the unmanned aerial vehicle. The path planning using reinforcement learning can interact with the environment in real time, and the agent will receive feedback rewards or punishments from the environment, so as to continuously adjust the strategy to achieve the purpose of planning the optimal path. At present, a part of research work focuses on the reinforcement learning method based on value function for strategy evaluation, for example, the unmanned aerial vehicle multi-destination path planning method based on Q-learning, which evaluates the pros and cons of the strategy through the Q-value function, so as to determine the next action of the unmanned aerial vehicle. In a complex environment, the robot and the path planning process are a set of continuous actions, and the state of the unmanned aerial vehicle is diversified and complex, which is difficult to record and evaluate the action strategy. The performance of a single Q-table cannot support high-dimensional unmanned aerial vehicles and environmental data information, so a neural network is introduced to replace the Q-table, that is, a deep Q-network (DQN), which takes the behavior state of the robot as the input of the neural network, generates the output value function through network training, and greatly reduces the performance defect of the Q-table caused by excessive data in Q-learning. Although deep reinforcement learning has good autonomous learning ability and environmental adaptability, it has problems such as long training iteration period and invalid exploration in a complex environment.

[0006] Due to the particularity of the nuclear radiation environment, designing a method that can balance the efficiency of the path planning algorithm and the safety of the agent during the operation process, and enabling the agent to have good interaction ability with the environment and adaptability to the environmental map, has become the focus of the research on the path planning of the agent in the nuclear radiation environment. SUMMARY

[0007] In view of the above problems in the prior art, the path planning method based on deep reinforcement learning-fast exploration random tree provided by the present application dynamically adjusts the action state of the agent based on the DQN algorithm combined with the RRT* search strategy, improves the obstacle avoidance ability and search efficiency of the algorithm, adds the current path point evaluation mechanism in the reward function, improves the search efficiency of the algorithm without increasing the search time of the algorithm, and solves the problems of long training iteration period and invalid exploration of the agent path planning of deep reinforcement learning in a complex environment.

[0008] In order to achieve the above-mentioned purposes, the technical scheme adopted by the present application is as follows: a path planning method based on deep reinforcement learning-fast exploration random tree, comprising the following steps:

[0009] S1, obtain the start point and the end point, initialize the training parameters of the deep Q network;

[0010] S2, calculate the candidate path point according to the path node tree search strategy, and select the action in the action state according to the Q value;

[0011] S3, based on the reward function of the current path point merit evaluation mechanism, calculate the reward value and the new action state after executing the action;

[0012] S4, store the action state, action, reward value and new action state to the experience pool, in response to the number of stored experience pool being greater than the batch size, then randomly select the experience value of the batch size, and update the Q value and the time difference error through the strategy network;

[0013] S5, update the strategy network parameters through the mean square error loss, and calculate the target network update step according to the time difference error;

[0014] S6, judge whether the searched path reaches the end point or meets the set maximum path point search number, if yes, output the current path, if not, return to S2.

[0015] Further, in S2, the action is selected in the action state according to the Q value The expression is specifically:

[0016]

[0017] In the formula, indicates the action corresponding to the maximum Q value in the action state s t a t rand( m 1) indicates that the action is randomly selected within the length range of the dynamic state m 1 a t to avoid the algorithm falling into local optimum, is a constant, rand is a random number, is the parameter of the network.

[0018] Further, in S2, the expression for calculating the candidate path point is specifically:

[0019]

[0020]

[0021] In the formula, S indicates the action state matrix, R n indicates the current path point number,​R N This indicates the preset number of path points. m This represents the maximum number of candidate path points. r Indicates in m The last path point among the candidate path points f m Indicates calculation separately m The matrix formed by the cumulative radiation dose of each candidate path point and the searched paths. dose i and dose i+1 These represent the paths that have been searched and found. i The and the first i Radiation dose for +1 path point coordinates, R This indicates the number of path points that have been found. This indicates the cumulative radiation dose along the searched path. This means sorting the elements in the matrix from smallest to largest and selecting the first few. m One element, This represents the last waypoint in the searched path. dose R to candidate points in the action state matrix dose j Total cumulative radiation dose matrix, In the action state matrix The distance matrix from each candidate path point to the destination. This represents the distance from a candidate point in the action state matrix to the endpoint. Indicates in m Among the candidate path points, the th i The coordinates of each path point. Indicates the coordinates of the endpoint. This indicates the number of path points selected from the candidate path points. This means sorting the elements in the matrix from smallest to largest and selecting the first few. One element, This represents the maximum radiation dose that an intelligent agent can receive while walking. obs This indicates the number of obstacles around the current path point within a unit step.

[0022] The beneficial effects of the above-mentioned further scheme are as follows: A path node tree search strategy is adopted: in the early stages, based on the cumulative radiation dose and obstacle conditions, a path node tree is randomly generated from the environment. n Filtering from path points m The algorithm uses the action state as a basis to improve its obstacle avoidance and global optimization capabilities; later, the distance to the target point is used as the evaluation criterion, and the radiation dose safety threshold is used as a constraint to reselect the appropriate state. mThe algorithm improves convergence speed and search efficiency by implementing a multi-stage dynamic optimization of the action space. This approach balances safety and optimization efficiency in complex environments, enhancing the algorithm's adaptability and practicality in radiation-constrained scenarios.

[0023] Furthermore: In S3, after calculating the execution action... a t Reward value r t The specific expression is:

[0024]

[0025] In the formula, Represents the normalization function. Represents the average function, This represents the coordinates of all path points within a unit step size of the current path point. This represents the coordinates of the current path point.

[0026] The beneficial effects of the above-mentioned further scheme are as follows: By introducing a quality evaluation mechanism for the current path point into the reward function, the safety of the agent during operation is improved by calculating the average radiation dose and the number of obstacles within a unit step size; at the same time, by introducing temporal differential error to establish an adaptive step size adjustment model, the target network parameters are updated at an appropriate step size, thereby reducing the algorithm search time.

[0027] Furthermore: In S4, the expression for updating the Q-value through the policy network is specifically as follows:

[0028]

[0029] In the formula, This represents the Q-value after the policy network update. In the action state s t Take action below a t Expected returns For learning rate, As a discount factor for rewards, In the new action state The maximum expected return that can be obtained by taking different actions. In order to take new action The new state that is obtained later;

[0030] Update timing difference error The specific expression is:

[0031] .

[0032] Furthermore: In S5, the mean square error loss The specific expression is:

[0033]

[0034] In the formula, B Indicates the number of samples. Indicates the first i The true value of each sample Indicates the first i The predicted value for each sample.

[0035] Furthermore: In S5, the target network update step size is calculated. L The specific expression is:

[0036]

[0037] In the formula, This indicates rounding down. This represents the average value of the timing difference error for the current batch. This represents the standard deviation of the timing difference error for the current batch.

[0038] The beneficial effects of this invention are as follows: This invention designs a path planning algorithm that is not limited to a specific map environment. Taking into account the special characteristics of radiation environments, it improves the algorithm's search efficiency without increasing the search time, providing low-risk, collision-free adaptive operating paths for agents working in nuclear radiation environments, thereby supporting emergency decision-making for nuclear robots. Compared with existing technologies, it has the following advantages:

[0039] (1) By utilizing the ability of deep reinforcement learning algorithms to provide real-time feedback to the environment and the tree search method of the RRT* algorithm, and taking into full account the influence of obstacles and radiation environment factors, the path planning is not limited to the map construction method, and randomly selects candidate path points that meet the requirements, so as to improve the obstacle avoidance ability, search efficiency and environmental adaptability of the algorithm.

[0040] (2) A reward function is constructed by combining the radiation dose and obstacle information around the current path point, and an adaptive step size update model for target network parameters is constructed by adding time-series difference error, which reduces the algorithm search time and the radiation risk of the agent during the operation, and plans the path more accurately and efficiently.

[0041] (3) By planning a path with low radiation risk for the intelligent agent, the service life of the intelligent agent can be greatly improved, and it can also explore places that cannot be reached by personnel, thus improving emergency response efficiency. Attached Figure Description

[0042] fig. 1 This is a flowchart of a path planning method based on deep reinforcement learning and fast exploration of random trees according to the present invention.

[0043] fig. 2 Structure diagram of DQN-RRT* path planning algorithm.

[0044] fig. 3 Schematic diagram of adaptive step target network update based on timing difference error. DETAILED DESCRIPTION

[0045] The specific embodiments of the present application are described below to facilitate the understanding of the present application for those skilled in the art, but it should be clear that the present application is not limited to the scope of the specific embodiments, and for those skilled in the art, it is obvious that various changes are within the spirit and scope of the present application defined and determined by the appended claims, and all the inventions utilizing the concept of the present application are within the scope of protection.

[0046] As shown in fig. 1 and fig. 2 In one embodiment of the present application, a path planning method based on deep reinforcement learning-fast exploration random tree includes the following steps:

[0047] S1, obtaining the starting point and the ending point, and initializing the training parameters of the deep Q network;

[0048] S2, calculating the candidate path point according to the path node tree search strategy, and selecting the action in the action state according to the Q value;

[0049] S3, based on the reward function of the current path point merit evaluation mechanism, calculating the reward value and the new action state after executing the action;

[0050] S4, storing the action state, action, reward value and new action state to the experience pool, in response to the number of stored experience values in the experience pool being greater than the batch size, randomly selecting the experience values of the batch size, and updating the Q value and the timing difference error through the policy network;

[0051] S5, updating the policy network parameters through the mean square error loss, and calculating the target network update step according to the timing difference error;

[0052] S6, judging whether the searched path reaches the ending point or meets the set maximum path point search number, if yes, outputting the current path, and if no, returning to S2.

[0053] fig. 2 In the present application, represents the action selected by the prediction network with the maximum reward value, The basic idea of the application is based on the DQN algorithm, combined with the particularity of the nuclear emergency radiation environment, the RRT* algorithm path node tree search strategy is introduced to improve the selected action state (i.e. the path point that the agent can select) in the DQN algorithm, and the obstacle avoidance ability and search efficiency of the algorithm are improved; the average value of the radiation dose of the current path point in the unit step range is calculated, and the current path point evaluation mechanism is added to the reward function, which further improves the ability of the algorithm to search for a path with small cumulative radiation dose, thereby reducing the radiation risk of the agent; finally, the timing difference error is introduced to establish an adaptive step adjustment model of whether to update the target network parameters, and the search time of the algorithm is reduced.

[0054] In S1, the training parameters of the deep Q network are initialized, including:

[0055] The initialization of the experience pool D and the capacity N, the memory unit of the obstacle and the storage unit of the radiation dose distribution, the action value function and the target action value function, the action state, the batch size, the maximum search times, the target network update step and the path point set;

[0056] In S2, the action is selected in the action state according to the Q value The expression of the formula is specifically:

[0057]

[0058] In the formula, Indicates the action corresponding to the maximum Q value in the action state s t a t , rand( m 1) indicates that the action is randomly selected within the length range of the dynamic state m 1 a t , to avoid the algorithm falling into local optimum, is a constant, rand is a random number, is the parameter of the network.

[0059] In S2, the method for calculating the candidate path point is specifically:

[0060] The number of current path points R n is less than the preset number of path points R N , the path points are randomly generated in the environment, and the agent is selected m Path points are updated to the action state according to the cumulative radiation dose and obstacle situation of the agent reaching the target point, the obstacle avoidance ability and global optimization ability of the algorithm are improved; the number of current path points​R n More than the preset number of path points R N At that time, the distance from the agent to the target point is used as the evaluation criterion, and the cumulative radiation dose safety threshold is used as the constraint condition, and the target point is reselected. The action state is updated at each path point, improving the algorithm's convergence speed and search efficiency. The specific expression for calculating candidate path points is:

[0061]

[0062]

[0063] In the formula, S Represents the action state matrix. R n Indicates the number of current path points. R N This indicates the preset number of path points. m This represents the maximum number of candidate path points. r Indicates in m The last path point among the candidate path points f m Indicates calculation separately m The matrix formed by the cumulative radiation dose of each candidate path point and the searched paths. dose i and dose i+1 These represent the paths that have been searched and found. i The and the first i Radiation dose for +1 path point coordinates, R This indicates the number of path points that have been found. This indicates the cumulative radiation dose along the searched path. This means sorting the elements in the matrix from smallest to largest and selecting the first few. m One element, This represents the last waypoint in the searched path. dose R to candidate points in the action state matrix dose j Total cumulative radiation dose matrix, In the action state matrix The distance matrix from each candidate path point to the destination. This represents the distance from a candidate point in the action state matrix to the endpoint. Indicates in m Among the candidate path points, the th i The coordinates of each path point. Indicates the coordinates of the endpoint. represents the number of path points selected in the candidate path points, represents the elements in the matrix are sorted from small to large, and the first elements are selected, represents the maximum radiation dose value that the agent can accept during walking, obs represents the number of obstacles within a unit step around the current path point.

[0064] In this embodiment, the path node tree search strategy: in the early stage, according to the cumulative radiation dose and obstacle situation, from the randomly generated n path points in the environment, select m as the action state, improve the obstacle avoidance ability and global optimization ability of the algorithm; in the later stage, take the distance from the target point as the evaluation basis, and take the radiation dose safety threshold as the constraint, reselect m action state, improve the convergence speed and search efficiency of the algorithm. Through the phased dynamic optimization of the action space, the safety and optimization efficiency in the complex environment are taken into account, and the adaptability and practicality of the algorithm in the radiation constraint scene are enhanced.

[0065] In S3, the reward value of the action is calculated a t r t The expression of

[0066]

[0067] In the formula, represents a normalization function, represents an average value function, represents all path point coordinates within a unit step from the current path point coordinate, represents the current path point coordinate.

[0068] In this embodiment, the application sets the basic idea of the reward value r t of the current path point evaluation mechanism is that if the end point has been reached, a large reward value is set; if the end point has not been reached, the reward value is set according to the average value of the dose around the current path point and the number of obstacles, and it needs to be noted that in order to balance the number of obstacles and the radiation dose value, the application normalizes the results of the two to the size of the reward value.

[0069] In S4, the expression for updating the Q value through the strategy network is specifically:

[0070]

[0071] In the formula, represents the updated Q value of the strategy network, is the action state​s t Take action below a t Expected returns For learning rate, As a discount factor for rewards, In the new action state The maximum expected return that can be obtained by taking different actions. In order to take new action The new state that is obtained later;

[0072] Update timing difference error The specific expression is:

[0073] .

[0074] In S5, the mean square error loss The specific expression is:

[0075]

[0076] In the formula, B Indicates the number of samples. Indicates the first i The true value of each sample Indicates the first i The predicted value for each sample.

[0077] In S5, the target network update step size is calculated. L The specific expression is:

[0078]

[0079] In the formula, This indicates rounding down. This represents the average value of the timing difference error for the current batch. This represents the standard deviation of the timing difference error for the current batch.

[0080] Generally, a fixed step size is set to update the target network parameters without considering whether the trend of the network parameters in the current batch is changing in an optimal direction. Therefore, this embodiment calculates the average and standard deviation of the temporal difference error, and then calculates the average and standard deviation of the current batch, adaptively adjusting the target network update step size. fig. 3 As shown.

[0081] In the description of the application, it needs to be understood that the terms "center", "thickness", "upper", "lower", "horizontal", "top", "bottom", "inner", "outer", "radial" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the application and simplifying the description, and do not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the application. In addition, the terms "first", "second", "third" are only for the purpose of description, and cannot be understood as indicating or implying relative importance or implying the number of technical features indicated. Therefore, the features defined by "first", "second", "third" can explicitly or implicitly include one or more of the features.

Claims

1. A path planning method based on deep reinforcement learning-fast exploration random tree, characterized in that, The method comprises the following steps: S1, acquiring a starting point and an ending point, and initializing training parameters of a deep Q network; S2, calculating a candidate path point according to a path node tree search strategy, and selecting an action in an action state according to a Q value; S3, calculating a reward value and a new action state after the action is executed based on a reward function of a current path point merit evaluation mechanism; S4, storing the action state, the action, the reward value and the new action state to an experience pool, randomly selecting experience values of a batch size from the experience pool in response to a storage number in the experience pool being greater than the batch size, and updating the Q value and a time difference error through a policy network; S5, updating policy network parameters through a mean square error loss, and calculating a target network update step based on the time difference error; S6, judging whether a searched path reaches the ending point or satisfies a set maximum path point search number, and outputting a current path if yes, or returning to S2 if no; In S2, an expression for calculating the candidate path point is specifically as follows: In the formula, S Represents the action state matrix. R n Indicates the number of current path points. R N This indicates the preset number of path points. m This represents the maximum number of candidate path points. r Indicates in m The last path point among the candidate path points f m Indicates calculation separately m The matrix formed by the cumulative radiation dose of each candidate path point and the searched paths. dose i and dose i+1 These represent the paths that have been searched and found. i The and the first i Radiation dose for +1 path point coordinates, R This indicates the number of path points that have been found. This indicates the cumulative radiation dose along the searched path. This means sorting the elements in the matrix from smallest to largest and selecting the first few. m One element, This represents the last waypoint in the searched path. dose R to candidate points in the action state matrix dose j Total cumulative radiation dose matrix, In the action state matrix The distance matrix from each candidate path point to the destination. This represents the distance from a candidate point in the action state matrix to the endpoint. Indicates in m Among the candidate path points, the th i The coordinates of each path point. Indicates the coordinates of the endpoint. This indicates the number of path points selected from the candidate path points. This means sorting the elements in the matrix from smallest to largest and selecting the first few. One element, This represents the maximum radiation dose that an intelligent agent can receive while walking. obs This indicates the number of obstacles around the current path point within a unit step. In S3, the calculation is performed after the action a t The reward value of r t The expression of is specifically: wherein denotes a normalization function, denotes a mean value function, denotes all path point coordinates within a unit step length of the current path point coordinate, denotes the coordinate of the current path point.

2. The path planning method based on deep reinforcement learning-fast exploring random tree according to claim 1, characterized in that, In S2, an action is selected in the action state according to the Q value The expression is specifically: In the formula, Indicates the action state s t Select the action corresponding to the maximum Q value a t , rand( m 1) indicates that the action is randomly selected within the range of the dynamic state m 1 length a t Avoiding the algorithm from falling into local optimum, is a constant, rand is a random number, is the parameter of the network.

3. The path planning method based on deep reinforcement learning-fast exploring random tree according to claim 1, characterized in that, In S4, an expression for updating the Q value through the policy network is specifically as follows: wherein Qnew represents the updated Q value of the policy network, Qnew represents the updated Q value of the policy network, s t the expected return of taking action a t the expected return of taking action is the learning rate, is the discount factor of the reward, is the maximum expected return of taking a different action is the maximum expected return of taking a different action is the new state resulting from taking the new action is the new state resulting from taking the new action Updating timing difference error The expression of the updating timing difference error is specifically as follows: 。 4. The path planning method based on deep reinforcement learning-fast exploring random tree according to claim 1, characterized in that, In S5, the mean square error loss The expression of the mean square error loss is given by In the formula, B denotes the number of samples, denotes the true value of the i th sample, denotes the predicted value for the i th sample.

5. The path planning method based on deep reinforcement learning-fast exploring random tree according to claim 3, characterized in that, In S5, the target network update step is calculated L The expression is specifically: wherein denotes the floor function, denotes the average of the current batch timing difference errors, denotes the standard deviation of the current batch timing difference errors.

Images