Vehicle path planning intelligent algorithm based on improved integrated reinforcement learning

Abstract

The invention discloses a vehicle path planning intelligent algorithm based on improved integrated reinforcement learning. The vehicle path planning intelligent algorithm specifically comprises the following steps: step 1, constructing a simulation environment; step 2, constructing a Markov decision model; 3, solving an optimal path by adopting a Maxmean-DQN algorithm, and obtaining an optimal strategy of vehicle path planning; according to the Maxmean-DQN algorithm, vehicle path planning in a complex environment can be efficiently and quickly realized, a vehicle can be prevented from colliding with an obstacle, repeated paths are effectively avoided, the problems of incomplete search, repeated existing paths and low efficiency in existing path planning are solved, and adaptive optimal path planning is realized; and obstacle collision is set in a reward function of the Markov decision model, an existing route is repeated to serve as punishment, and particularly, great punishment is set for obstacle collision, so that loss is avoided.

Description

Technical Field

[0001] The present invention relates to the field of vehicle path planning, and specifically, to an intelligent algorithm for vehicle path planning based on improved integrated reinforcement learning, which is used to implement path planning for vehicles in different scenarios and find the optimal path. Background Art

[0002] With the continuous development of modern science and technology, vehicles have been more and more widely used in all walks of life in social life. How to carry out effective and reasonable path planning is one of the important problems faced by intelligent vehicles.

[0003] The diversity, complexity and unpredictability of the environment will lead to problems such as incomplete search, slow convergence speed, and repetition of existing routes during path planning. Therefore, during the path planning process, it is not only required that the vehicle has a certain self-learning ability for the environment, but also has a certain self-adaptability to obstacles and interferences in the environment. At present, reinforcement learning learns through the "trial and error" method, can interact with the external environment autonomously and in real time without prior system information; at the same time, it can perform autonomous learning and online learning, and thus has been widely used in vehicle path planning methods. However, the existing reinforcement learning methods are not only restricted by high-dimensional states, but also still have problems such as incomplete search and repetition of existing routes in multi-obstacle and complex environments, and need to be improved urgently. Summary of the Invention

[0004] One technical problem to be solved by this application is to overcome the defects of the above related technologies, and provide an intelligent algorithm for vehicle path planning, based on improved integrated reinforcement learning, to intelligently, efficiently and reasonably plan paths, and solve problems such as incomplete search, repetition of existing routes, and low efficiency in path planning.

[0005] One technical solution adopted by the present invention to solve the technical problem is: an intelligent algorithm for vehicle path planning based on improved integrated reinforcement learning, which specifically includes the following steps: Step 1, construct a simulation environment, set some obstacles in the simulation environment, and set the starting point position and the ending point position of the vehicle; Step 2, construct a Markov decision model, and set a state set representing the position of the vehicle S an action set representing the action selection of the vehicle A and a reward function representing the reward value or penalty value given under a certain action in a certain state during the operation R ; Step 3, solve the optimal path; In Step 3, the Maxmean-DQN algorithm is used to solve the optimal path, specifically: Put M parallel Q functions , The average value of Q is used as the ; In the formula, is the average value of the current M th Q function, is the h th Q value function of the current, represents taking the average of ; where is the h th Q function estimator to calculate the value of performing different actions under a certain state Q ; The vehicle uses the -greedy algorithm to select the optimal action; ; In the formula, represents the optimal action; represents returning the current selected action of the vehicle that makes the Q value function in the Q network the largest , represents the exploration probability; represents Q network parameters; and the following formula is used to Q update the value function; ; In the formula, represents the learning rate, represents the current reward value, is the discount factor, is the state when performing a certain action to obtain the maximum Q value function, represents the next state; The vehicle obtains the optimal action at each moment from the starting point to the end point through repeated updates and interactions with the simulation environment, and obtains the optimal path.

[0006] Compared with the related technologies, the present application has the following advantages: The Maxmean-DQN algorithm is used to solve the optimal path, and the optimal action at each moment from the starting point to the end point is obtained. The optimal strategy is obtained by maximizing the cumulative discounted reward value, which can efficiently and quickly realize the vehicle path planning in a complex environment, effectively avoid obstacles, and avoid taking repeated routes, so as to obtain an intelligent, efficient and reasonable path planning.

[0007] Preferably, The reward function at a moment is: . Reinforcement learning obtains the optimal strategy by maximizing the cumulative reward value. Therefore, the setting of the reward function is crucial. The reward function is set according to whether the vehicle reaches the end point, hits an obstacle, or repeats an existing route at this time: When the vehicle hits an obstacle, it means that the currently selected action is unreasonable and the next state is negative. At this time, a large penalty should be given to reduce the number of collisions and avoid losses; when the vehicle reaches the end point, it means that the currently selected action is reasonable and a reward value should be given; when repeating a route, it means that the vehicle has taken a detour and a certain penalty value is given. Brief Description of the Drawings

[0008] Figure 1 is the overall framework diagram for realizing the optimal vehicle path planning of the present invention.

[0009] Figure 2 is the simulation environment for vehicle path planning.

[0010] Figure 3 is the vehicle path planning result diagram in the Figure 2 simulation environment by using the vehicle path planning intelligent algorithm based on improved integrated reinforcement learning of the present invention.

[0011] Figure 4 is the vehicle path planning result diagram in the Figure 2 simulation environment by using the vehicle path planning method that is the same as the first two steps of the present invention but uses the DQN algorithm to solve the optimal path in step 3.

[0012] Figure 5 is the vehicle path planning result diagram in the Figure 2 simulation environment by using the vehicle path planning method that is the same as the first two steps of the present invention but uses the A* algorithm to solve the optimal path in step 3.

[0013] Figure 6 is the path planning length comparison diagram of the vehicle path planning method of the present invention and two other vehicle path planning methods.

[0014] Figure 7 is the path planning time comparison diagram of the vehicle path planning method of the present invention and two other vehicle path planning methods. Specific embodiments

[0015] First of all, those skilled in the art should understand that these embodiments are only used to explain the technical principles of the embodiments of this application, and are not intended to limit the protection scope of the embodiments of this application. Those skilled in the art can adjust them as needed to adapt to specific application scenarios.

[0016] The present invention will be further described in detail below with reference to specific embodiments.

[0017] This preferred embodiment is an intelligent algorithm for vehicle path planning based on improved integrated reinforcement learning, which specifically includes the following steps: Step 1: Construct a simulation environment Build a virtual path planning environment through simulation to simulate how a vehicle makes an effective and reasonable path plan from the starting point to the end point in a certain environment. The constructed simulation environment is as Figure 2 shown. In the simulation environment, set some obstacles to increase the difficulty of path planning and better simulate the real environment. The obstacles are represented by gray long strips; the solid circle with a slash represents the vehicle, and the position where the vehicle is located at this time is also the starting point; the solid circle with a cross slash represents the end point. The purpose of the present invention is to find an optimal path for the vehicle from the starting point position to the end point position in this simulation environment.

[0018] Step 2: Construct a Markov decision model When using reinforcement learning for path planning of the vehicle, first construct a Markov decision process (MDP) model. The MDP model includes states, actions, reward functions, etc., specifically: (1) Action set In the simulation environment, the vehicle needs to make different action choices according to its current position. The action set A is denoted as: .

[0019] (2) State set When the vehicle takes a certain action at the current position, the state will change. Define the current position of the vehicle as the state, that is: , where represents the current position of the vehicle. The state set S is denoted as: .

[0020] (3) Reward function The reward function is used to represent the reward value given at a certain state and a certain action during the running process. The reward value given according to the vehicle position at a certain state and a certain action is a reward value or a penalty value. Reinforcement learning obtains the optimal policy by maximizing the cumulative reward value. Therefore, the reward function R setting is crucial. In this embodiment, the reward function is set by whether the vehicle reaches the end point, whether it hits an obstacle, and whether it repeats the existing route at this time. When the vehicle hits an obstacle, it means that the currently selected action is unreasonable and the next state is negative. At this time, a penalty value should be given; when the vehicle reaches the end point, it means that the currently selected action is reasonable and the next state is positive. At this time, a reward value should be given; when repeating the route, it means that the vehicle takes a detour and a certain penalty value is given. Therefore, the reward function at time t is defined as: (1) It can be seen that compared with the reward obtained for reaching the end point, a larger penalty is set for hitting an obstacle, aiming to reduce the number of collisions of the vehicle and avoid losses.

[0021] Step 3: Use the Maxmean-DQN algorithm to solve the optimal path The purpose of realizing the vehicle path planning is to find an optimal policy , so that the vehicle makes the best policy choice in the presence of obstacles and realizes path optimization. The choice of policy is obtained by the vehicle repeatedly trying and making mistakes in the simulation environment. Finally, the policy with the largest sum of immediate rewards and future reward values under a certain policy is the optimal policy.

[0022] Specifically: At a certain moment, the vehicle is at a certain position (i.e., a certain state) and has a corresponding reward value. According to the reward value, the action that the vehicle should execute currently is determined, and then the position of the vehicle changes and the corresponding reward value is received. In this process, the present invention uses an improved integrated reinforcement learning algorithm (Maxmean-DQN algorithm) to implement it. The overall framework diagram of the improved integrated reinforcement learning algorithm model is as Figure 1 shown.

[0023] The Maxmean-DQN algorithm obtains the best policy by maximizing the cumulative discounted reward value. The core idea is to calculate the Q value of taking a certain action a in a certain state s through M identical parallel Q functions (denoted as Qh(s,a), where Qh(s,a) represents the Q value calculated by the h-th Q function estimator) shared by the neural network. Q function , ( represents the Q value calculated by the h-th Q function estimator Q Q value for Qvalue. Different from the traditional deep Q learning (DeepQ-Network, DQN) algorithm that directly uses a single Q function estimator to select the maximum value, the Maxmean-DQN algorithm takes the mean of the outputs of multiple parallel Q networks as the estimated Q value, and then selects the maximum value from it as the basis for the vehicle's action selection.

[0024] Specifically, the average value of M Q functions is used as the Q value function, which is defined as: (2) where is the average value of the current M Q functions, is the current h th Q value function, represents taking the average of .

[0025] The vehicle uses the -greedy algorithm to perform optimal action selection, and the specific formula is: (3) where represents the optimal action; represents returning the vehicle's current selected action Q in the Q network that makes the value function the largest, represents the exploration probability; represents Q network parameters.

[0026] And the following formula is used to update the Q value function: (4) where represents the learning rate, represents the current reward value, represents the next state, is the discount factor, is the maximum value function obtained by performing different actions a at state Q .

[0027] Through repeated updates and interactions between the vehicle and the simulation environment, the optimal actions at each moment from the starting point to the ending point are obtained, effectively avoiding obstacles, preventing taking repeated routes, achieving adaptive optimal path planning, and obtaining the optimal path.

[0028] Using the intelligent vehicle path planning algorithm based on improved integrated reinforcement learning, according to Figure 2 in the simulation environment for simulation analysis, and making a simulation comparison with the traditional DQN algorithm and A* algorithm. The path planning results are respectively as Figure 3 、 4 and 5 shown. From Figure 3 and Figure 4 and Figure 5 comparisons, it can be seen that the vehicle path planning method proposed by the present invention is the best and can achieve more efficient and reasonable path planning in a complex environment. The main reason is that the improved integrated reinforcement learning algorithm can more effectively prevent the vehicle from hitting obstacles and taking repeated routes, avoiding their adverse effects on path planning. The comparison results of the path planning lengths and required times of different algorithms are respectively as Figure 6 and Figure 7 shown. From Figure 6 and Figure 7 it can be seen that the vehicle path planning method of the present invention performs best in terms of path length and has the shortest path. In addition, the differences in running times among the three algorithms are small and all within an acceptable range. This result shows that the vehicle path planning method of the present invention can not only effectively optimize the path planning length but also be comparable to other algorithms in terms of running time, having a good performance balance.

[0029] As mentioned above, it is only the specific implementation manner of this application, but the protection scope of this application is not limited thereto. Any changes or substitutions that can be easily thought of by those skilled in the art within the technical scope disclosed by this application should be covered within the protection scope of this application. Therefore, the protection scope of this application should be subject to the protection scope of the claims.

Claims

1. An intelligent algorithm for vehicle path planning based on improved integrated reinforcement learning, specifically including the following steps: Step 1: Construct a simulation environment, set some obstacles in the simulation environment, and set the starting point position and ending point position of the vehicle; Step 2: Construct a Markov decision model and set up a state set representing the vehicle's location S , an action set representing the vehicle's action selection A , and a reward function representing the reward value or penalty value given for a certain action in a certain state during operation R ; Step 3: Solve the optimal path; It is characterized in that In step 3, the Maxmean-DQN algorithm is used to solve the optimal path, specifically: Take M parallel Q functions , and use the average value as Q the value function; ； In the formula, is the average value of the current M number of Q functions, is the h number of Q value functions for the current, represents taking the average of ; among them, is the h number of Q function estimators that calculate the value of performing different actions under a certain state Q ; The vehicle adopts -greedy algorithm to make optimal action selection; ； In the formula, represents the optimal action; represents returning the Q current selected action of the vehicle that makes the Q value function in the network the largest , represents the exploration probability; represents the Q network parameters; And the following formula is used for Q the update of the value function; ； wherein, represents the learning rate, represents the current reward value, is the discount factor, is the state for performing different actions the maximum Q value function obtained, represents the next state; Through repeated update interactions between the vehicle and the simulation environment, the optimal action at each moment from the starting point to the ending point is obtained, and the optimal path is obtained.

2. An intelligent algorithm for vehicle path planning based on improved integrated reinforcement learning according to claim 1, characterized in that, The reward function at a moment is as follows: 。

Images