Automatic driving decision planning method based on deep reinforcement learning and A-Star algorithm

Abstract

This invention discloses an autonomous driving decision-making and planning method based on deep reinforcement learning and the A-Star algorithm. The A-Star algorithm is used for global path planning, and a deep reinforcement learning model is used to process local environmental perception information and make real-time decisions. A reward function is set to train the neural network model, enabling it to effectively handle complex traffic environments while ensuring safety, and exhibiting good real-time performance and flexibility. A dynamic safe following distance calculation mechanism based on vehicle speed and a reward and penalty mechanism for behaviors such as deviating from the global path and approaching the vehicle in front are introduced to improve driving safety. Through simulation environment training, the agent can learn the optimal driving strategy, achieving more efficient and intelligent autonomous driving. This invention's method can efficiently and accurately provide driving decisions for autonomous vehicles and has good generalization ability and adaptability to adapt to complex and changing traffic environments.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, specifically to an autonomous driving decision-making and planning method based on deep reinforcement learning and the A-Star algorithm. Background Technology

[0002] Autonomous driving technology has made significant progress in recent years, but it still faces many challenges, especially in decision-making and planning in complex and dynamic environments. Autonomous vehicles need to perform safe and reliable path planning and real-time decision-making in various complex environments to ensure passenger safety, improve driving efficiency, and reduce traffic accidents.

[0003] Path planning is one of the core tasks of autonomous driving, typically involving determining the optimal path from a starting point to a destination in a known environment. Existing path planning techniques can be broadly categorized into two types: global path planning and local path planning.

[0004] Global path planning: Commonly used for planning long-distance paths from the starting point to the destination. The classic A-Star algorithm is widely used due to its efficiency and optimality. The A-Star algorithm uses a heuristic function to guide the search process and can find the shortest path from the starting point to the target in a known map environment. However, the A-Star algorithm assumes that the environment is static and is poorly adaptable to dynamically changing environments. Once the environment changes, such as the appearance of new obstacles, the A-Star algorithm needs to replan the path, resulting in insufficient real-time performance.

[0005] Local path planning refers to the local planning problem encountered by autonomous vehicles in dynamic environments during actual driving, especially in obstacle avoidance and handling unexpected situations. Existing methods include those based on artificial potential fields, dynamic windows, and Rapidly Expanding Random Trees (RRT). These methods have certain advantages in handling dynamic environments, but they often have limitations in solving local optima problems and struggle to achieve global optima simultaneously.

[0006] Deep reinforcement learning, as an emerging machine learning method, has demonstrated outstanding performance in multiple fields. Particularly in the field of autonomous driving, deep reinforcement learning, through interaction with the environment to learn optimal policies, has shown great potential in handling high-dimensional perceptual data and continuous action spaces. For example, existing research has used deep Q-networks (DQNs) to achieve end-to-end autonomous driving behavior learning in simulated environments. Despite the great potential of reinforcement learning in autonomous driving decision-making and planning, some challenges and limitations remain. First, reinforcement learning algorithms typically require a large number of samples to learn effective policies, and in autonomous driving scenarios, they still face problems such as low sample efficiency, slow convergence, and difficulty in policy interpretation. Second, in real-world driving environments, the safety and reliability requirements of reinforcement learning models are extremely high, and using reinforcement learning methods alone may not be able to meet these requirements.

[0007] While the A-Star algorithm excels in path planning, its application in dynamic environments is limited. Deep reinforcement learning, on the other hand, performs well in handling dynamic decision-making problems, but its global planning capabilities are weak, making it prone to getting trapped in local optima. Currently, existing technologies struggle to simultaneously achieve optimal global path planning and real-time, flexible local decision-making. Summary of the Invention

[0008] The purpose of this invention is to provide an autonomous driving decision-making and planning method based on deep reinforcement learning and the A-Star algorithm to address the above-mentioned shortcomings.

[0009] To achieve the above objectives, the present invention provides the following technical solution:

[0010] An autonomous driving decision-making and planning method based on deep reinforcement learning and the A-Star algorithm includes the following steps:

[0011] S1. Use the A-Star algorithm to perform global path planning on a known map;

[0012] S2. Construct a neural network model based on deep reinforcement learning, wherein the neural network model is responsible for processing local environmental perception information and making real-time decisions;

[0013] S3. Set the reward function for the neural network model;

[0014] S4. The neural network model receives the perceptual information acquired and transmitted by the perceptual system;

[0015] S5. Using the global path information from step S1, the reward function from step S3, and the perception information from step S4, train the neural network model from step S2.

[0016] S6. The reinforcement learning module in the neural network model is based on perceptual information and global path planning provided by the A-Star algorithm. It uses the trained neural network model to optimize local paths and then outputs a decision.

[0017] Preferably, in step S5, the deep neural network model from step S2 is trained, including training for independent scenes and training for mixed scenes.

[0018] The independent scenario is one of the following scenarios: straight road scenario, sharp curve scenario, intersection scenario, ramp scenario, and roundabout scenario; the mixed scenario is a combination of two or more of the following scenarios: straight road scenario, sharp curve scenario, intersection scenario, ramp scenario, and roundabout scenario.

[0019] During training, the Q function used is: Q(s,a;θ)=W2·f(W1·s+b1)+b2, where s represents the input state vector, i.e. the current state, a represents the action taken in the current state s, θ represents the parameters of the neural network, W1 is the weight matrix from the input layer to the hidden layer, W2 is the weight matrix from the hidden layer to the output layer, b1 is the bias vector of the hidden layer, b2 is the bias vector of the output layer, and f(·) is the non-linear activation function;

[0020] The loss function L(θ) is defined as the mean square error between the target Q-value y and the Q-value Q(s,a;θ) predicted by the current Q-network. The formula for the loss function L(θ) is:

[0021] L(θ) = E[(yQ(s,a;θ)) 2 ],

[0022] In the formula, E represents the expected value, which is usually the mean of a mini-batch of samples, used to stabilize gradient updates;

[0023] The target Q-value y is calculated based on the current reward and the maximum Q-value of the next state, and its calculation method is as follows:

[0024] y=R+γ·a′maxQ(s′,a′;θ - ),

[0025] In the formula, R is the immediate reward obtained by the agent based on the neural network model after taking action a in state s; γ is the discount factor, ranging from [0,1], representing the discount rate of future rewards; s′ represents the next state reached by the agent after taking action a, max a′ Q(s′,a′;θ - θ represents the action with the largest Q value among all possible actions a′ in the next state s′. - These represent the parameters of the target Q-network;

[0026] Substituting the target Q value y into the loss function L(θ), we obtain the specific expression for the loss function:

[0027]

[0028] By minimizing this loss function, the parameters θ in the DON network are gradually adjusted to enable the network to more accurately predict the Q-value of each state-action pair, thereby improving the decision-making ability of the neural network model.

[0029] Preferably, in step S1, the path generated by global path planning is the optimal path from the starting point to the target point. This optimal path serves as a reference for the reinforcement learning module in the subsequent neural network model, guiding the optimization process of local decisions.

[0030] Preferably, in step S2, the neural network model is built based on the DQN algorithm; the neural network model includes an input layer, an output layer, and a hidden layer.

[0031] Preferably, in step S3, when setting the reward function, the reward and penalty indicators include path execution, collision avoidance, distance maintenance, traffic rule compliance, and driving stability, to comprehensively evaluate the driving behavior of the model, and continuously optimize the model's strategy through reinforcement learning algorithms.

[0032] Preferably, in step S4, the perceived information includes: the deviation D between the autonomous vehicle and the planned path. m The closest longitudinal distance D between the autonomous vehicle and the vehicle in front. y The lateral distance D between the autonomous vehicle and the nearest vehicle in the left and right directions in other lanes of the current road segment. x The angular deviation D between the current heading angle of the autonomous vehicle and the line connecting two waypoints on the navigation path ahead. a The current speed v of the automatic car, the current steering wheel angle a of the automatic car, and the current longitudinal acceleration a of the automatic car. y The current lateral acceleration magnitude a of the autonomous vehicle x The longitudinal relative speed v between the vehicle ahead that is approaching the autonomous vehicle the fastest in the current road segment and the autonomous vehicle. t .

[0033] Preferably, the deviation D between the autonomous vehicle and the planned path m The calculation formula is: In the formula, d represents the distance between the current position of the autonomous vehicle and the matching point on the planned path, and D represents the maximum width of the autonomous vehicle;

[0034] The closest longitudinal distance D between the autonomous vehicle and the vehicle in front y R, the ratio of dynamic safety distance d The calculation formula is:

[0035] In the formula, v represents the current vehicle speed, β represents the dynamic safety distance coefficient at the current vehicle speed, and d base This indicates the basic safety distance.

[0036] Preferably, the global path planning and local path optimization processes are performed in real time during vehicle operation to dynamically adjust the driving strategy.

[0037] The beneficial effects of this invention are as follows:

[0038] (1) The present invention is based on the autonomous driving decision planning method of deep reinforcement learning and A-Star algorithm. It constructs a hierarchical system. The top layer uses the A-Star algorithm for global path planning, and the bottom layer uses reinforcement learning to handle local decisions, such as obstacle avoidance and speed control. By combining the path planning capability of the A-Star algorithm with the adaptive decision-making capability of reinforcement learning, it can effectively plan the optimal path in complex and ever-changing road environments, and dynamically adjust the driving strategy to ensure the safe and efficient driving of the vehicle in various scenarios.

[0039] (2) In the method of the present invention, by introducing a dynamic safe distance calculation mechanism based on vehicle speed, the system ensures that the vehicle can maintain an appropriate safe distance at different speeds. Combined with the reward and punishment mechanism of reinforcement learning, the system can give gradually increasing punishment when the distance is insufficient, thereby effectively avoiding rear-end collisions and other collision accidents and improving driving safety.

[0040] (3) In the method of this invention, deep reinforcement learning combined with the A-Star algorithm not only improves the decision-making and planning efficiency of the autonomous driving system, but also reduces the computational overhead of traditional rule-based decision-making systems in complex environments. The agent based on the neural network model can be trained extensively in a simulation environment to learn the best driving strategy, thereby achieving more efficient and intelligent autonomous driving.

[0041] (4) The method of the present invention can provide driving decisions for autonomous vehicles efficiently and accurately, and has good generalization ability and adaptability to adapt to complex and ever-changing traffic environments. Attached Figure Description

[0042] Figure 1 : A schematic flowchart of the method of the present invention;

[0043] Figure 2 : A schematic diagram of the perceptual information received by the intelligent neural network model in this invention;

[0044] Figure 3 : A schematic diagram illustrating the relationship between the neural network model and the state and decision commands in this invention. Detailed Implementation

[0045] The present invention will be further described below with reference to the embodiments. It should be noted that these are merely examples and descriptions of the inventive concept. Those skilled in the art can make various modifications or additions to the specific embodiments described or use similar methods to replace them, as long as they do not deviate from the inventive concept or exceed the scope defined in the claims, they should all be considered to fall within the protection scope of the present invention.

[0046] Example 1:

[0047] Figure 1This is a schematic diagram of the process of the method of the present invention, such as... Figure 1 As shown, the autonomous driving decision-making and planning method based on deep reinforcement learning and the A-Star algorithm includes the following steps:

[0048] S1. Use the A-Star algorithm to perform global path planning on the known map.

[0049] The path generated by global path planning on a known road map using the A-Star algorithm is the optimal path from the starting point to the destination. This optimal path serves as a reference for the reinforcement learning module in the subsequent neural network model, guiding the optimization process of local decisions. The A-Star algorithm is a heuristic search-based algorithm that finds the path with the minimum cost in graph search problems by combining the actual cost and the estimated cost of the path.

[0050] S2. Build a neural network model based on deep reinforcement learning.

[0051] The constructed neural network model, based on the DQN algorithm, includes a three-layer structure: an input layer, an output layer, and a hidden layer. It is responsible for processing local environmental perception information and making real-time decisions.

[0052] S3. Set the reward function for the neural network model.

[0053] When setting the reward function, the reward and penalty indicators include path execution, collision avoidance, distance maintenance, traffic rule compliance, and driving stability, so as to comprehensively evaluate the driving behavior of the model and continuously optimize the model's strategy through reinforcement learning algorithms.

[0054] Events for rewards and punishments include:

[0055] Deviation from the global path results in a penalty for the autonomous vehicle straying from the road centerline. The further the vehicle deviates from the center, the heavier the penalty. The reward R is: R = -D m D m The deviation of the autonomous vehicle from the planned path;

[0056] When approaching a vehicle ahead, if the distance between the automated vehicle and the vehicle ahead in the same direction is less than the dynamic safe distance, an unsafe situation is entered. The smaller the ratio of the distance between the automated vehicle and the vehicle ahead in the same direction to the dynamic safe distance, the heavier the penalty. The reward is: D y The distance between the autonomous vehicle and the vehicle in front is the closest longitudinal distance, where v represents the current vehicle speed, β represents the dynamic safety distance coefficient at the current vehicle speed, and d represents the distance between the vehicle and the vehicle in front. base Indicates the basic safety distance;

[0057] If an autonomous vehicle approaches vehicles on either side of an intersection and enters an unsafe situation, it will be given a small penalty as a warning and a reward of -3.

[0058] Excessive longitudinal acceleration, especially during rapid acceleration and braking, can cause severe vehicle pitching, affecting both comfort and safety. When the longitudinal acceleration exceeds 5 m / s², it can cause significant pitching. 2 As the number of values ​​increases, the penalty becomes more severe, and the reward is: R = 5 - |a y |,a y This represents the current longitudinal acceleration of the autonomous vehicle.

[0059] Excessive lateral acceleration can lead to loss of vehicle control or passenger discomfort. When the magnitude of lateral acceleration exceeds 3 m / s², it can cause significant damage. 2 As the number of values ​​increases, the penalty becomes more severe, and the reward is: R = 3 - |a x |,a x This represents the current lateral acceleration of the autonomous vehicle.

[0060] For speeding, if the speed of an autonomous vehicle exceeds the road speed limit by 80%, a penalty will be imposed based on the difference in speed.

[0061] If the speed is too low, such as when the speed of the autonomous vehicle is lower than the walking speed (1.5m / s), a slight penalty is imposed, and a reward of -1 is given for reducing road congestion.

[0062] S4. The neural network model receives the perceptual information acquired and transmitted by the perceptual system.

[0063] Figure 2 A schematic diagram of the perceptual information received by the intelligent neural network model, such as... Figure 2 As shown, the perceived information includes: the deviation D between the autonomous vehicle and the planned path. m The closest longitudinal distance D between the autonomous vehicle and the vehicle in front. y The lateral distance D between the autonomous vehicle and the nearest vehicle in the left and right directions in other lanes of the current road segment. x The angular deviation D between the current heading angle of the autonomous vehicle and the line connecting two waypoints on the navigation path ahead. a The current speed v of the automatic car, the current steering wheel angle a of the automatic car, and the current longitudinal acceleration a of the automatic car. y The current lateral acceleration magnitude a of the autonomous vehicle x The longitudinal relative speed v between the vehicle ahead that is approaching the autonomous vehicle the fastest in the current road segment and the autonomous vehicle. t .

[0064] The deviation D of the autonomous vehicle from the planned path m The calculation formula is: In the formula, d represents the distance between the current position of the autonomous vehicle and the matching point on the planned path, and D represents the maximum width of the autonomous vehicle.

[0065] The closest longitudinal distance D between the autonomous vehicle and the vehicle in front y R, the ratio of dynamic safety distance d The calculation formula is:

[0066] In the formula, v represents the current vehicle speed, β represents the dynamic safety distance coefficient at the current vehicle speed, and d base This indicates the basic safety distance.

[0067] All perceived information is merged into a 13-dimensional vector as the input state, including: path deviation distance, speed, longitudinal acceleration, lateral acceleration, steering angle, relative speed of the nearest vehicle, lateral distance, and heading angle deviation (dimension 1); relative position of the nearest vehicle (dimension 2); and distance to the vehicle in front (dimension 3).

[0068] S5. Using the global path information from step S1, the reward function from step S3, and the perception information from step S4, train the neural network model from step S2.

[0069] This includes training the neural network model, including training for independent scenarios and training for mixed scenarios.

[0070] The independent scenario is one of the following: straight road scenario, sharp curve scenario, intersection scenario, ramp scenario, and roundabout scenario; the mixed scenario is a combination of two or more of the following scenarios: straight road scenario, sharp curve scenario, intersection scenario, ramp scenario, and roundabout scenario.

[0071] During training, the Q function used is: Q(s,a;θ)=W2·f(W1·s+b1)+b2, where s represents the input state vector, i.e. the current state, a represents the action taken in the current state s, θ represents the parameters of the neural network, W1 is the weight matrix from the input layer to the hidden layer, W2 is the weight matrix from the hidden layer to the output layer, b1 is the bias vector of the hidden layer, b2 is the bias vector of the output layer, and f(·) is a non-linear activation function, such as ReLU (Rectified Linear Unit): f(x)=max(0,x).

[0072] The loss function L(θ) is defined as the mean square error between the target Q-value y and the Q-value Q(s,a;θ) predicted by the current Q-network. The formula for the loss function L(θ) is:

[0073] L(θ) = E[(yQ(s,a;θ)) 2 ],

[0074] In the formula, E represents the expected value, which is usually the mean of a mini-batch of samples and is used to stabilize gradient updates.

[0075] The target Q-value y is calculated based on the current reward and the maximum Q-value of the next state, and its calculation method is as follows:

[0076] y=R+γ·a′maxQ(s′,a′;θ - ),

[0077] In the formula, R is the immediate reward obtained by the agent based on the neural network model after taking action a in state s; γ is the discount factor, ranging from [0,1], representing the discount rate of future rewards; s′ represents the next state reached by the agent after taking action a, max a′ Q(s′,a′;θ - θ represents the action with the largest Q value among all possible actions a′ in the next state s′. - This represents the parameters of the target Q-network.

[0078] Substituting the target Q value y into the loss function L(θ), we obtain the specific expression for the loss function:

[0079]

[0080] By minimizing this loss function, the parameters θ in the DON network are gradually adjusted to enable the network to more accurately predict the Q-value of each state-action pair, thereby improving the decision-making ability of the neural network model.

[0081] S6. The reinforcement learning module in the neural network model, based on perceptual information and the global path planning provided by the A-Star algorithm, optimizes the local path using the trained neural network model, and then outputs a decision for execution by the controller. Both global path planning and local path optimization are performed in real time during vehicle operation to dynamically adjust the driving strategy.

[0082] (1) The present invention is based on the autonomous driving decision planning method of deep reinforcement learning and A-Star algorithm. It constructs a hierarchical system. The top layer uses the A-Star algorithm for global path planning, and the bottom layer uses reinforcement learning to handle local decisions, such as obstacle avoidance and speed control. By combining the path planning capability of the A-Star algorithm with the adaptive decision-making capability of reinforcement learning, it can effectively plan the optimal path in complex and ever-changing road environments, and dynamically adjust the driving strategy to ensure the safe and efficient driving of the vehicle in various scenarios.

[0083] (2) In the method of the present invention, by introducing a dynamic safe distance calculation mechanism based on vehicle speed, the system ensures that the vehicle can maintain an appropriate safe distance at different speeds. Combined with the reward and punishment mechanism of reinforcement learning, the system can give gradually increasing punishment when the distance is insufficient, thereby effectively avoiding rear-end collisions and other collision accidents and improving driving safety.

[0084] (3) In the method of this invention, deep reinforcement learning combined with the A-Star algorithm not only improves the decision-making and planning efficiency of the autonomous driving system, but also reduces the computational overhead of traditional rule-based decision-making systems in complex environments. The agent based on the neural network model can be trained extensively in a simulation environment to learn the best driving strategy, thereby achieving more efficient and intelligent autonomous driving.

[0085] (4) The method of the present invention can provide driving decisions for autonomous vehicles efficiently and accurately, and has good generalization ability and adaptability to adapt to complex and ever-changing traffic environments.

[0086] The above is an exemplary description of the invention. Obviously, the specific implementation of the invention is not limited to the above-described manner. Any non-substantial improvement made using the inventive concept and technical solution of the invention, or the direct application of the inventive concept and technical solution to other situations without modification, is within the protection scope of the invention.

Claims

1. An autonomous driving decision-making and planning method based on deep reinforcement learning and the A-Star algorithm, characterized in that, Includes the following steps: S1. Use the A-Star algorithm to perform global path planning on a known map; S2. Construct a neural network model based on deep reinforcement learning, wherein the neural network model is responsible for processing local environmental perception information and making real-time decisions; S3. Set the reward function for the neural network model; S4. The neural network model receives the perceptual information acquired and transmitted by the perceptual system; S5. Using the global path information from step S1, the reward function from step S3, and the perception information from step S4, train the neural network model from step S2. S6. The reinforcement learning module in the neural network model is based on perceptual information and global path planning provided by the A-Star algorithm. It uses the trained neural network model to optimize the local path and then outputs the decision. In step S5, the neural network model from step S2 is trained, including training for independent scenes and training for mixed scenes. The independent scenario is one of the following scenarios: straight road scenario, sharp curve scenario, intersection scenario, ramp scenario, and roundabout scenario; the mixed scenario is a combination of two or more of the following scenarios: straight road scenario, sharp curve scenario, intersection scenario, ramp scenario, and roundabout scenario. The Q-function used during training is: Where s represents the input state vector, i.e., the current state. This represents the action taken in the current state s. The parameters of the neural network, This is the weight matrix from the input layer to the hidden layer. This is the weight matrix from the hidden layer to the output layer. The bias vector of the hidden layer. This is the bias vector for the output layer. It is a non-linear activation function; loss function Defined as target Q value And the Q-value predicted by the current Q-network Mean squared error between, loss function The formula is: ， In the formula, This represents the expected value, typically the mean of a mini-batch of samples, used to stabilize gradient updates. Target Q value It is calculated based on the current reward and the maximum Q value of the next state, and the calculation method is as follows: ， In the formula, For agents based on neural network models in state The immediate reward obtained after taking action a; This is a discount factor, with a value between [0,1], representing the discount rate of future rewards; This represents the next state reached by the agent after taking action 'a'. Indicates the next state The action with the maximum Q value among all possible actions. , These represent the parameters of the target Q-network; Target Q value Substitute into the loss function In this process, we obtain the specific expression for the loss function: ， By minimizing this loss function, the parameters in the DQN network... The network will be gradually adjusted to make it more accurate in predicting the Q-value of each state-action pair, thereby improving the decision-making ability of the neural network model.

2. The autonomous driving decision-making and planning method based on deep reinforcement learning and A-Star algorithm according to claim 1, characterized in that, In step S1, the path generated by global path planning is the optimal path from the starting point to the target point. This optimal path serves as a reference for the reinforcement learning module in the subsequent neural network model, guiding the optimization process of local decisions.

3. The autonomous driving decision-making and planning method based on deep reinforcement learning and A-Star algorithm according to claim 1, characterized in that, In step S3, when setting the reward function, the reward and penalty indicators include path execution, collision avoidance, distance maintenance, traffic rule compliance, and driving stability, comprehensively evaluating the driving behavior of the model, and continuously optimizing the model's strategy through reinforcement learning algorithms.

4. The autonomous driving decision-making and planning method based on deep reinforcement learning and A-Star algorithm according to claim 1, characterized in that, In step S4, the perceived information includes: the deviation of the autonomous vehicle from the planned path. The closest longitudinal distance between an autonomous vehicle and the vehicle in front. The lateral distance between the autonomous vehicle and the nearest vehicle in other lanes of the current road segment in the left and right directions. The angular deviation between the current heading angle of the autonomous vehicle and the line connecting two waypoints on the navigation path ahead. The current speed of the automatic car The current steering wheel angle of the automatic car The current longitudinal acceleration of the autonomous vehicle The current lateral acceleration magnitude of the autonomous vehicle The longitudinal relative speed between the vehicle ahead that is approaching the autonomous vehicle the fastest in the current road segment and the autonomous vehicle. .

5. The autonomous driving decision-making and planning method based on deep reinforcement learning and A-Star algorithm according to claim 4, characterized in that, The deviation of the autonomous vehicle from the planned path The calculation formula is: In the formula, d represents the distance between the current position of the autonomous vehicle and the matching point on the planned path, and D represents the maximum width of the autonomous vehicle; The closest longitudinal distance between the autonomous vehicle and the vehicle in front The ratio of dynamic safety distance The calculation formula is: , In the formula, v represents the current vehicle speed. This represents the dynamic safety distance coefficient at the current vehicle speed. This indicates the basic safety distance.

6. The autonomous driving decision-making and planning method based on deep reinforcement learning and A-Star algorithm according to claim 1, characterized in that, The global path planning and local path optimization processes are performed in real time during vehicle operation to dynamically adjust the driving strategy.

Images