Hierarchical autonomous vehicle decision control method fusing traffic rules

Abstract

The application discloses a kind of hierarchical automatic driving vehicle decision control methods of fusing traffic rules, comprising the following steps: S1: definition observation space, traffic signal light information, path point information, vehicle state information, obstacle information are as observation space element;S2: according to observation space element extraction state space, construct hierarchical automatic driving vehicle decision control model of fusing traffic rules, including upper layer brake submodel and lower layer drive submodel;S3: design reward function of upper layer brake submodel and lower layer drive submodel, according to upper layer brake submodel and lower layer drive submodel output definition action space;S4: using double-delay deep deterministic policy gradient algorithm, upper layer brake submodel and lower layer drive submodel are iteratively trained, until double-layer submodel converges;S5: vehicle decision control model is placed in vehicle controller, and vehicle is automatically driven using vehicle controller, and vehicle automatic driving task is completed.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and more specifically, to a hierarchical autonomous driving vehicle decision-making and control method that integrates traffic rules for navigation of autonomous vehicles in urban environments. Background Technology

[0002] Currently, the development of safe and reliable autonomous driving technology remains a key research area, and research on autonomous driving systems that better reflect real-world scenarios is receiving increasing attention. The behavior decision-making and path planning layer, as a crucial component of modular research in autonomous vehicles, plays a vital role in safety and reliability. However, in practical applications, the decision-making layer may experience problems such as poor stability and low accuracy. Once these issues arise, the safety and reliability of autonomous vehicles will be severely threatened, becoming a major challenge for real-world applications.

[0003] Deep reinforcement learning methods are increasingly being applied to the decision-making layers of autonomous vehicles to better control their behavior in various scenarios. However, current methods are only applicable to specific driving tasks, such as lane-changing scenarios, and suffer from excessively long training times. Currently, deep reinforcement learning methods are not being applied to navigation tasks in urban scenarios where autonomous vehicles adhere to traffic rules, and there is a need to improve training time for practical applications. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings and deficiencies of the prior art and provide a hierarchical decision-making and control method for autonomous vehicles that integrates traffic rules. This method applies deep reinforcement learning to the decision-making layer of autonomous vehicles, enabling decision-making and control applications of autonomous vehicles in urban scenarios. By adopting the idea of ​​hierarchical decision-making, the training time of the vehicle decision-making and control model is increased, thereby improving the safety, stability, and reliability of the autonomous driving system.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] A hierarchical autonomous vehicle decision-making and control method integrating traffic rules includes the following steps:

[0007] S1: Define the observation space, taking traffic light information, waypoint information, vehicle status information, and obstacle information as elements of the observation space;

[0008] S2: Extract the state space based on the observation space elements, and construct a hierarchical autonomous driving vehicle decision control model that integrates traffic rules. The vehicle decision control model includes an upper-level braking sub-model and a lower-level driving sub-model.

[0009] S3: Design the reward functions for the upper-level braking sub-model and the lower-level driving sub-model, and define the action space based on the output instructions of the upper-level braking sub-model and the lower-level driving sub-model;

[0010] S4: Use the dual-delay deep deterministic strategy gradient algorithm to iteratively train the upper-level braking sub-model and the lower-level driving sub-model until the two-layer sub-model converges, and obtain the trained vehicle decision control model.

[0011] S5: Configure the vehicle decision control model on the vehicle controller, and use the vehicle controller to perform autonomous driving control of the vehicle to complete the autonomous driving task.

[0012] Furthermore, traffic light information is obtained through high-precision maps, waypoint information is obtained through a global path planner, vehicle status information is obtained through vehicle body sensors, and obstacle information is obtained through a vehicle perception module.

[0013] The global path planner uses the AStar algorithm to plan a series of waypoints to navigate to the destination. The waypoint information is described as including location information and road curvature information. Vehicle status information includes vehicle position, vehicle speed, the angle between the vehicle centerline and the road centerline, and the distance between the vehicle centerline and the road centerline. Obstacle information uses semantic images and depth images to detect and locate obstacles and estimate the distance between obstacles and the vehicle.

[0014] Furthermore, obstacles are detected and located using semantic and depth images, and the distance between the obstacles and the vehicle is estimated, specifically as follows:

[0015] The semantic image is preprocessed to remove data from the oncoming lane;

[0016] The road line is fitted using a quadratic polynomial, and all pixel values ​​to the left of the curve are set to zero.

[0017] Extract distance maps from depth images and detect regions containing vehicles and pedestrians in processed semantic images;

[0018] Extract the binary mask of the obstacle region from the preprocessed semantic image, and calculate the average distance value of each pedestrian and vehicle in the obstacle region;

[0019] The minimum of the average distances between pedestrians and vehicles is used as the estimated distance to the nearest obstacle.

[0020] Furthermore, the state space includes: traffic light status, vehicle speed, obstacle distance, vehicle position, road curvature, distance between the vehicle center and the road centerline, and the angle between the vehicle centerline and the road centerline;

[0021] The upper-level braking sub-model outputs braking or driving commands based on the state space, and the braking or driving commands reflect the magnitude of braking force or driving force; the lower-level driving sub-model outputs steering commands based on the commands from the upper-level braking sub-model and the state space, and the steering commands reflect the steering wheel angle.

[0022] Furthermore, the upper-level braking sub-model takes the current values ​​of traffic light status, vehicle speed, and obstacle distance as input and outputs braking or driving actions; the lower-level driving sub-model takes the distance between the vehicle center and the road centerline, the angle between the vehicle centerline and the road centerline, the vehicle position, and the road curvature as input and outputs steering actions.

[0023] Furthermore, the design goal of the reward function of the upper-level braking sub-model is to comprehensively judge the output of the corresponding driving force or braking force based on the vehicle speed and the distance to the obstacle in front, so as to ensure the highest possible speed to reach the destination while ensuring safety.

[0024] The design goal of the reward function of the lower-level driving sub-model is to comprehensively judge and output the corresponding steering angle based on the angle between the vehicle centerline and the road centerline, the lateral distance between the vehicle center and the path point, the vehicle speed, and the road curvature, so as to drive along the global path point and the road centerline as much as possible while ensuring comfort.

[0025] Furthermore, the upper-level actuator model includes an input layer, two hidden layers, and an output layer. The input layer has three inputs, one hidden layer has eight hidden units, and the other hidden layer has four hidden units.

[0026] Furthermore, the lower-level driving sub-model includes an input layer, two hidden layers, and an output layer. The input layer has four inputs, one hidden layer has 10 hidden units, and the other hidden layer has 5 hidden units.

[0027] Furthermore, the vehicle decision control model outputs commands, and the vehicle controller applies braking force or driving force and steering wheel angle to the vehicle chassis through a PID algorithm.

[0028] Furthermore, when training with the dual-delay deep deterministic strategy gradient algorithm, the problem of sensitivity to hyperparameters and other types of adjustments is solved by truncated double Q learning. The smoothing idea is introduced to add noise to the target action, which is uncorrelated Gaussian noise with a mean of 0.

[0029] Compared with existing technologies, this invention incorporates traffic rules such as traffic lights, making it more suitable for practical application scenarios. By preprocessing the image information and vehicle status observed by vehicle sensors, key elements relevant to the navigation and driving task are extracted, significantly reducing the complexity of the state space and facilitating the subsequent training of the vehicle decision-making and control model, thus improving the stability of the model's decisions.

[0030] This invention divides the vehicle decision-making and control model into an upper-level braking sub-model and a lower-level driving sub-model. This hierarchical sub-model framework allows for parallel training, resulting in a shorter overall training cycle. By representing different elements of the state space as the focus of each of the two sub-models, the vehicle decision-making and control model training converges more easily. The upper-level braking sub-model determines the presence of obstacles ahead based on sensor information, enabling collision-free driving and ensuring the safety and reliability of applying deep reinforcement learning in engineering. Attached Figure Description

[0031] Figure 1 A flowchart illustrating a hierarchical decision-making and control method for autonomous vehicles that integrates traffic rules.

[0032] Figure 2 This is a schematic diagram of the vehicle decision control model framework for an example.

[0033] Figure 3 This is a schematic diagram of vehicle status information for an example.

[0034] Figure 4 This is a schematic diagram of the obstacle information processing process in an embodiment.

[0035] Figure 5 This is a schematic diagram of the upper-level brake element model for an embodiment.

[0036] Figure 6 This is a schematic diagram of the lower-level driver sub-model in an embodiment. Detailed Implementation

[0037] The hierarchical autonomous driving vehicle decision-making and control method integrating traffic rules of the present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0038] Please see Figure 1 and Figure 2 This invention discloses a hierarchical autonomous driving vehicle decision-making and control method that integrates traffic rules, comprising the following steps:

[0039] S1: Define the observation space, using traffic light information, waypoint information, vehicle status information, and obstacle information as elements of the observation space.

[0040] S2: Extract the state space based on the observation space elements, and construct a hierarchical autonomous driving vehicle decision control model that integrates traffic rules. The vehicle decision control model includes an upper-level braking sub-model and a lower-level driving sub-model.

[0041] S3: Design the reward functions for the upper-level braking sub-model and the lower-level driving sub-model, and define the action space based on the output instructions of the upper-level braking sub-model and the lower-level driving sub-model.

[0042] S4: Employ a dual-delay deep deterministic strategy gradient algorithm to iteratively train the upper-level braking sub-model and the lower-level driving sub-model until the two-layer sub-model converges, thus obtaining a trained vehicle decision control model.

[0043] S5: Configure the vehicle decision control model on the vehicle controller, and use the vehicle controller to perform autonomous driving control of the vehicle to complete the autonomous driving task.

[0044] Please see Figure 3 Traffic light information is obtained through high-precision maps, waypoint information through a global path planner, vehicle status information through vehicle sensors, and obstacle distance information through a vehicle perception module. The global path planner uses the AStar algorithm to plan a series of waypoints to the destination, describing the waypoint information as including location information and road curvature information. Vehicle status information includes vehicle position, vehicle speed v, the angle φ between the vehicle's centerline and the road centerline (i.e., the vehicle yaw angle), and the distance d between the vehicle's center and the road centerline. Obstacle information utilizes semantic and depth images to detect and locate obstacles, such as vehicles and pedestrians, to estimate the distance between obstacles and vehicles.

[0045] Please see Figure 4 It utilizes semantic and depth images to detect and locate obstacles, and estimates the distance between obstacles and vehicles, specifically:

[0046] First, the semantic image is preprocessed to remove data from oncoming lanes, as oncoming vehicles should not trigger the vehicle decision control model to implement braking strategies. A quadratic polynomial is used to fit the road line, and all pixel values ​​to the left of the curve are set to zero. Then, a distance map is extracted from the depth image, and regions containing vehicles and pedestrians are detected in the processed semantic image. Finally, binary masks of obstacle regions are extracted from the preprocessed semantic image, and the average distance value for each pedestrian and vehicle in the obstacle region is calculated. The minimum of the average distance values ​​for pedestrians and vehicles is then used as the estimated distance to the nearest obstacle. (See figure.) This represents the average distance traveled by pedestrians. d represents the average distance traveled by the vehicle. obs It is the minimum of the average distance for pedestrians and the average distance for vehicles.

[0047] The state space is extracted based on the observed spatial elements. The state space includes traffic light status, distance between the vehicle center and the road centerline, angle between the vehicle centerline and the road centerline, vehicle position, vehicle speed, obstacle distance, and road curvature. The vehicle decision control model includes an upper-level braking sub-model and a lower-level driving sub-model. The upper-level braking sub-model outputs driving or braking commands based on the state space, while the lower-level driving sub-model outputs steering commands based on the commands from the upper-level braking sub-model and the state space.

[0048] This paper employs twin-delayed deep deterministic policy gradient (TD3) to learn policies for vehicles to complete driving tasks in traffic flow, enabling unsupervised offline training of the vehicle decision control model. TD3 allows the vehicle decision control model to output continuous action values, while using target networks and experience replay techniques to stabilize the training process. Clipped dobule Q-learning addresses the sensitivity to hyperparameters and other types of tuning.

[0049] TD3 learns two Q-functions simultaneously by minimizing the mean squared error. Both Q-functions use a target, and the smaller value is used as the Q-target. Simultaneously training the action network and evaluation network without using the target network leads to instability during training; however, when only the action network is fixed, the evaluation network often converges to the correct result. Therefore, the action network is updated at a lower frequency, and the evaluation network is updated at a higher frequency, with the policy updated every two updates to the evaluation network.

[0050] A smoothing approach is introduced, adding noise to the target actions to smooth the changes in Q along the actions, making it harder for the policy to exploit the error in the Q function. Since the policy is deterministic, if the vehicle decision control model is to explore the policy, it may initially attempt to take a sufficiently wide range of actions to find useful learning signals. To enable the TD3 policy to explore better, noise, typically uncorrelated Gaussian noise with a mean of 0, is added to its actions during training. To facilitate obtaining high-quality training data, the noise level can be reduced during training. During testing, to examine how well the policy utilizes the learned knowledge, no noise is added to the actions.

[0051] Please see Figure 5 The upper-level actuator model includes an input layer, two hidden layers, and an output layer. The input layer has three inputs, one hidden layer has eight hidden units, the other hidden layer has four hidden units, and the output layer has one output.

[0052] The upper-level braking sub-model takes traffic light status, vehicle speed, and the current distance to the obstacle as input, and outputs braking or driving actions (i.e., the magnitude of braking force or driving force). The upper-level braking sub-model is trained on a straight road, with obstacle vehicles generated at random distances from the autonomous vehicle. The goal is to enable the autonomous vehicle to travel at the fastest possible speed without colliding with the vehicle in front. A reward function is designed to reward the autonomous vehicle for maintaining a high speed at the road-prescribed speed when far from obstacle vehicles, and for gradually decelerating to a safe distance when approaching the obstacle vehicle, while penalizing rounds in which collisions occur.

[0053] Please see Figure 6 The lower-level driving sub-model consists of an input layer, two hidden layers, and an output layer. The input layer has four inputs, one hidden layer has 10 hidden units, the other hidden layer has 5 hidden units, and the output layer has one output.

[0054] The lower-level driving sub-model takes the distance between the vehicle's center and the road centerline, the angle between the vehicle's centerline and the road centerline, the vehicle's position, and the road curvature as inputs, and outputs the steering action, i.e., the steering wheel angle. The lower-level driving sub-model is trained over 40 training cycles, alternating between left and right turns, enabling the autonomous vehicle to explore these two scenarios. The reward function assigned to the autonomous vehicle aims to ensure comfort and vehicle dynamics while keeping the vehicle on the road and following the trajectory planned by the global planner, while penalizing collisions.

[0055] The dual-delay deep deterministic policy gradient algorithm TD3 is employed to achieve unsupervised offline training of the vehicle decision control model. After the average reward value converges, a well-trained hierarchical autonomous driving vehicle decision control model incorporating traffic rules is finally formed and deployed on the vehicle to complete the autonomous driving navigation task from the starting point to the destination. The vehicle decision control model outputs commands, and the vehicle controller applies braking force or driving force and steering wheel angle to the vehicle chassis through a PID algorithm.

[0056] A hierarchical decision-making and control model for autonomous vehicles, incorporating traffic rules, is constructed. First, a series of waypoints for navigation to the destination are obtained based on a global planner. Then, a hierarchical approach is adopted, combining an upper-level braking sub-model and a lower-level driving sub-model obtained through training. The upper-level braking sub-model is used to ensure driving safety and prevent collisions when obstacles are too close. Before the vehicle decision-making and control model performs action prediction, the upper-level braking sub-model checks for the presence of traffic lights.

[0057] If the traffic light is red, the upper-level braking sub-model makes a stopping decision. If the traffic light is green, the upper-level braking sub-model uses state-space information to predict whether it is safe to drive or whether the vehicle should brake. If it is determined to be safe to drive, the lower-level driving sub-model uses state-space information to predict which steering action should be taken. This method allows autonomous vehicles to drive autonomously in traffic while ensuring safety, combining the upper-level braking sub-model and the lower-level driving sub-model to make intelligent decisions in various situations.

[0058] This invention incorporates traffic rules such as traffic lights, making it more suitable for practical application scenarios. It preprocesses the image information and vehicle status observed by vehicle sensors, extracting key elements relevant to the navigation and driving task. This significantly reduces the complexity of the state space, facilitating the training of the subsequent vehicle decision-making and control model and improving the stability of the model's decisions.

[0059] This invention divides the vehicle decision-making and control model into an upper-level braking sub-model and a lower-level driving sub-model. This hierarchical sub-model framework allows for parallel training, resulting in a shorter overall training cycle. By representing different elements of the state space as the focus of each of the two sub-models, the vehicle decision-making and control model training converges more easily. The upper-level braking sub-model determines the presence of obstacles ahead based on sensor information, enabling collision-free driving and ensuring the safety and reliability of applying deep reinforcement learning in engineering.

[0060] The above description is a detailed description of the preferred embodiments of the present invention. However, the embodiments are not intended to limit the scope of the patent application of the present invention. All equivalent changes or modifications made under the technical spirit disclosed in the present invention should fall within the patent scope covered by the present invention.

Claims

1. A hierarchical decision-making and control method for automated vehicles that integrates traffic rules, characterized in that, Includes the following steps: S1: Define the observation space, taking traffic light information, waypoint information, vehicle status information, and obstacle information as elements of the observation space; S2: Extract the state space based on the observation space elements, and construct a hierarchical autonomous driving vehicle decision control model that integrates traffic rules. The vehicle decision control model includes an upper-level braking sub-model and a lower-level driving sub-model. S3: Design the reward functions for the upper-level braking sub-model and the lower-level driving sub-model, and define the action space based on the output instructions of the upper-level braking sub-model and the lower-level driving sub-model; S4: Use the dual-delay deep deterministic strategy gradient algorithm to iteratively train the upper-level braking sub-model and the lower-level driving sub-model until the two-layer sub-model converges, and obtain the trained vehicle decision control model. S5: Configure the vehicle decision control model on the vehicle controller, and use the vehicle controller to perform autonomous driving control of the vehicle to complete the autonomous driving task. Traffic light information is obtained through high-precision maps, waypoint information is obtained through a global path planner, vehicle status information is obtained through vehicle body sensors, and obstacle information is obtained through a vehicle perception module. The global path planner uses the A*Star algorithm to plan a series of waypoints to navigate to the destination. The waypoint information is described as including location information and road curvature information; vehicle status information includes vehicle position, vehicle speed, the angle between the vehicle centerline and the road centerline, and the distance between the vehicle centerline and the road centerline. Obstacle information is used to detect and locate obstacles using semantic and depth images, and to estimate the distance between obstacles and vehicles; Obstacles are detected and located using semantic and depth images, and the distance between obstacles and vehicles is estimated, specifically as follows: The semantic image is preprocessed to remove data from the oncoming lane; The road line is fitted using a quadratic polynomial, and all pixel values ​​to the left of the curve are set to zero. Extract distance maps from depth images and detect regions containing vehicles and pedestrians in processed semantic images; Extract the binary mask of the obstacle region from the preprocessed semantic image, and calculate the average distance value of each pedestrian and vehicle in the obstacle region; The minimum of the average distances between pedestrians and vehicles is used as the estimated distance to the nearest obstacle. The state space includes: traffic light status, vehicle speed, obstacle distance, vehicle position, road curvature, distance between the vehicle center and the road centerline, and the angle between the vehicle centerline and the road centerline. The upper-level braking sub-model outputs braking or driving commands based on the state space, and the braking or driving commands reflect the magnitude of braking force or driving force; the lower-level driving sub-model outputs steering commands based on the commands from the upper-level braking sub-model and the state space, and the steering commands reflect the steering wheel angle. The upper-level braking sub-model takes the current values ​​of traffic light status, vehicle speed, and obstacle distance as input and outputs braking or driving actions; the lower-level driving sub-model takes the distance between the vehicle center and the road centerline, the angle between the vehicle centerline and the road centerline, the vehicle position, and the road curvature as input and outputs steering actions. The design goal of the reward function of the upper braking sub-model is to comprehensively judge the magnitude of the driving force or braking force based on the vehicle speed and the distance to the obstacle in front, and to ensure the highest possible speed to reach the destination while ensuring safety. The design goal of the reward function of the lower-level driving sub-model is to comprehensively judge and output the corresponding steering angle based on the angle between the vehicle centerline and the road centerline, the lateral distance between the vehicle center and the path point, the vehicle speed, and the road curvature, so as to drive along the global path point and the road centerline as much as possible while ensuring comfort.

2. The hierarchical autonomous driving vehicle decision-making and control method integrating traffic rules according to claim 1, characterized in that, The upper-level actuator model consists of an input layer, two hidden layers, and an output layer. The input layer has three inputs, one hidden layer has eight hidden units, and the other hidden layer has four hidden units.

3. The hierarchical autonomous driving vehicle decision-making and control method integrating traffic rules according to claim 1, characterized in that, The lower-level driving sub-model consists of an input layer, two hidden layers, and an output layer. The input layer has four inputs, one hidden layer has 10 hidden units, and the other hidden layer has 5 hidden units.

4. The hierarchical autonomous driving vehicle decision-making and control method integrating traffic rules according to claim 1, characterized in that, The vehicle decision control model outputs commands, and the vehicle controller applies braking force or driving force and steering wheel angle to the vehicle chassis through PID algorithm.

5. The hierarchical autonomous driving vehicle decision-making and control method integrating traffic rules according to claim 1, characterized in that, When training with the dual-delay deep deterministic gradient algorithm, the problem of sensitivity to hyperparameters and other types of adjustments is solved by truncated double Q learning. The smoothing idea is introduced to add noise to the target action. The noise is uncorrelated Gaussian noise with a mean of 0.

Images