Crossroad unprotected left turn decision control method based on deep reinforcement learning
By using a deep reinforcement learning-based approach combined with Sim-One stochastic traffic flow simulation and obstacle filtering, the left-turn decision-making of autonomous vehicles at unprotected intersections is optimized, solving the decision-making and control problem in complex traffic environments and improving intelligence and training efficiency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHAANXI HEAVY DUTY AUTOMOBILE CO LTD
- Filing Date
- 2024-10-17
- Publication Date
- 2026-04-17
AI Technical Summary
Existing technologies are insufficient to effectively solve the left-turn decision-making problem for autonomous vehicles at unprotected intersections, especially in complex traffic environments. It is difficult to cover all scenarios with rule-based decision control algorithms, and the cost of acquiring training data is high, while the risk of model validation is significant.
A deep reinforcement learning-based approach is adopted, which optimizes the decision control of autonomous vehicles by building a Sim-One random traffic flow simulation scenario, a data preprocessing module, an environmental control module, an operation module, an autonomous driving agent module, and an experience pool module, combined with obstacle filtering, intent judgment, and reward functions.
It improves the decision-making and control intelligence of autonomous vehicles at intersections on open roads, adapts to traffic flows with different characteristics, solves the decision-making and control problem in complex traffic environments at unprotected intersections, shortens training time, and reduces the state space dimension.
Smart Images
Figure CN121871618A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of autonomous driving motion behavior prediction technology, and in particular to a method for unprotected left turn decision control at intersections based on deep reinforcement learning. Background Technology
[0002] Unprotected left turns at intersections are among the most challenging scenarios in urban environments. An unprotected left turn at an intersection occurs when an autonomous vehicle is at a traffic light where a circular green light (without a green arrow, and not distinguishing between left turns and straight-ahead traffic) indicates that either left turn or straight-ahead traffic is permitted, or there is no designated left-turn waiting area. If an autonomous vehicle wants to turn left, it will face a series of problems, such as whether to start turning left as soon as the light turns green, wait for the oncoming traffic to move, or turn left immediately once there is space to enter; whether there are oncoming vehicles in the opposite lane, and whether the distance between oncoming vehicles and the autonomous vehicle is sufficient for the vehicle to complete the turn; whether pedestrians and non-motorized vehicles have entered the lane to which the vehicle is turning, etc. A 2010 study by the U.S. Department of Transportation, after investigating 2 million accidents, found that human drivers' misjudgments of left turns accounted for 22.2% of all accidents, while right turns accounted for only 1.2%. For any autonomous vehicle, turning left requires observing a variety of information to determine the appropriate turning time. Rule-based decision control algorithms have difficulty fully covering all scenarios, especially in the absence of left-turn protection. Even the most skilled autonomous vehicles today have difficulty making smooth left turns.
[0003] Existing technologies, such as reinforcement learning algorithms, employ a self-supervised learning approach. This allows the decision-making system of autonomous vehicles to autonomously learn the optimal decision model by continuously improving its own strategy during interactions with the environment, providing direction for building effective decision-making systems. Based on the input current environment state and the current behavioral state of the autonomous vehicle, actions are selected and output from an experience pool. If no action corresponding to the current environment state is found in the experience pool, a deep reinforcement learning structure calculates and outputs the action. However, acquiring training data is extremely costly, and the risk of real-world validation of the trained model is high. The challenges of this algorithm lie in verifying the correctness of the policy, verifying whether the Q-function eventually converges to a near-realistic estimate, and adjusting the value function based on the value curve. Summary of the Invention
[0004] To address the shortcomings of existing technologies, this invention proposes a decision control method for unprotected left turns at intersections based on deep reinforcement learning.
[0005] To address the problems mentioned above in the background section, the present invention is achieved through the following technical solution:
[0006] A deep reinforcement learning-based method for unprotected left-turn decision control at intersections includes: Sim-One random traffic flow simulation scenario construction, data preprocessing module, environmental control module, operation module, autonomous driving agent module, and experience pool module.
[0007] Sim-One Random Traffic Flow Simulation Scenario Setup: The simulation environment is set up using the Sim-One Random Traffic Flow Simulation Scenario Setup module.
[0008] Data preprocessing module: used to filter obstacles based on their location and the intentions of other traffic participants to reduce the dimensionality of the state space;
[0009] Environment control module: Used to control the Sim-One simulation module and interact with the running module;
[0010] The operation module is used for information interaction between the data preprocessing module, the environmental control module, the operation module, the autonomous driving intelligent agent module, and the experience pool module.
[0011] Autonomous Driving Intelligent Agent Module: The status information sent by the data processing module is transmitted to the autonomous driving intelligent agent module through the environmental control module and the operation module. The autonomous driving intelligent agent module makes decision-making actions based on the status information.
[0012] Experience pool module: Used to store experience transferred from the running module. When the stored experience reaches the set maximum value, the oldest experience is deleted and the latest experience is stored. It is also used to perform batch sampling from the stored experience and pass the sampled experience to the autonomous driving agent module for updating the autonomous vehicle policy network.
[0013] Furthermore, the simulation environment setup includes road network configuration, vehicle configuration, and traffic flow configuration;
[0014] The road network is configured to load different intersection maps to provide a rich training dataset, and the origin of the simulation environment coordinate system is the center point of the intersection.
[0015] The vehicle is configured such that the simulated vehicle is equipped with sensors to acquire real-time simulation environment status information and vehicle status information. The simulation environment status information includes obstacle information, obstacle position, obstacle speed, and obstacle type; the vehicle status information includes vehicle position information and vehicle speed information.
[0016] The traffic flow configuration is to set the simulation test road as an urban road simulated using traffic flow parameters.
[0017] Furthermore, in order to reduce the number of obstacles and lower the dimensionality of the state space, the data processing module first preprocesses the obstacles;
[0018] S1, the preprocessing is as follows:
[0019] First, extract the environmental state representation data: S = {s0, s1, s2, s3, ..., s} final} where s0 is the initial state, i.e., the moment when the autonomous vehicle is aligned with the stop line, which is the reinforcement learning process; s1 is the state in the state space at time Δt, s2 is the state in the state space at time 2*Δt, s3 is the state in the state space at time 3*Δt, and so on, s final The state space is when the rear of the vehicle is aligned with the stop line; the state space ends when the rear of the vehicle is aligned with the stop line.
[0020] S2 divides obstacles into zones based on the vehicle's location to reduce the dimensionality of the state space:
[0021] First, the area of the unprotected left turn scenario is divided into a dangerous area and a safe area. The dangerous area includes dangerous area 1 and dangerous area 2.
[0022] Then, the trajectory of the upper left corner of the vehicle is calculated: When the vehicle turns left, the left front corner, i.e., the upper left corner of the vehicle, will be the first to collide. Therefore, it is necessary to calculate the position coordinates of the left front corner of the vehicle. The Sim-One simulation module outputs the position coordinates and heading of the rear axle midpoint of the vehicle in real time. The position from the front center point of the vehicle to the rear axle center is L, and the vehicle width is W. Then, the coordinates of the left front position point of the autonomous vehicle can be expressed as:
[0023]
[0024] Where x0 and y0 are the position coordinates of the midpoint of the rear axle of the vehicle, and yaw is the heading of the midpoint of the rear axle of the vehicle;
[0025] During the left turn, the distance D1 between the coordinates (x, y) of the vehicle's left front position and the right boundary of danger zone 1, and the distance D2 between the coordinates (x, y) of the vehicle's left front position and the right boundary of danger zone 2 are calculated. When the vehicle begins to enter danger zone 1, D1 = 0 and D2 > 0. At this time, all obstacles in danger zone 1 may collide with the vehicle. Therefore, the flag of obstacles in danger zone 1 is set to danger_area_flag = true; the flag of obstacles in danger zone 2 is set to danger_area_flag = false.
[0026] When the vehicle enters danger zone 2 and D2=0, for the vehicle, there is a possibility of collision with obstacles in danger zone 1 and danger zone 2, as well as straight-going and right-turning vehicles in danger zone 1, and pedestrians and non-motorized vehicles on the sidewalk in danger zone 2. Therefore, the flags of obstacles in danger zone 1 and danger zone 2 should be set to danger_area_flag=true.
[0027] S3, Intent Judgment
[0028] This includes vehicle intent judgment and pedestrian and non-motorized vehicle intent prediction. Vehicle intent judgment is as follows: when an autonomous vehicle turns left, vehicles going straight and turning right in the opposite lane will interfere with the autonomous vehicle entering the same lane, while vehicles turning left in the opposite straight lane will not affect the autonomous vehicle's left turn. Therefore, vehicles in the opposite lane with the intent to turn right or go straight are marked as dangerous vehicles, and vehicles turning left are marked as safe vehicles.
[0029] The obstacle types are divided into motor vehicles, non-motor vehicles and pedestrians. The Sim-One simulation module sends the obstacle category truth value through sensors and outputs the obstacle's position information. By using the current position information and the position information of 9 historical moments, the speed, heading and position of the obstacle are determined by the trajectory points, thereby judging the collision risk between the obstacle and the autonomous vehicle. This filters the obstacles, reduces the state space dimension of the input of the deep reinforcement learning network, and shortens the training time.
[0030] Pedestrian and non-motorized vehicle intention prediction: When the vehicle is about to complete a left turn and enter the crosswalk, pedestrians and non-motorized vehicles near the crosswalk will interfere with the vehicle. Therefore, the intention of pedestrians and non-motorized vehicles is judged to filter out obstacles that are not related to the autonomous vehicle's movement.
[0031] The intentions of pedestrians or non-motorized vehicles on crosswalks are divided into two types: When a pedestrian or non-motorized vehicle is within a certain range of the side of the road before the vehicle enters the lane, if the pedestrian or non-motorized vehicle is stationary, it is difficult to determine their intention, and they are classified as dangerous obstacles. In this case, the dangerous obstacle flag is set to danger_obj_flag=true. When a pedestrian or non-motorized vehicle enters the crosswalk and is already on the zebra crossing, if the obstacle is outside the lane before the vehicle enters the lane and has a speed away from the direction the vehicle is about to enter the lane, it is classified as a safe obstacle, and the safe obstacle flag is set to danger_obj_flag=false.
[0032] When the vehicle collides with an obstacle, the intention is to terminate and the sign position is closed; when the vehicle is aligned with the road stop line, the vehicle is determined to have reached the destination, the intention is to terminate and the sign position is closed.
[0033] Furthermore, after the Sim-One simulation module builds the simulation environment, the environment processing module encapsulates the environment and provides the env.reset() and env.step() interface functions for data interaction with the running module: the env.reset() function resets the environment and restarts the random traffic flow case.
[0034] The `env.step()` function serves as the interface between the environment control module and the Sim-One simulation module. The environment control module receives actions from the runtime module and inputs them into the simulation environment via ROS topic control commands. The vehicle in the simulation environment controls itself based on the input information. The simulation environment then proceeds to the next simulation step and returns the state, reward, and whether the simulation has ended action information at the next moment to the environment control module via ROS topic control commands.
[0035] Furthermore, the operation module includes an interaction interface for the environment control module, an interaction interface for the autonomous driving module, and an interaction interface for the experience pool module; the interaction interface for the environment control module receives the status, reward, and whether the simulation has ended action information transmitted back from the environment control module, and transmits the action information to the environment control module.
[0036] The autonomous driving module interaction interface forwards the environmental state obtained from the environmental control module to the autonomous driving intelligent agent module, and obtains the generated actions from the autonomous driving intelligent agent module.
[0037] The experience pool module's interaction interface obtains a collection of experience data from the environment control module, including status, rewards, and whether the simulation has ended, as well as from the autonomous driving agent module, including status, actions, rewards, next-moment status, and simulation action end information. This collection is then sent to the experience pool module.
[0038] Furthermore, the autonomous driving intelligent agent module includes an action selection interface, a random action selection interface, a neural network training interface, a model storage interface, and a model loading interface;
[0039] The action selection interface takes the current environmental state as input and outputs an action. The output action is a fixed value or a definite probability distribution, which is used for testing after the vehicle training is completed.
[0040] The actions are longitudinal control actions and lateral control actions of the vehicle. The longitudinal control actions are vehicle acceleration control, including acceleration 1, acceleration 0 and acceleration -1. Acceleration 1 is an acceleration action, acceleration 0 is a constant speed action, and acceleration -1 is a deceleration action.
[0041] The lateral control action of the vehicle is to use PID control to track a fixed trajectory. The fixed trajectory is a quadratic B-sample curve with the control points being the midpoint of the rear axle of the vehicle when the front of the vehicle is aligned with the road stop line, the midpoint of the rear axle of the vehicle when the rear of the vehicle is aligned with the road stop line, and the center point of the road at the intersection.
[0042] The random action selection interface outputs randomly selected environmental state output actions, which are used in the training phase of the vehicle to explore the environment through random actions, expand the policy search range, and avoid getting trapped in local optimal policies.
[0043] The neural network training interface accepts sampled data from the experience pool module and uses the sampled data to train the network.
[0044] The model storage interface is used to store neural network models at different stages during training;
[0045] The model loading interface is used to load neural network models during the testing phase.
[0046] Furthermore, the autonomous driving intelligent agent module is trained using DQN as a function approximator to obtain the optimal Q value, wherein DQN is a deep Q-network.
[0047] Furthermore, the autonomous driving intelligent agent module has a built-in reward function based on traffic rules:
[0048]
[0049] Furthermore, in step S3, the filtering of obstacles during the intent determination specifically involves:
[0050] The simulation environment outputs the position information of the obstacle at time intervals of Δt. The current coordinates of the obstacle are (x0, y0), and the coordinates of the obstacle at previous times are recorded as (x1, y1), (x2, y2), ..., (x9, y9) respectively.
[0051] To ensure the stability of the velocity calculation, the velocity is calculated using the three most recent points on the historical trajectory.
[0052]
[0053]
[0054] The included angle is:
[0055] vec0 = (x0 - x9, y0 - y9)
[0056] vec1 = (x1 - x9, y1 - y9)
[0057] vec8 = (x8 - x9, y8 - y9)
[0058] θ1 = vec0 × vec1
[0059] θ2=vec0×vec2
[0060] …
[0061] θ8 = vec0 × vec8
[0062] such as θ i If (i = 1, 2, ..., 8) are all greater than zero, then the vehicle is turning right; θ i (i = 1, 2, ..., 8) are all less than zero, indicating that the vehicle is turning left; θ i (i = 1, 2, ..., 8) can be positive or negative, indicating that the vehicle is going straight.
[0063] Furthermore, it also includes data transmission paths, which include status / action data transmission paths and experience data transmission paths;
[0064] The state action data transmission path is as follows: the state information is sent from the data preprocessing module, transmitted to the autonomous driving intelligent agent module via the environmental control module and the operation module, the autonomous driving intelligent agent module makes a decision action based on the obtained environmental state information, and sends it back to the Sim-One simulation module via the operation module and the data preprocessing module, the Sim-One simulation module executes the corresponding instructions according to the action, and performs simulation to obtain the environmental state at the next moment.
[0065] The experience data transmission path is as follows: obstacle and vehicle information is sent from the Sim-One random traffic flow simulation module, enters the data preprocessing module, filters and converts the obstacle and vehicle information into state and end-of-life information, and reaches the operation module via the environmental control module. Action data is sent from the autonomous driving agent module to the operation module. The operation module processes the action data and state data to obtain an experience set of state, action, reward, next-moment state, and simulation action end information, which is then sent to the experience pool module for storage. When the vehicle interacts with the environment, the experience pool module also sends sampled experience samples to the autonomous driving agent module. The autonomous driving agent module uses the obtained experience samples for training to improve the performance of the policy network.
[0066] Compared with the prior art, the present invention has the following beneficial technical effects:
[0067] It can better adapt to different traffic flow characteristics at intersections in open roads, and improve the intelligence of autonomous vehicle decision-making and control, so as to solve the decision-making and control problem of autonomous vehicles in complex traffic environments at unprotected intersections. Attached Figure Description
[0068] Figure 1 This is a framework diagram of the present invention;
[0069] Figure 2 This is a flowchart illustrating the simulation environment setup for the Sim-One simulation module of this invention.
[0070] Figure 3 This is a framework diagram of the data preprocessing module;
[0071] Figure 4 This is a map showing the vehicle's location and the obstacles.
[0072] Figure 5 Map showing the unprotected left-turn zone at an intersection;
[0073] Figure 6 This is a schematic diagram illustrating a scenario where vehicles are approaching from the opposite straight lane.
[0074] Figure 7 This is a map predicting the intentions of oncoming vehicles.
[0075] Figure 8 Obstacle intention filtering map;
[0076] Figure 9 For vehicle intent determination;
[0077] Figure 10 Predicted maps for pedestrians and non-motorized vehicles;
[0078] Figure 11 A prediction diagram for determining the intentions of pedestrians and non-motorized vehicles;
[0079] Figure 12 This is a schematic diagram of the environmental control module;
[0080] Figure 13 This is a schematic diagram of the DQN algorithm.
[0081] Figure 14 Here is a flowchart of the DQN algorithm training process;
[0082] Figure 15 A path diagram for transferring state and action data;
[0083] Figure 16 This is a diagram of the data transmission path for experience data. Detailed Implementation
[0084] like Figures 1-16 As shown, the unprotected left turn decision control method for intersections based on deep reinforcement learning includes Sim-One random traffic flow simulation scenario construction, data preprocessing module, environmental control module, operation module, autonomous driving intelligent agent module and experience pool module;
[0085] Sim-One Random Traffic Flow Simulation Scenario Setup: The simulation environment is set up using the Sim-One Random Traffic Flow Simulation Scenario Setup module.
[0086] Data preprocessing module: used to filter obstacles based on their location and the intentions of other traffic participants to reduce the dimensionality of the state space;
[0087] Environment control module: Used to control the Sim-One simulation module and interact with the running module;
[0088] The operation module is used for information interaction between the data preprocessing module, the environmental control module, the operation module, the autonomous driving intelligent agent module, and the experience pool module.
[0089] Autonomous Driving Intelligent Agent Module: The status information sent by the data processing module is transmitted to the autonomous driving intelligent agent module through the environmental control module and the operation module. The autonomous driving intelligent agent module makes decision-making actions based on the status information.
[0090] Experience pool module: Used to store experience transferred from the running module. When the stored experience reaches the set maximum value, the oldest experience is deleted and the latest experience is stored. It is also used to perform batch sampling from the stored experience and pass the sampled experience to the autonomous driving agent module for updating the autonomous vehicle policy network.
[0091] The simulation environment setup includes road network configuration, vehicle configuration, and traffic flow configuration;
[0092] The road network is configured to load different intersection maps to provide a rich training dataset, with the origin of the simulation environment coordinate system being the center point of the intersection;
[0093] The vehicle configuration is as follows: the simulated vehicle is equipped with sensors to acquire real-time simulation environment status information and vehicle status information. The simulation environment status information includes obstacle information, obstacle position, obstacle speed, and obstacle type; the vehicle status information includes vehicle position information and vehicle speed information.
[0094] The traffic flow configuration sets the simulation test road to be an urban road simulated using traffic flow parameters.
[0095] To reduce the number of obstacles and lower the dimensionality of the state space, the data processing module first preprocesses the obstacles;
[0096] S1, the preprocessing is as follows:
[0097] First, extract the environmental state representation data: S = {s0, s1, s2, s3, ..., s} final} where s0 is the initial state, i.e., the moment when the autonomous vehicle is aligned with the stop line, which is the reinforcement learning process; s1 is the state in the state space at time Δt, s2 is the state in the state space at time 2*Δt, s3 is the state in the state space at time 3*Δt, and so on, s final The state space is when the rear of the vehicle is aligned with the stop line; the state space ends when the rear of the vehicle is aligned with the stop line.
[0098] S2 divides obstacles into zones based on the vehicle's location to reduce the dimensionality of the state space:
[0099] First, the area of the unprotected left turn scenario is divided into a dangerous area and a safe area. The dangerous area includes dangerous area 1 and dangerous area 2.
[0100] Then, the trajectory of the upper left corner of the vehicle is calculated: When the vehicle turns left, the left front corner, i.e., the upper left corner of the vehicle, will be the first to collide. Therefore, it is necessary to calculate the position coordinates of the left front corner of the vehicle. The Sim-One simulation module outputs the position coordinates and heading of the rear axle midpoint of the vehicle in real time. The position from the front center point of the vehicle to the rear axle center is L, and the vehicle width is W. Then, the coordinates of the left front position point of the autonomous vehicle can be expressed as:
[0101]
[0102] Where x0 and y0 are the position coordinates of the midpoint of the rear axle of the vehicle, and yaw is the heading of the midpoint of the rear axle of the vehicle;
[0103] During the left turn, the distance D1 between the coordinates (x, y) of the vehicle's left front position and the right boundary of danger zone 1, and the distance D2 between the coordinates (x, y) of the vehicle's left front position and the right boundary of danger zone 2 are calculated. When the vehicle begins to enter danger zone 1, D1 = 0 and D2 > 0. At this time, all obstacles in danger zone 1 may collide with the vehicle. Therefore, the flag of obstacles in danger zone 1 is set to danger_area_flag = true; the flag of obstacles in danger zone 2 is set to danger_area_flag = false.
[0104] When the vehicle enters danger zone 2 and D2=0, for the vehicle, there is a possibility of collision with obstacles in danger zone 1 and danger zone 2, as well as straight-going and right-turning vehicles in danger zone 1, and pedestrians and non-motorized vehicles on the sidewalk in danger zone 2. Therefore, the flags of obstacles in danger zone 1 and danger zone 2 should be set to danger_area_flag=true.
[0105] S3, Intent Judgment
[0106] This includes vehicle intent judgment and pedestrian and non-motorized vehicle intent prediction. Vehicle intent judgment is as follows: when an autonomous vehicle turns left, vehicles going straight and turning right in the opposite lane will interfere with the autonomous vehicle entering the same lane, while vehicles turning left in the opposite straight lane will not affect the autonomous vehicle's left turn. Therefore, vehicles in the opposite lane with the intent to turn right or go straight are marked as dangerous vehicles, and vehicles turning left are marked as safe vehicles.
[0107] The obstacle types are divided into motor vehicles, non-motor vehicles and pedestrians. The Sim-One simulation module sends the obstacle category truth value through sensors and outputs the obstacle's position information. By using the current position information and the position information of 9 historical moments, the speed, heading and position of the obstacle are determined by the trajectory points, thereby judging the collision risk between the obstacle and the autonomous vehicle. This filters the obstacles, reduces the state space dimension of the input of the deep reinforcement learning network, and shortens the training time.
[0108] Pedestrian and non-motorized vehicle intention prediction: When the vehicle is about to complete a left turn and enter the crosswalk, pedestrians and non-motorized vehicles near the crosswalk will interfere with the vehicle. Therefore, the intention of pedestrians and non-motorized vehicles is judged to filter out obstacles that are not related to the autonomous vehicle's movement.
[0109] The intentions of pedestrians or non-motorized vehicles on crosswalks are divided into two types: When a pedestrian or non-motorized vehicle is within a certain range of the side of the road before the vehicle enters the lane, if the pedestrian or non-motorized vehicle is stationary, it is difficult to determine their intention, and they are classified as dangerous obstacles. In this case, the dangerous obstacle flag is set to danger_obj_flag=true. When a pedestrian or non-motorized vehicle enters the crosswalk and is already on the zebra crossing, if the obstacle is outside the lane before the vehicle enters the lane and has a speed away from the direction the vehicle is about to enter the lane, it is classified as a safe obstacle, and the safe obstacle flag is set to danger_obj_flag=false.
[0110] When the vehicle collides with an obstacle, the intention is to terminate and the sign position is closed; when the vehicle is aligned with the road stop line, the vehicle is determined to have reached the destination, the intention is to terminate and the sign position is closed.
[0111] After the Sim-One simulation module sets up the simulation environment, the environment processing module encapsulates the environment and provides the `env.reset()` and `env.step()` interface functions for data interaction with the runtime module.
[0112] The env.reset() function resets the environment and restarts random traffic flow.
[0113] The `env.step()` function serves as the interface between the environment control module and the Sim-One simulation module. The environment control module receives actions from the runtime module and inputs them into the simulation environment via ROS topic control commands. The vehicle in the simulation environment controls itself based on the input information. The simulation environment then proceeds to the next simulation step and returns the state, reward, and whether the simulation has ended action information at the next moment to the environment control module via ROS topic control commands.
[0114] The operation module includes an interaction interface for the environment control module, an interaction interface for the autonomous driving module, and an interaction interface for the experience pool module. The interaction interface for the environment control module receives the status, reward, and action information (whether the simulation has ended) returned from the environment control module, and then transmits the action information to the environment control module.
[0115] The autonomous driving module's interaction interface forwards the environmental state obtained from the environmental control module to the autonomous driving agent module, and obtains the generated actions from the autonomous driving agent module.
[0116] The experience pool module's interaction interface obtains the status, reward, and simulation termination information from the environment control module, as well as the status, action, reward, next moment status, and simulation termination information from the autonomous driving agent module. This information is processed into an experience set and sent to the experience pool module.
[0117] The autonomous driving intelligent agent module includes an action selection interface, a random action selection interface, a neural network training interface, a model storage interface, and a model loading interface;
[0118] The action selection interface takes the current environment state as input and outputs the action. The output action is a fixed value or a definite probability distribution, which is used for testing after the vehicle training is completed.
[0119] The actions are longitudinal control actions and lateral control actions of the vehicle. The longitudinal control actions are vehicle acceleration control, including acceleration 1, acceleration 0 and acceleration -1. Acceleration 1 is an acceleration action, acceleration 0 is a constant speed action, and acceleration -1 is a deceleration action.
[0120] The lateral control action of the vehicle adopts PID control to track a fixed trajectory. The fixed trajectory is a quadratic B-sample curve with the control points being the midpoint of the rear axle of the vehicle when the front of the vehicle is aligned with the road stop line, the midpoint of the rear axle of the vehicle when the rear of the vehicle is aligned with the road stop line, and the center point of the road at the intersection.
[0121] The random action selection interface outputs randomly selected environmental state output actions, which are used in the training phase of the vehicle. By using random actions to explore the environment, the search range of the policy is expanded, and the vehicle avoids getting stuck in local optima.
[0122] The neural network training interface accepts sampled data from the experience pool module and uses the sampled data to train the network;
[0123] The model storage interface is used to store neural network models at different stages during training;
[0124] The model loading interface is used to load neural network models during the testing phase.
[0125] The autonomous driving intelligent agent module is trained using DQN as a function approximator to obtain the optimal Q value. DQN is a deep Q-network.
[0126] The autonomous driving agent module has a built-in reward function based on traffic rules:
[0127]
[0128] S3, the specific steps for filtering obstacles during intent determination are as follows:
[0129] The simulation environment outputs the position information of the obstacle at time intervals of Δt. The current coordinates of the obstacle are (x0, y0), and the coordinates of the obstacle at previous times are recorded as (x1, y1), (x2, y2), ..., (x9, y9) respectively.
[0130] To ensure the stability of the velocity calculation, the velocity is calculated using the three most recent points on the historical trajectory.
[0131]
[0132] like Figure 9 As shown, in the formula for calculating the direction of velocity... The vector represents the initial trajectory point in the set of 10 trajectory points, that is, the vector from the trajectory points before time 9*Δt to the current trajectory point. The vector represents the vector from other historical trajectory points to the current trajectory point, and is calculated sequentially. Vector and The cross product of vectors.
[0133] The included angle is:
[0134] vec0 = (x0 - x9, y0 - y9)
[0135] vec1 = (x1 - x9, y1 - y9)
[0136] vec8 = (x8 - x9, y8 - y9)
[0137] θ1 = vec0 × vec1
[0138] θ2=vec0×vec2
[0139] …
[0140] θ8 = vec0 × vec8
[0141] such as θ i If (i = 1, 2, ..., 8) are all greater than zero, then the vehicle is turning right; θ i (i = 1, 2, ..., 8) are all less than zero, indicating that the vehicle is turning left; θ i (i = 1, 2, ..., 8) can be positive or negative, indicating that the vehicle is going straight.
[0142] It also includes data transmission paths, which include status / action data transmission paths and experience data transmission paths;
[0143] The state and action data transmission path is as follows: the state information is sent from the data preprocessing module, transmitted to the autonomous driving intelligent agent module via the environmental control module and the operation module, the autonomous driving intelligent agent module makes decision actions based on the obtained environmental state information, and sends them back to the Sim-One simulation module via the operation module and the data preprocessing module, the Sim-One simulation module executes the corresponding instructions according to the actions, and performs simulation to obtain the environmental state at the next moment.
[0144] The experience data transmission path is as follows: obstacle and vehicle information is sent from the Sim-One random traffic flow simulation module, enters the data preprocessing module, filters and converts the obstacle and vehicle information into state and end-of-life information, and reaches the operation module via the environmental control module. Action data is sent from the autonomous driving agent module to the operation module. The operation module processes the action data and state data to obtain an experience set of state, action, reward, next-moment state, and simulation action end information, which is then sent to the experience pool module for storage. When the vehicle interacts with the environment, the experience pool module also sends sampled experience samples to the autonomous driving agent module. The autonomous driving agent module uses the obtained experience samples for training to improve the performance of the policy network.
Claims
1. A decision-making and control method for unprotected left turns at intersections based on deep reinforcement learning, characterized by: It includes the Sim-One random traffic flow simulation scenario construction, data preprocessing module, environmental control module, operation module, autonomous driving intelligent agent module, and experience pool module; Sim-One Random Traffic Flow Simulation Scenario Setup: The simulation environment is set up using the Sim-One Random Traffic Flow Simulation Scenario Setup module. Data preprocessing module: used to filter obstacles based on their location and the intentions of other traffic participants to reduce the dimensionality of the state space; Environment control module: Used to control the Sim-One simulation module and interact with the running module; The operation module is used for information interaction between the data preprocessing module, the environmental control module, the operation module, the autonomous driving intelligent agent module, and the experience pool module. Autonomous Driving Intelligent Agent Module: The status information sent by the data processing module is transmitted to the autonomous driving intelligent agent module through the environmental control module and the operation module. The autonomous driving intelligent agent module makes decision-making actions based on the status information. Experience pool module: Used to store experience transferred from the running module. When the stored experience reaches the set maximum value, the oldest experience is deleted and the latest experience is stored. It is also used to perform batch sampling from the stored experience and pass the sampled experience to the autonomous driving agent module for updating the autonomous vehicle policy network.
2. The method for unprotected left turn decision control at intersections based on deep reinforcement learning according to claim 1, characterized in that, The simulation environment setup includes road network configuration, vehicle configuration, and traffic flow configuration; The road network is configured to load different intersection maps to provide a rich training dataset, and the origin of the simulation environment coordinate system is the center point of the intersection. The vehicle is configured such that the simulated vehicle is equipped with sensors to acquire real-time simulation environment status information and vehicle status information. The simulation environment status information includes obstacle information, obstacle position, obstacle speed, and obstacle type. The vehicle status information includes the vehicle's location information and vehicle speed information; The traffic flow configuration is to set the simulation test road as an urban road simulated using traffic flow parameters.
3. The method for unprotected left turn decision control at intersections based on deep reinforcement learning according to claim 1, characterized in that, To reduce the number of obstacles and lower the dimensionality of the state space, the data processing module first preprocesses the obstacles. S1, the preprocessing is as follows: First, extract the environmental state representation data: S = {s0, s1, s2, s3, ..., s} final } where s0 is the initial state, i.e., the moment when the autonomous vehicle is aligned with the stop line, which is the reinforcement learning process; s1 is the state in the state space at time Δt, s2 is the state in the state space at time 2*Δt, s3 is the state in the state space at time 3*Δt, and so on, s final The state space is when the rear of the vehicle is aligned with the stop line; the state space ends when the rear of the vehicle is aligned with the stop line. S2 divides obstacles into zones based on the vehicle's location to reduce the dimensionality of the state space: First, the area of the unprotected left turn scenario is divided into a dangerous area and a safe area. The dangerous area includes dangerous area 1 and dangerous area 2. Then, the trajectory of the upper left corner of the vehicle is calculated: When the vehicle turns left, the left front corner, i.e., the upper left corner of the vehicle, will be the first to collide. Therefore, it is necessary to calculate the position coordinates of the left front corner of the vehicle. The Sim-One simulation module outputs the position coordinates and heading of the rear axle midpoint of the vehicle in real time. The position from the front center point of the vehicle to the rear axle center is L, and the vehicle width is W. Then, the coordinates of the left front position point of the autonomous vehicle can be expressed as: Where x0, y0 are the position coordinates of the midpoint of the rear axle of the vehicle, and yaw is the heading of the midpoint of the rear axle of the vehicle; During the left turn, the distance D1 between the coordinates (x, y) of the vehicle's left front position and the right boundary of danger zone 1, and the distance D2 between the coordinates (x, y) of the vehicle's left front position and the right boundary of danger zone 2 are calculated. When the vehicle begins to enter danger zone 1, D1 = 0 and D2 > 0. At this time, all obstacles in danger zone 1 may collide with the vehicle. Therefore, the flag of obstacles in danger zone 1 is set to danger_area_flag = true; the flag of obstacles in danger zone 2 is set to danger_area_flag = false. When the vehicle enters danger zone 2 and D2=0, for the vehicle, there is a possibility of collision with obstacles in danger zone 1 and danger zone 2, as well as straight-going and right-turning vehicles in danger zone 1, and pedestrians and non-motorized vehicles on the sidewalk in danger zone 2. Therefore, the flags of obstacles in danger zone 1 and danger zone 2 should be set to danger_area_flag=true. S3, Intent Judgment This includes vehicle intent judgment and pedestrian and non-motorized vehicle intent prediction. Vehicle intent judgment is as follows: when an autonomous vehicle turns left, vehicles going straight and turning right in the opposite lane will interfere with the autonomous vehicle entering the same lane, while vehicles turning left in the opposite straight lane will not affect the autonomous vehicle's left turn. Therefore, vehicles in the opposite lane with the intent to turn right or go straight are marked as dangerous vehicles, and vehicles turning left are marked as safe vehicles. The obstacle types are divided into motor vehicles, non-motor vehicles and pedestrians. The Sim-One simulation module sends the obstacle category truth value through sensors and outputs the obstacle's position information. By using the current position information and the position information of 9 historical moments, the speed, heading and position of the obstacle are determined by the trajectory points, thereby judging the collision risk between the obstacle and the autonomous vehicle. This filters the obstacles, reduces the state space dimension of the input of the deep reinforcement learning network, and shortens the training time. Pedestrian and non-motorized vehicle intention prediction: When the vehicle is about to complete a left turn and enter the crosswalk, pedestrians and non-motorized vehicles near the crosswalk will interfere with the vehicle. Therefore, the intention of pedestrians and non-motorized vehicles is judged to filter out obstacles that are not related to the autonomous vehicle's movement. The intentions of pedestrians or non-motorized vehicles on crosswalks are divided into two types: When a pedestrian or non-motorized vehicle is within a certain range of the side of the road before the vehicle enters the lane, if the pedestrian or non-motorized vehicle is stationary, it is difficult to determine their intention, and they are classified as dangerous obstacles. In this case, the dangerous obstacle flag is set to danger_obj_flag=true. When a pedestrian or non-motorized vehicle enters the crosswalk and is already on the zebra crossing, if the obstacle is outside the lane before the vehicle enters the lane and has a speed away from the direction the vehicle is about to enter the lane, it is classified as a safe obstacle, and the safe obstacle flag is set to danger_obj_flag=false. When the vehicle collides with an obstacle, the intention is to terminate and the sign position is closed; when the vehicle is aligned with the road stop line, the vehicle is determined to have reached the destination, the intention is to terminate and the sign position is closed.
4. The method for unprotected left turn decision control at intersections based on deep reinforcement learning according to claim 3, characterized in that, After the Sim-One simulation module builds the simulation environment, the environment processing module encapsulates the environment and provides the env.reset() and env.step() interface functions for data interaction with the runtime module. The env.reset() function resets the environment and restarts random traffic flow. The `env.step()` function serves as the interface between the environment control module and the Sim-One simulation module. The environment control module receives actions from the runtime module and inputs them into the simulation environment via ROS topic control commands. The vehicle in the simulation environment controls itself based on the input information. The simulation environment then proceeds to the next simulation step and returns the state, reward, and whether the simulation has ended action information at the next moment to the environment control module via ROS topic control commands.
5. The method for unprotected left turn decision control at intersections based on deep reinforcement learning according to claim 1, characterized in that, The operation module includes an interaction interface for the environment control module, an interaction interface for the autonomous driving module, and an interaction interface for the experience pool module. The interaction interface for the environment control module receives the status, reward, and whether the simulation has ended action information transmitted back from the environment control module, and transmits the action information to the environment control module. The autonomous driving module interaction interface forwards the environmental state obtained from the environmental control module to the autonomous driving intelligent agent module, and obtains the generated actions from the autonomous driving intelligent agent module. The experience pool module's interaction interface obtains a collection of experience data from the environment control module, including status, rewards, and whether the simulation has ended, as well as from the autonomous driving agent module, including status, actions, rewards, next-moment status, and simulation action end information. This collection is then sent to the experience pool module.
6. The method for unprotected left turn decision control at intersections based on deep reinforcement learning according to claim 1, characterized in that, The autonomous driving intelligent agent module includes an action selection interface, a random action selection interface, a neural network training interface, a model storage interface, and a model loading interface. The action selection interface takes the current environmental state as input and outputs an action. The output action is a fixed value or a definite probability distribution, which is used for testing after the vehicle training is completed. The actions are longitudinal control actions and lateral control actions of the vehicle. The longitudinal control actions are vehicle acceleration control, including acceleration 1, acceleration 0 and acceleration -1. Acceleration 1 is an acceleration action, acceleration 0 is a constant speed action, and acceleration -1 is a deceleration action. The lateral control action of the vehicle is to use PID control to track a fixed trajectory. The fixed trajectory is a quadratic B-sample curve with the control points being the midpoint of the rear axle of the vehicle when the front of the vehicle is aligned with the road stop line, the midpoint of the rear axle of the vehicle when the rear of the vehicle is aligned with the road stop line, and the center point of the road at the intersection. The random action selection interface outputs randomly selected environmental state output actions, which are used in the training phase of the vehicle to explore the environment through random actions, expand the policy search range, and avoid getting trapped in local optimal policies. The neural network training interface accepts sampled data from the experience pool module and uses the sampled data to train the network. The model storage interface is used to store neural network models at different stages during training; The model loading interface is used to load neural network models during the testing phase.
7. The method for unprotected left turn decision control at intersections based on deep reinforcement learning according to claim 2, characterized in that, The autonomous driving intelligent agent module is trained using DQN as a function approximator to obtain the optimal Q value. DQN is a deep Q-network.
8. The method for unprotected left turn decision control at intersections based on deep reinforcement learning according to claim 3, characterized in that, The autonomous driving agent module has a built-in reward function based on traffic rules: 。 9. The method for unprotected left turn decision control at intersections based on deep reinforcement learning according to claim 3, characterized in that, S3, the intention to filter obstacles in the determination of intent, specifically involves: The simulation environment outputs the position information of the obstacle at time intervals of Δt. The current coordinates of the obstacle are denoted as (x0, y0), and the coordinates at historical times are denoted as (x1, y1), (x2, y2), ..., (x9, y9). To ensure the stability of the velocity calculation, the velocity is calculated using the three most recent points on the historical trajectory. The included angle is: vec o =(x0-x9,y0-y9) vec1 = (x1 - x9, y1 - y9) vec8 = (x8 - x9, y8 - y9) θ1 = vec0 × vec1 θ2=vec0×vec2 … θ8 = vec0 × vec8 such as θ i If (i1, 2, ..., 8) are all greater than zero, it means the vehicle is turning right; θ i (i = 1, 2, ..., 8) are all less than zero, indicating that the vehicle is turning left; θ i (i = 1, 2, ..., 8) can be positive or negative, indicating that the vehicle is going straight.
10. The method for unprotected left turn decision control at intersections based on deep reinforcement learning according to claim 1, characterized in that, It also includes data transmission paths, which include status / action data transmission paths and experience data transmission paths; The state action data transmission path is as follows: the state information is sent from the data preprocessing module, transmitted to the autonomous driving intelligent agent module via the environmental control module and the operation module, the autonomous driving intelligent agent module makes a decision action based on the obtained environmental state information, and sends it back to the Sim-One simulation module via the operation module and the data preprocessing module, the Sim-One simulation module executes the corresponding instructions according to the action, and performs simulation to obtain the environmental state at the next moment. The experience data transmission path is as follows: obstacle and vehicle information is sent from the Sim-One random traffic flow simulation module, enters the data preprocessing module, filters and converts the obstacle and vehicle information into state and end-of-life information, and reaches the operation module via the environmental control module. Action data is sent from the autonomous driving agent module to the operation module. The operation module processes the action data and state data to obtain an experience set of state, action, reward, next-moment state, and simulation action end information, which is then sent to the experience pool module for storage. When the vehicle interacts with the environment, the experience pool module also sends sampled experience samples to the autonomous driving agent module. The autonomous driving agent module uses the obtained experience samples for training to improve the performance of the policy network.