Non-signal intersection automatic driving decision-making method
By decoupling vehicle control into horizontal and vertical control, and using tracking algorithms and collision probability prediction combined with deep reinforcement learning, the problem of deep reinforcement learning relying on reward shaping and poor stability in signal-free intersection driving is solved, and driving decisions are improved safety and efficiency.
Patent Information
- Application Number
- CN202411174701.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-26
- Publication Date
- 2024-11-12
AI Technical Summary
Deep reinforcement learning relies too much on reward shaping in signalless intersection driving decisions and poor algorithm stability, resulting in low learning efficiency and inaccurate strategies.
Vehicle control is decoupled into lateral and vertical control, which is implemented by tracking algorithms. Vertical control is determined by co-deciding based on collision probability prediction and deep reinforcement learning, and using risk assessment to select appropriate decision models to reduce the risk of deviating lane and improve safety.
It improves the safety and efficiency of driving without signal intersections, reduces the dependence on reward shaping, enhances the stability and learning efficiency of the algorithm, and reduces the number of iterative training times.
Smart Images

Figure CN118928471A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of autonomous driving, and in particular relates to an autonomous driving decision method for an unsignaled intersection. Background Art
[0002] Autonomous driving decision-making is a core component of autonomous driving technology. It is responsible for making real-time decisions and controls based on information such as the vehicle's status, environmental conditions, and driving strategies. Unsignalized intersection driving decision-making is an important branch of autonomous driving technology, requiring the intelligent agent to make reasonable decisions and drive in the absence of traffic lights to avoid collisions and congestion and achieve safe and efficient traffic flow. Deep reinforcement learning, as a method of learning strategies by interacting with the environment, has received widespread attention and research in the field of autonomous driving in recent years.
[0003] In the driving decision-making task at unsignalized intersections, traditional reinforcement learning methods such as Q-learning and SARSA have been extensively studied. However, these methods usually use sparse reward function design (such as Cao et al.'s Trustworthy safety improvement for autonomous driving using reinforcement learning), which results in the agent receiving rewards only at a few time steps during training, thus reducing data utilization efficiency and learning speed.
[0004] To overcome this problem, reward shaping techniques are widely used. Reward shaping introduces auxiliary rewards in the learning process, so that the agent can obtain more reward signals in the intermediate stages of task execution to improve learning efficiency. However, reward shaping methods require a lot of engineering design and experiments, and may cause the agent to behave strangely or converge to local minima prematurely (Vecerik et al., "Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards").
[0005] Although reward shaping technology has alleviated the problems caused by sparse rewards to a certain extent, it still faces the following challenges in practical applications: 1. Dependence on reward shaping: The effectiveness of reward shaping is highly dependent on the design of auxiliary rewards. Incorrect reward design may cause the agent to learn the wrong strategy, affecting the accuracy of driving decisions. 2. Algorithm instability: Due to the large number of random processes in the task of driving decision-making at unsignaled intersections, traditional reinforcement learning algorithms show significant instability during training and are difficult to converge to an ideal strategy. 3. A large number of experimental requirements: In order to find a well-performing model, existing methods require a large number of iterative training verifications, which is time-consuming and labor-intensive in actual driving decision-making applications. Summary of the invention
[0006] In view of this, the present invention is committed to solving the problems of over-reliance on reward shaping and poor algorithm stability faced by deep reinforcement learning in driving decisions at unsignalized intersections, so as to achieve more reliable driving decisions. In response to this challenge, the present invention proposes an innovative method for alleviating the reliance of deep reinforcement learning algorithms on reward shaping and reducing algorithm instability.
[0007] In order to solve the above problems, the present invention provides an automatic driving decision method for an unsignalized intersection, the method comprising:
[0008] Obtain the target lane centerline, real-time status data and historical trajectory data; the target lane centerline is the position set of the target lane center point; the real-time status data includes the state of the autonomous vehicle and the state of surrounding vehicles, and the state is the coordinates and speed in two-dimensional space; the historical trajectory data is the position set at each time;
[0009] Inputting the real-time status data into a lateral driving decision model based on a tracking algorithm to output a lateral control decision of the autonomous vehicle;
[0010] Based on the similarity distance between the center line of the target lane and the historical trajectory data, the surrounding vehicles are divided into relevant vehicles and non-relevant vehicles; the risk of the relevant vehicles is evaluated based on the entry time of the relevant vehicles;
[0011] When the risk assessment result is greater than or equal to the global risk threshold, the real-time status data is input into a longitudinal driving decision model based on collision probability prediction, and a longitudinal control decision of the autonomous vehicle is output; the autonomous vehicle executes the longitudinal control instruction based on the collision probability prediction;
[0012] When the risk assessment result is less than the global risk threshold, the real-time status data is input into a longitudinal driving decision model based on deep reinforcement learning, and a longitudinal control decision of the autonomous vehicle is output; the autonomous vehicle executes the longitudinal control instruction based on deep reinforcement learning.
[0013] Beneficial effects of the present invention:
[0014] The present invention decouples vehicle control into lateral control and longitudinal control. The lateral behavior control of the autonomous vehicle is implemented by a pure tracking algorithm, and the longitudinal behavior control is jointly determined by a deep reinforcement learning algorithm and a driving strategy based on collision probability prediction. The present invention adopts a lateral driving decision model based on a tracking algorithm, which is suitable for lateral driving decision scenarios, helps to reduce the risk of lane deviation, and improves overall driving safety. The present invention adopts a longitudinal driving decision model based on collision probability prediction. By predicting whether the future driving path of surrounding vehicles conflicts with the future driving path of the autonomous vehicle, the wind direction level of the relevant vehicles is evaluated, and different risk levels are defined to determine the risk assessment result of the autonomous vehicle. The relationship between the risk assessment result and the global risk threshold is used to select whether to use a longitudinal driving decision model based on collision probability prediction or a longitudinal driving decision model based on deep reinforcement learning to obtain the longitudinal control decision of the autonomous vehicle. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] Figure 1 A schematic diagram of a flow chart of a method for making decisions on an automatic driving at an unsignalized intersection according to an embodiment of the present invention;
[0016] Figure 2 A schematic diagram of a scenario of a lateral driving decision model based on a tracking algorithm according to an embodiment of the present invention; DETAILED DESCRIPTION
[0017] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0018] Figure 1 FIG. 1 is a flow chart of a method for making decisions on an automatic driving at an unsignalized intersection according to an embodiment of the present invention. Figure 1 As shown, the method includes:
[0019] 101. Obtain a target lane centerline, real-time status data, and historical trajectory data; the target lane centerline is a position set of the target lane center point; the real-time status data includes the state of the autonomous vehicle and the state of surrounding vehicles, and the state is the coordinates and speed in a two-dimensional space; the historical trajectory data is a position set at each time;
[0020] In an embodiment of the present invention, the historical trajectory data of the vehicle is modeled as:
[0021]
[0022] in, Indicates vehicle v i In t j The position at a certain moment, j∈{1,2,...,m}, where m represents the length of the point set of the historical trajectory.
[0023] In the embodiment of the present invention, the target lane centerline is expressed as:
[0024]
[0025] in, Indicates the lane centerline c l The position of the kth point, k∈{1,2,…,n}, where n represents the length of the point set of the lane centerline.
[0026] In the embodiment of the present invention, the real-time status data is represented as:
[0027] S={S a ,S1,S2,…,S n}
[0028] Among them, S a represents the state of the autonomous vehicle, S i represents the state of the surrounding vehicles, and n represents the number of observed surrounding vehicles. Specifically:
[0029]
[0030] Where i represents the vehicle number, s x and y They represent the coordinates of the vehicle in two-dimensional space, and the speed of the vehicle in the x-axis direction and y-axis direction are represented by v and x and v y express.
[0031] In an embodiment of the present invention, in order to obtain better autonomous driving decisions, the present invention decouples vehicle control into lateral control and longitudinal control. The lateral control of the autonomous vehicle is implemented by a lateral driving decision model based on a tracking algorithm, and the longitudinal control of the autonomous vehicle is jointly determined by a longitudinal driving decision model based on collision probability prediction and a longitudinal driving decision model based on deep reinforcement learning.
[0032] 102. Input the real-time state data into a lateral driving decision model based on a tracking algorithm, and output a lateral control decision of the autonomous vehicle; in an embodiment of the present invention, the lateral behavior control of the autonomous vehicle is implemented by a pure tracking algorithm, such as Figure 2 As shown, the steering angle is expressed as:
[0033]
[0034] The algorithm defines the foresight distance l f , the purpose is to obtain the steering angle based on the positional relationship between the rear axle of the vehicle and the target point, which defines the curvature of the trajectory between the rear axle and the target. Figure 2 shown. Figure 2 Medium l The foresight distance is l f At the target point under, the curvature of the path can be expressed as:
[0035]
[0036] Where R is the path radius, α is the angle between the vehicle heading and the forward distance vector, and l f Indicates the foresight distance.
[0037] Therefore, the steering angle can be further expressed as:
[0038]
[0039] Where δ represents the vehicle steering angle, L represents the distance from the center of the front axle to the center of the rear axle of the autonomous vehicle, and αl f
[0040] In an embodiment of the present invention, the pure pursuit algorithm (Pure Pursuit) is one of the widely used methods based on geometric tracking, which is particularly suitable for low-speed driving scenarios. The pure pursuit algorithm is a tracking method based on a geometric model. Its basic principle is to simulate human driving behavior and control the front wheel angle to make the vehicle travel along an arc passing through the pre-drawn points. This method has a good tracking effect under low-speed driving conditions and is relatively simple to implement. In the pure pursuit algorithm, the vehicle is usually simplified to a bicycle model (Bicycle Model), and the movement of the vehicle in the Z-axis direction is ignored, and only the movement of the XY horizontal plane is considered. At the same time, assuming that the left and right wheel angles are the same, the left and right tires are merged into one tire to facilitate the construction of a bicycle model. In addition, the effect of changes in the vehicle's driving speed on the transfer of loads on the front and rear axles, as well as the non-rigid characteristics of the vehicle body and suspension system are also ignored.
[0041] In an embodiment of the present invention, the process of obtaining the lateral control decision of the autonomous vehicle using the pure tracking algorithm may include obtaining the coordinates and speed of the autonomous vehicle; drawing a circle with the center of mass of the rear wheel of the vehicle as the center and the foresight distance as the radius, and the point intersecting with the planned path is the pre-drawn point. Based on the geometric relationship between the pre-drawn point, the center of mass of the rear wheel of the vehicle and the steering angle of the vehicle, the front wheel steering angle is derived using trigonometric functions and a bicycle model. The calculated front wheel steering angle is used to control the vehicle to perform steering movement. The position and steering angle of the vehicle are updated according to the motion state of the vehicle, and the above steps are repeated to perform the next round of path tracking.
[0042] The longitudinal behavior of the autonomous vehicle is achieved by a combination of deep reinforcement learning methods and prediction-based methods, which will be explained in detail in the following content.
[0043] 103. Based on the similarity distance between the center line of the target lane and the historical trajectory data, the surrounding vehicles are divided into relevant vehicles and irrelevant vehicles; and the risk of the relevant vehicles is evaluated based on the entry time of the relevant vehicles;
[0044] In an embodiment of the present invention, based on the similarity distance between the target lane centerline and the historical trajectory data, dividing the surrounding vehicles into relevant vehicles and non-relevant vehicles includes calculating the similarity distance between the target lane centerline and each historical trajectory data of the surrounding vehicles, and determining the lane centerline on which the surrounding vehicles are most likely to travel from each historical trajectory data; and dividing the relevant vehicles and non-relevant vehicles according to whether the lane centerline on which the surrounding vehicles are most likely to travel and the target lane centerline are repeated, if they overlap, they are relevant vehicles, and if they do not overlap, they are non-relevant vehicles.
[0045] In the embodiment of the present invention, considering that the road structure of the intersection is relatively fixed, the Hausdorff distance is used to calculate the similarity between the vehicle's historical trajectory and the center line of a given lane.
[0046] For vehicle history and lane centerline The Hausdorff distance between them is defined as:
[0047]
[0048] in:
[0049]
[0050]
[0051] Among them, m and n are the lengths of the discretized point sets of the vehicle's historical trajectory and the lane centerline, respectively. Tips To point The Euclidean distance.
[0052] After calculating the similarity between the vehicle's historical trajectory and the center line of a given lane through the Hausdorff distance, the surrounding vehicles are divided into relevant vehicles and non-relevant vehicles. The risk of relevant vehicles is then assessed through TTE, as follows:
[0053] Use TTE (Time to Entry) to evaluate surrounding vehicles at an intersection. TTE is defined as:
[0054]
[0055] in, Represents the relevant vehicle r at time t i About c j The TTE value, Represents the relevant vehicle r at time t i speed, Represents the relevant vehicle r at time t i To the collision area c j distance; TTE means time required to enter.
[0056] The probability function of different risk levels of surrounding vehicles is defined as follows:
[0057]
[0058] Among them, D represents danger and S represents safety. Therefore, the probability distribution of surrounding vehicles at different risk levels is:
[0059]
[0060] in, Represents the relevant vehicles at time t About the collision area c j The probability of different risk levels under a given TTE, Represents the relevant vehicles at time t About the collision area c j The risk level Represents the relevant vehicles at time t About the collision area c j TTE value; Represents the relevant vehicles at time t About the collision area c j The probability of different TTEs under a given risk level, P(z) is the prior probability of each risk level, z represents the risk level, and Z represents the number of risk levels.
[0061] In the embodiment of the present invention, a variable global risk threshold is designed to realize the use of demonstrations in the early stage of training and the exploration of better strategies in the later stage of training; the global risk threshold is a variable value, and its initial value is 0. As the number of iterations increases, its iteration value gradually increases. At the beginning of training, is set to 0, at which point the autonomous vehicle completely follows the longitudinal control instructions calculated by the longitudinal driving decision model based on collision probability prediction, which allows the autonomous vehicle to quickly accumulate more successful experiences and thus fit a simple initial strategy. As training progresses, Where n represents the number of training rounds. In the straight and left turn tasks, is set to 400. In the simpler right turn task, is set to 50. As the threshold increases, the proportion of longitudinal control instructions executed by the autonomous vehicle based on the deep reinforcement learning longitudinal driving decision model also increases. At this time, a better strategy is learned through the exploration ability of DRL.
[0062] 104. When the risk assessment result is greater than or equal to the global risk threshold, the real-time status data is input into a longitudinal driving decision model based on collision probability prediction, and a longitudinal control decision of the autonomous vehicle is output; the autonomous vehicle executes the longitudinal control instruction based on the collision probability prediction;
[0063] In an embodiment of the present invention, the real-time status data is input into a longitudinal driving decision model based on collision probability prediction, and the longitudinal control decision of the autonomous vehicle is output, including obtaining the coordinates and speed of the autonomous vehicle and the risk assessment results of related vehicles, obtaining the collision area between the autonomous vehicle and the related vehicles through a trained neural network model or a probability prediction model, and outputting a driving instruction for the autonomous vehicle to stop before the collision area, pass through an intersection, or pass through the collision area; and the longitudinal control decision of the autonomous vehicle is output according to the driving instruction.
[0064] Among them, the longitudinal driving decision model based on collision probability prediction can be a classifier model such as logistic regression, decision tree, random forest, support vector machine (SVM), naive Bayes classifier, etc., or a neural network model, such as a convolutional neural network model, a long short-term memory network model, etc., or a gray prediction model and a Bayesian model, etc. These models can be used to predict whether the future driving path of the surrounding vehicles conflicts with the future driving path of the autonomous vehicle, to evaluate the wind direction level of the relevant vehicles, and thus obtain the longitudinal control decision of the autonomous vehicle.
[0065] In the embodiment of the present invention, this embodiment also defines a set of high-level driving instructions, namely "stop before the collision area", "pass the intersection" and "pass the collision area". After obtaining the high-level driving instructions, the longitudinal control of the autonomous vehicle is generated by a proportional integral differential (PID) controller. The process may include: defining a set C representing all areas where collisions may occur. Initialize the set C to an empty set, for each collision area, if the risk level of the most recent collision area that is about to arrive is D, then return to stop before the collision area; if the risk level of the most recent collision area that is about to arrive is not D, then return to pass through the collision area; for each collision area, if the ego vehicle has passed the collision area, then remove the collision area from the set C.
[0066] 105. When the risk assessment result is less than the global risk threshold, the real-time status data is input into a longitudinal driving decision model based on deep reinforcement learning, and the longitudinal control decision of the autonomous vehicle is output; the autonomous vehicle executes the longitudinal control instruction based on deep reinforcement learning.
[0067] In the embodiment of the present invention, in order to better implement the longitudinal control decision of the autonomous vehicle through the longitudinal driving decision model of deep reinforcement learning, the embodiment of the present invention models the driving decision of the unsignalized intersection as a Markov decision process:
[0068] (1) State space:
[0069] S={S a ,S1,S2,…,S n}
[0070] Among them, S a represents the state of the autonomous vehicle, S i represents the state of the surrounding vehicles, and n represents the number of observed surrounding vehicles. Specifically:
[0071]
[0072] Among them, i represents the number of the vehicle, s x and y They represent the coordinates of the vehicle in two-dimensional space, and the speed of the vehicle in the x-axis direction and y-axis direction are represented by v and x and v y express.
[0073] (2) Action Space:
[0074] Since the vehicle's steering is implemented by a pure tracking algorithm, DRL only controls the longitudinal behavior of the autonomous vehicle. Therefore, the action space is designed as:
[0075] A=[throttle,brake]
[0076] Among them, throttle represents the throttle size, brake represents the brake size, throttle,brake∈[0,1].
[0077] In an embodiment of the present invention, the TD3 algorithm can be used to solve the Markov decision process, and demonstrations are used to guide the learning of the autonomous vehicle in this process;
[0078] The driving policy is updated based on the driving experience data generated by the demonstration action and the action based on deep reinforcement learning. The off-policy type of TD3 algorithm enables the autonomous vehicle to learn the policy from any action. Both the demonstration action and the DRL-based action will be stored in the experience pool. Then the parameters of TD3 are updated by random sampling.
[0079] Both demonstration actions and DRL-based actions are stored in the experience pool. Then the parameters of TD3 are updated by random sampling:
[0080]
[0081] in is the loss function about gradient. Q-value is about α is the learning rate of the value network. is the objective function J(θ μ ) about θ μ gradient. is the Q-value about a t gradient. is the strategy μ with respect to θ μ gradient.
[0082] Based on the above embodiments, the present invention alleviates the problem of deep reinforcement learning methods' dependence on reward shaping in the task of driving decision-making at unsignaled intersections, and provides a more reliable and efficient strategy optimization solution. The introduction of demonstrations also improves the stability of the algorithm, bringing important improvements to the driving decision-making of autonomous vehicles.
[0083] A person skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer-readable storage medium, which can include: ROM, RAM, disk or CD, etc.
[0084] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.
Claims
1. A method for automatic driving decision-making at an unsignalized intersection, characterized in that: The method comprises: Obtain the target lane centerline, real-time status data and historical trajectory data; the target lane centerline is the position set of the target lane center point; the real-time status data includes the state of the autonomous vehicle and the state of surrounding vehicles, and the state is the coordinates and speed in two-dimensional space; the historical trajectory data is the position set at each time; Inputting the real-time status data into a lateral driving decision model based on a tracking algorithm to output a lateral control decision of the autonomous vehicle; Based on the similarity distance between the center line of the target lane and the historical trajectory data, the surrounding vehicles are divided into relevant vehicles and non-relevant vehicles; the risk of the relevant vehicles is evaluated based on the entry time of the relevant vehicles; When the risk assessment result is greater than or equal to the global risk threshold, the real-time status data is input into a longitudinal driving decision model based on collision probability prediction, and a longitudinal control decision of the autonomous vehicle is output; the autonomous vehicle executes the longitudinal control instruction based on the collision probability prediction; When the risk assessment result is less than the global risk threshold, the real-time status data is input into a longitudinal driving decision model based on deep reinforcement learning, and a longitudinal control decision of the autonomous vehicle is output; the autonomous vehicle executes the longitudinal control instruction based on deep reinforcement learning.
2. The automatic driving decision-making method for an unsignalized intersection according to claim 1, characterized in that: The real-time state data is input into a lateral driving decision model based on a tracking algorithm, and the lateral control decision of the autonomous vehicle is output, including obtaining the coordinates and speed of the autonomous vehicle; drawing a circle with the center of mass of the rear wheel of the vehicle as the center and the foresight distance as the radius, and the point intersecting with the planned path is the preview point; Based on the geometric relationship between the pre-mapped points, the center of mass of the vehicle's rear wheels and the vehicle's steering angle, the front wheel steering angle is derived using trigonometric functions and a bicycle model; the calculated front wheel steering angle is used to control the vehicle to perform steering movement and execute lateral control decisions.
3. The automatic driving decision-making method for an unsignalized intersection according to claim 1, characterized in that: Based on the similarity distance between the target lane centerline and the historical trajectory data, the surrounding vehicles are divided into relevant vehicles and non-relevant vehicles, including calculating the similarity distance between the target lane centerline and each historical trajectory data of the surrounding vehicles, and determining the lane centerline on which the surrounding vehicles are most likely to travel from each historical trajectory data; and the relevant vehicles and non-relevant vehicles are divided according to whether the lane centerline on which the surrounding vehicles are most likely to travel and the target lane centerline are repeated, if they overlap, they are relevant vehicles, and if they do not overlap, they are non-relevant vehicles.
4. The automatic driving decision-making method for an unsignalized intersection according to claim 3, characterized in that: The similarity distance is determined based on the Hausdorff distance, and the historical trajectory data having the smallest Hausdorff distance to each historical trajectory data of surrounding vehicles is used as the most likely lane centerline.
5. The automatic driving decision-making method for an unsignalized intersection according to claim 1, characterized in that: The real-time status data is input into a longitudinal driving decision model based on collision probability prediction, and the longitudinal control decision of the autonomous vehicle is output, including obtaining the coordinates and speed of the autonomous vehicle and the risk assessment results of related vehicles, obtaining the collision area between the autonomous vehicle and the related vehicles through a trained neural network model or a probability prediction model, and outputting a driving instruction for the autonomous vehicle to stop before the collision area, pass through the intersection, or pass through the collision area; and the longitudinal control decision of the autonomous vehicle is output according to the driving instruction.
6. The automatic driving decision-making method for an unsignalized intersection according to claim 1, characterized in that: Assessing the risk of the relevant vehicle based on its entry time includes: in, Represents the relevant vehicle r at time t i j About the collision area c j The probability of different risk levels under a given TTE, Represents the relevant vehicle r at time t i j About the collision area c j The risk level Represents the relevant vehicle r at time t i j About the collision area c j TTE value; Represents the relevant vehicle r at time t i j About the collision area c j The probability of different TTEs at a given risk level, P(z) is the prior probability of each risk level, z represents the risk level, and Z represents the number of risk levels; Represents the relevant vehicle r at time t i About the collision area c j The TTE value, Represents the relevant vehicle r at time t i speed, Represents the relevant vehicle r at time t i To the collision area c j distance; TTE means time required to enter.
7. The automatic driving decision-making method for an unsignalized intersection according to claim 1, characterized in that: The global risk threshold is a variable value, whose initial value is 0, and whose iteration value gradually increases with the increase of the number of iterations.
8. The automatic driving decision-making method for an unsignalized intersection according to claim 1, characterized in that: The reward function in the longitudinal driving decision model based on deep reinforcement learning is a sparse reward; if the autonomous vehicle successfully passes through the intersection, it obtains a reward of +1; if the autonomous vehicle collides during driving, it obtains a reward of -2; if the autonomous vehicle stays in the intersection for more than a specified time, it obtains a reward of -2.
