Intelligent automobile roundabout driving decision-making method based on behavior risk reinforcement learning
By designing a reinforcement learning method based on behavioral risks in intelligent cars, building an island-round scene model and self-learning behavioral risk classifier, combining multi-layer perception machine and strategy entropy optimization, the safety of intelligent cars' driving strategies in complex traffic scenarios is solved. and convergence issues have achieved safer and more efficient island driving decisions.
Patent Information
- Application Number
- CN202411047023.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-01
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-08-01
AI Technical Summary
Smart cars face high complexity and strong interactivity problems when performing driving tasks in complex urban traffic scenarios such as island-round scenarios. The existing deep reinforcement learning algorithms are insufficient in terms of security and generalization capabilities. Inadequate reward function design leads to failure to make the strategy impossible. Converge or fall into local optimality.
A reinforcement learning method based on behavioral risk was designed. By building an island-round scene model, building a self-learning behavioral risk classifier and multi-objective reward function, combining multi-layer perceptron for behavioral risk assessment and strategy optimization, policy entropy and information divergence are introduced. Improve the direction of strategy iteration and reduce the impact of insufficient reward function design.
The security decision-making and strategy optimization of smart cars in the island-round scenarios are realized, the security and convergence speed of reinforcement learning algorithms are improved, the training process is simplified, the dependence on labeled data is reduced, and the security and efficiency of the strategy are improved.
Smart Images

Figure CN118991769A_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the field of automobile technology, and specifically is a smart automobile roundabout driving decision-making method based on behavioral risk reinforcement learning. Background Art
[0002] As an important part of complex urban traffic, roundabouts play an important role in diverting vehicles, reducing vehicle conflicts, and improving vehicle driving efficiency. Compared with simple driving conditions such as longitudinal cruise following and lateral high-speed lane changing, smart cars face high complexity and strong interactivity when performing dynamic and diverse driving tasks such as merging, driving around the island, and driving out in roundabouts, which poses a major challenge to the safety performance of smart car driving decisions.
[0003] Deep reinforcement learning combines the feature extraction capabilities of deep neural networks and the advantages of reinforcement learning environment interaction. It is considered to be an effective solution for the design of intelligent car driving decision strategies and has been widely used in simple urban traffic conditions such as lane keeping, lane changing and adaptive cruise. However, before applying deep reinforcement learning to car driving decisions, it is necessary to reasonably model the environment to improve the quality of strategy training. However, the above-mentioned simple urban traffic condition research lacks comprehensive consideration of the vehicle's lateral and longitudinal coupled action space, dynamic and diverse driving tasks, complex driving condition constraints, strong vehicle interactions and other issues, resulting in certain safety risks when they are directly applied to intelligent car driving decisions in complex urban traffic scenarios such as roundabouts.
[0004] In addition, the reward function is crucial to the optimization and convergence process of deep reinforcement learning-based automobile driving decision-making strategy training. The reward function of complex urban traffic scenes artificially designed based on expert experience may have certain experience blind spots, which may cause the automobile driving decision-making strategy based on deep reinforcement learning to fail to converge or fall into local optimality. Therefore, in recent years, methods such as imitation learning and inverse reinforcement learning have been applied in the design of automobile driving decision-making strategies in complex urban traffic scenes. However, imitation learning is highly dependent on effective expert demonstration data, and its generalization ability for scenes not covered by data is weak; inverse reinforcement learning also relies on expert demonstration data, and the heterogeneous driving characteristics of different experts will also cause the fitted reward function to deviate or be difficult to converge. Recently, reinforcement learning methods based on human preferences, such as human feedback reinforcement learning, have been widely used in the field of large language models. Its core idea is to compare and learn based on superior and inferior strategies, so that it can output superior strategies rather than inferior strategies as much as possible. Therefore, reinforcement learning methods that understand human preferences can also be applied to driving decisions, helping smart cars to output safer driving actions in complex traffic scenes. Among them, human preferences are characterized by establishing an objective reward function. However, the training of this reward function requires a large amount of driving decision data, which to some extent hinders the application of reinforcement learning algorithms based on human preferences in complex urban traffic scenarios. Summary of the invention
[0005] To solve the above problems, the present invention provides an intelligent car roundabout driving decision-making method based on behavioral risk reinforcement learning, which can adapt to the complex interactive scenes of the roundabout, ensure safe decision-making during the training process of the deep reinforcement learning algorithm, and realize the rapid implementation of the reinforcement learning algorithm.
[0006] The technical solution of the present invention is described as follows in conjunction with the accompanying drawings:
[0007] A method for making decisions about driving a roundabout for an intelligent vehicle based on reinforcement learning, comprising the following steps:
[0008] Step 1: Build a roundabout scene model;
[0009] Step 2: Build a self-learning behavior risk classifier.
[0010] Step 3: Model the behavioral risk reinforcement learning driving decision algorithm to enable smart cars to make decisions on roundabout driving.
[0011] Furthermore, the specific method of step one is as follows:
[0012] 11) Design action space;
[0013] 12) Design state space;
[0014] 13) Design a multi-objective reward function for intelligent cars.
[0015] Further, the specific method of step 11) is as follows:
[0016] Combine intelligence and drive into the normalized signal p required for simulating vehicle longitudinal acceleration and deceleration control x ∈[-1, 1], and the front wheel active steering signal δ e ∈[-1, 1] to construct the horizontal and vertical coupling action space A of the intelligent vehicle e :
[0017] A e =[p ex ,δ e ] (1).
[0018] Further, the specific method of step 12) is as follows:
[0019] Design a driving task vector V consisting of key tasks and driving tasks T ,as follows:
[0020]
[0021] In the formula, ΔP ENDis the coordinate of the vehicle’s current position relative to the end point; and are the path points P of the vehicle relative to the front i The distance and orientation deviation, P i Progressive as the vehicle moves forward;
[0022] Design the surrounding environment state vector V consisting of surrounding vehicle information E ; Through the region cutting coding method, the surrounding environment of the vehicle is divided into P 0 ~P 6 7 detection intervals; P 0 , P 1 , P 4 , P 5 The high interaction detection interval is expanded; when the external perception module detects other vehicles with the same driving direction as the self-vehicle and whose center point projection is located in the detection interval, it is marked as the perceived vehicle, and the corresponding vehicle information V i It is expressed as:
[0023] V i =[F,ΔL,d -1 ,Δv,δθ,I],i={0,...,6}, (3)
[0024] Where F∈{0,1} is the flag of whether there is a vehicle; ΔL,d,Δv=[Δv x , Δv y ] and δθ are the relative lane offset, distance, speed and relative orientation between the other vehicle and the vehicle itself; I∈[-1, 1] is the possibility of the other vehicle changing lanes;
[0025] When there is no other vehicle in the detection range, the corresponding setting Finally, we get the surrounding environment vector V E for:
[0026] V E =[V 0 ,V 1 ,V 2 ,V 3 ,V 4 ,V 5 ,V 6 ]. (4)
[0027] Design includes the vehicle speed information v e =[v ex , v ey ], acceleration information a e =[a ex , a ey ], driving direction information e =[cosθe ,sinθ e ] and yaw rate information ω e The vehicle state vector, with the future target point preview information added To comprehensively establish the state vector V e :
[0028] V e =[v e ,a e ,h e ,ω e ,Δθ e ,A e ], (5)
[0029] In the formula, θ e is the heading angle of the vehicle; is the angle difference between the directions of the three future target points and the heading angle of the vehicle;
[0030] In summary, the state space is formed:
[0031] S=[V T ,V e ,V E ] T . (6).
[0032] Further, the specific method of step 13) is as follows:
[0033] Design collision reward function r c for:
[0034]
[0035] In the formula, min{d i} is the shortest distance between all vehicles and the self-vehicle in the detection interval; D c is the dangerous collision distance threshold;
[0036] Through vertical and horizontal safety rewards f and r l Boot as follows:
[0037] r f =-0.5|min(0,Δv fx )|(D f -d f ), (8)
[0038] r l =-2.5|min(0,Δv ky )|(D l -d k ) 2 , (9)
[0039] In the formula, Δv fx and d f ∈(D c , D f ) is the relative longitudinal speed and distance between the front vehicle and the vehicle; Δv ky and d k ∈(D c , D l ) is the relative lateral speed and distance between the nearest vehicle next to the vehicle and the vehicle; D f and D l The longitudinal and lateral safe vehicle distances;
[0040] Design roundabout entrance safety reward RE , as shown below:
[0041] r RE =-0.25||Δv i ||(D RE -d i ), (10)
[0042] Where, d i ∈(D c , D RE ) and Δv i are the distance and relative speed vector between the ego vehicle and the high-risk vehicle respectively; D RE Safe distance for high-risk interactions;
[0043] The security reward function is obtained as follows:
[0044] R S =r c +r f +r l +r RE , (11)
[0045] Create a reward for reaching the end point r succ , as shown below:
[0046] r succ =100,||ΔP End ||≤D P , (12)
[0047] Where D P is the distance threshold between the vehicle and the path point P;
[0048] Introducing path tracking reward r tag , as shown below:
[0049]
[0050] In the formula, is the driving distance of the vehicle between the two target points;
[0051] Build Speed Bonus sp , as shown below:
[0052]
[0053] Design Course Rewards δθ Guide the vehicle to move in the specified road direction as shown below:
[0054]
[0055] Where V lim is the road speed limit; δθ e is the deviation between the vehicle heading and the road direction; Θ δθ1 , Θ δθ2 is the vehicle driving direction deviation threshold;
[0056] Design the driving task reward function as follows:
[0057] R T =r succ +r tag +r δθ +r sp (16)
[0058] Design Turn Rewards st , as shown below:
[0059] r st =-2|δ e |v x (17)
[0060] Designing Lane Keeping Rewards lk , as shown below:
[0061]
[0062] Where δd is the lateral offset of the current position of the main vehicle relative to the center line of the road; L is the road width; when the vehicle crosses the road boundary, that is, |δd|≥L, r lk =-20 ends the current round and resets the environment;
[0063] Define the control command reward r control , as shown below:
[0064] r control =-4||A cur -A per || 2 (19)
[0065] In the formula, A per , Acur are the action vectors at the previous and next moments respectively;
[0066] Establishing yaw comfort bonus ω , guiding the vehicle's yaw rate to change smoothly, as shown below:
[0067] r ω =-((ω-Ω ω ) / 5) 2 ,Ω ω <ω (20)
[0068] In the formula, Ω ω is the yaw angular velocity threshold;
[0069] Design driving behavior reward function R C , as shown below:
[0070] R C =r lk +r ω +r st +r control (twenty one)
[0071] Finally, the multi-objective reward function of the intelligent car is obtained as follows:
[0072] R=R S +R T +R C (twenty two).
[0073] Furthermore, the specific method of step 2 is as follows:
[0074] 21) Construct a multi-layer perceptron with inputs of state space s and action space a, and outputs the probabilities of three behavioral risk categories, namely safe, low risk, and high risk;
[0075] 22) Train the classifier.
[0076] Further, the specific method of step 22) is as follows:
[0077] 1) Data collection and labeling: Based on the existing roundabout decision-making strategy, collect roundabout driving simulation data; randomly sample state-action pairs {s, a} from the collected data, label them with behavioral risk categories, and evaluate the risk of their driving decision-making behaviors; the three behavioral risks are judged based on the following criteria: high-risk behaviors refer to behaviors that will continue to deteriorate driving safety after being executed in dangerous conditions such as collision, lane departure, or high-speed turning; safe behaviors refer to behaviors such as following or changing lanes under safe conditions, or behaviors that avoid further deterioration of the conditions under dangerous conditions; low-risk behaviors refer to behaviors that will deteriorate driving safety after being executed in safe conditions;
[0078] 2) Pre-training: Use labeled data to pre-train the classifier based on supervised learning, so that it has the initial ability to distinguish different risk categories;
[0079] 3) Self-learning training: In the prediction phase, the classifier first predicts the unlabeled roundabout data; for high-confidence screening, for those state-action pairs with risk category prediction confidence exceeding 90%, they are labeled as the corresponding prediction category and added to the next round of training sets. The training cycle repeats the above self-learning process to continuously update the classifier;
[0080] 4) Termination of training: When the classification accuracy of the classifier on the test set no longer improves, the training ends;
[0081] A joint loss consisting of focal loss and label distribution-aware margin loss is used to deal with the problems of class imbalance and difficult sample mining in the pre-training and self-learning training process. The joint loss function is constructed as follows:
[0082]
[0083] In the formula, γ is the hyperparameter for adjusting the weight factor; p y ∈[0,1] is the predicted value; α y =3∑ j n j / n y is the weight parameter for balancing the number of samples, where n j is the number of samples in each category;
[0084] Design the parameter α related to the category proportion in the loss function y , n y As the training changes dynamically, the loss function is adaptively adjusted.
[0085] Furthermore, the specific method of step three is as follows:
[0086] 31) By introducing policy entropy To encourage the policy to produce more diverse exploration behaviors, as follows:
[0087]
[0088] In the formula, π φ is the strategy function;
[0089] 32) Reinforcement learning algorithm considers policy entropy through Bellman regression operator and temporal difference pair The Q-value function is iteratively solved as follows:
[0090]
[0091] The cost function of the algorithm value network is defined as
[0092]
[0093] Where s, a, r, s′ are the current state, action, reward and next state of the sample respectively; α is the entropy weight coefficient; are the value network and the target network respectively; φ is the strategy network;
[0094] Under the constraints Under this condition, the entropy weight coefficient α is automatically adjusted to balance the strategy entropy The ratio between the expected return and the design cost function of the hyperparameter α is as follows:
[0095]
[0096] In the formula, is the entropy constraint threshold;
[0097] 33) The iteration objective is designed to minimize the information divergence between the policy function π and the Q-value function, that is, to maximize the expected reward and policy entropy at the same time As shown below:
[0098]
[0099] In the formula, π φ (s) = f φ (s, ξ) is the sampling value of the strategy distribution under the reparameterized skill; is the noise that obeys the standard normal distribution; K(s) is the partition function and does not participate in the strategy optimization;
[0100] 34) Using the state-action dataset, the reward network is optimized iteratively by maximizing the expected reward difference between good and bad behaviors under the same input, so that the reward network has the ability to accurately evaluate good and bad strategies. The cost function is as follows:
[0101]
[0102] In the formula, r ψ To reward the network; a + , a - are the superior and inferior behaviors corresponding to the same state s; σ is the sigmoid function;
[0103] 35) Information divergence is added to limit the deviation between the new strategy and the original strategy. The cost function is defined as:
[0104]
[0105] In the formula, π φ is the policy model for training; π ref is the initial model; β is the parameter that controls the degree of deviation between the two strategies; Z = ∑ a π ref (a|s)exp(r(s,a) / β) is the partition function, which is related only to the states s and π ref is related to the optimized strategy π φ Not relevant;
[0106] Strategy π φ The optimization goal is equivalent to minimizing the difference between the numerator and the denominator. * Defined as:
[0107]
[0108] Further substituting formula (31) into formula (30), we get the strategy π φ The optimization goal is to minimize the training strategy π φ With the optimal strategy π * The deviation between the distributions is as follows:
[0109]
[0110] From formula (30), we get the reward network r ψ and the optimal strategy π * The relationship between them is:
[0111]
[0112] 36) Use the relationship between the reward network and the optimal strategy to eliminate the reward network r in formula (28) ψ , and based on the maximum likelihood optimization idea and the strategy π of formula (32) φ Optimize the objective by replacing the optimal policy with policy π φ , to directly optimize strategy π using human-annotated data φ The purpose is to maximize the probability difference between the strategy outputting good and bad behaviors under the same state, as shown below:
[0113]
[0114] By evaluating the risk of sampled driving behaviors, we can improve the direction of policy iteration and reduce the impact of insufficient reward function design on policy optimization, as shown below:
[0115]
[0116] In the formula, π φ′ is π φ The objective function of +, a - They are respectively the advantages and disadvantages of behavior risk classification, and the target strategy function π φ′ Two random samplings are performed, and risk classification is performed through the behavioral risk classifier. The lower the driving risk, the better the behavior; β′=β+(C - -C + ) / 4 is the behavioral risk category difference coefficient; C∈{0, 1, 2} corresponds to the three risk behavior labels of safe, low-risk and high-risk, respectively.
[0117] The beneficial effects of the present invention are:
[0118] 1) The present invention designs a roundabout scenario model, which includes the vehicle's lateral and longitudinal coupled action space, a multi-scale information state space consisting of driving task-surrounding environment-self-vehicle information, and a multi-objective reward function considering safety-task-driving behavior. It can fully characterize the characteristics of roundabout driving, so that reinforcement learning can fully understand and process complex roundabout interaction scenarios
[0119] 2) Based on the theory of human preference reinforcement learning, the present invention designs behavioral risk reinforcement learning that can be applied to the field of driving decision-making, and compensates for the inadequate reward function design in the reinforcement learning training process by introducing behavioral risk classification optimization objectives;
[0120] 3) The present invention derives the behavioral risk optimization objective function based on reward training and strategy training, realizes the simplification from the two-stage training process to the single-stage training, and directly uses the behavioral risk data to optimize the strategy iteration direction;
[0121] 4) In order to achieve parallel and accurate evaluation of the relative advantages and disadvantages of sampled behaviors during the strategy optimization process, the present invention adopts a behavior risk classifier based on three behavior risk categories: safe, low-risk, and high-risk;
[0122] 5) In order to realize the behavioral risk classifier and reduce the intensity and difficulty of labeling driving risk data, a self-learning based classifier training strategy is designed, which can complete the classifier training task based on multi-layer perceptron with only a small amount of labeled data. BRIEF DESCRIPTION OF THE DRAWINGS
[0123] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.
[0124] Figure 1 Schematic diagram of the working principle of the present invention
[0125] Figure 2a Schematic diagram of surrounding environment detection when merging into a roundabout;
[0126] Figure 2b This is a schematic diagram of the surrounding environment detection when passing through the roundabout entrance;
[0127] Figure 3 Schematic diagram of training a self-learning behavioral risk classifier;
[0128] Figure 4 This is a schematic diagram of the classifier architecture based on a multi-layer perceptron;
[0129] Figure 5 Schematic diagram of various reinforcement learning training reward curves. DETAILED DESCRIPTION
[0130] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0131] Embodiment 1
[0132] See also Figure 1 The present invention provides a method for making decisions about roundabout driving for intelligent vehicles based on behavioral risk reinforcement learning. First, the vehicle's lateral and longitudinal coupling action space, a multi-scale state space containing driving tasks, surrounding environment, and self-vehicle information, and a multi-objective reward function that considers safety, tasks, and driving behaviors are analyzed and established. Secondly, by applying a self-learning behavioral risk classifier of a multi-layer perceptron, the behavior risk of the intelligent agent is classified and assessed, and a strategy optimization target based on behavioral risk classification results is designed to optimize the reinforcement learning driving decision algorithm, so as to solve the impact of insufficient reward function design on driving strategy optimization, and guide driving decisions to converge in a safer direction. Specifically, the following steps are included:
[0133] Step 1: Build a roundabout scene model, as follows:
[0134] 11) Design action space;
[0135] In order to ensure that the intelligent car can complete the driving task in the roundabout scene, precise control of driving, steering and braking is required. Considering that the simultaneous output of driving and braking by the intelligent agent will reduce the comfort of the longitudinal motion of the car, the two are combined into the normalized signal p required for the longitudinal acceleration and deceleration control of the simulated vehicle. x ∈[-1, 1], and the front wheel active steering signal δ e∈[-1, 1] to construct the horizontal and vertical coupling action space A of the intelligent vehicle e :
[0136] A e =[p ex ,δ e ] (1)
[0137] 12) Design state space;
[0138] The state vector designed by the present invention is composed of vectors of three scales: driving task, surrounding environment and vehicle information, which fully restores the high complexity and strong interactivity problems faced by smart cars when performing dynamic and diverse driving tasks such as merging, driving around the island and driving out in roundabout scenarios.
[0139] The smart car needs to complete the driving task in the roundabout under the guidance of the planned global path and reach the planned destination smoothly. Therefore, the driving task vector V T Designed to consist of terminal tasks and driving tasks:
[0140]
[0141] In the formula, ΔP END is the coordinate of the vehicle’s current position relative to the end point; and are the path points P of the vehicle relative to the front i The distance and orientation deviation, P i Progressive as the vehicle moves forward;
[0142] The surrounding state vector V E It is composed of surrounding vehicle information. Figure 2a and Figure 2b The region cutting coding method shown in the figure divides the surrounding environment of the vehicle into P 0 ~P 6 In order to more effectively identify other vehicles competing for road rights with the vehicle, some high-interaction detection intervals are expanded, such as Figure 2a P 0 ,P 1 and Figure 2b P 4 , P 5 When the external perception module detects other vehicles that are traveling in the same direction as the vehicle and whose center point projection is within the detection interval, it marks them as sensed vehicles and the corresponding vehicle information V i It can be expressed as:
[0143] V i =[F,ΔL,d -1 ,Δv,δθ,I],i={0,...,6}, (3)
[0144] Where F∈{0,1} is the flag of whether there is a vehicle; ΔL,d,Δv=[Δv x , Δv y ] and δθ are the relative lane offset, distance, speed and relative orientation between the other vehicle and the vehicle itself; I∈[-1,1] is the possibility of the other vehicle changing lanes;
[0145] When there is no other vehicle in the detection range, the corresponding setting Finally, we get the surrounding environment vector V E for:
[0146] V E =[V 0 ,V 1 ,V 2 ,V 3 ,V 4 ,V 5 ,V 6 ]. (4)
[0147] The vehicle state vector V e Mainly includes the vehicle speed information v e =[v ex , v ey ], acceleration information a e =[a ex , a ey ], driving direction information e =[cosθ e , sinθ e ] and yaw rate information ω e Considering that the target trajectory needs to continuously guide the driving direction of the vehicle, the future target point preview information is added. To comprehensively establish the state vector V e :
[0148] V e =[v e ,a e ,h e ,ω e ,Δθ e ,A e ], (5)
[0149] In the formula, θ e is the heading angle of the vehicle; is the angle difference between the directions of the three future target points and the heading angle of the vehicle;
[0150] In summary, the state space is formed:
[0151] S=[V T ,V e,V E ] T . (6).
[0152] 13) Design multi-objective reward function for intelligent vehicles;
[0153] In order to promote smart cars to perform dynamic and diverse driving tasks in roundabout scenarios safely, efficiently and stably, the present invention designs a multi-objective reward function that considers safety, tasks and driving behavior.
[0154] First, the smart car should avoid collisions with other cars as much as possible when driving in a roundabout. Therefore, the collision reward function r is designed. c for:
[0155]
[0156] In the formula, min{d i} is the shortest distance between all vehicles and the self-vehicle in the detection interval; D c is the dangerous collision distance threshold.
[0157] Secondly, the vehicle should always maintain a safe distance from surrounding vehicles during driving to deal with emergencies such as emergency braking of the vehicle in front or lane change of the vehicle next to it. f and r l Guide
[0158] r f =-0.5|min(0,Δv fx )|(D f -d f ), (8)
[0159] r l =-2.5|min(0,Δv ky )|(D l -d k ) 2 , (9)
[0160] In the formula, Δv fx and d f ∈(D c , D f ) is the relative longitudinal speed and distance between the front vehicle and the vehicle; Δv ky and d k ∈(D c , D l ) is the relative lateral speed and distance between the nearest vehicle next to the vehicle and the vehicle; D f and D l The longitudinal and lateral safe vehicle distances;
[0161] Figure 2bIn the high-risk area of the roundabout entrance, the high-risk vehicles competing with the vehicle for the right of way are far more threatening to the vehicle than other vehicles traveling in the same direction. R To improve the ability of the vehicle to deal with high-risk vehicles;
[0162] r RE =-0.25||Δv i ||(D RE -d i ), (10)
[0163] Where, d i ∈(D c , D RE ) and Δv i are the distance and relative speed vector between the ego vehicle and the high-risk vehicle respectively; D RE Safe distance for high-risk interactions;
[0164] In summary, the safety reward function can be obtained as:
[0165] R S =r c +r f +r l +r RE , (11)
[0166] Smart cars need to reach the designated destination according to the driving task requirements. To this end, a reward r is established for reaching the destination. succ :
[0167] r succ =100,||ΔP End ||≤D P , (12)
[0168] Where D P is the distance threshold between the vehicle and the path point P;
[0169] In order to drive the intelligent car to follow the global planning path as much as possible, the path tracking reward r is introduced. tag :
[0170]
[0171] In the formula, is the driving distance between the two target points; considering the efficient driving requirements of smart cars, the speed reward r is established sp :
[0172]
[0173] On this basis, the heading reward r is designed δθGuiding the vehicle to move in the specified road direction can be expressed as:
[0174]
[0175] Where V lim is the road speed limit; δθ e is the deviation between the vehicle heading and the road direction; Θ δθ1 , Θ δθ2 is the vehicle driving direction deviation threshold;
[0176] Finally, the driving task reward function is:
[0177] R T =r succ +r tag +r δθ +r sp (16)
[0178] When driving in a roundabout, the car should try to reduce frequent turning behaviors and stay in the center of the road as much as possible to avoid blocking other traffic vehicles. Therefore, the turning reward r is designed. st :
[0179] r st =-2|δ e |v x (17)
[0180] and lane keeping reward r lk :
[0181]
[0182] Where δd is the lateral offset of the current position of the main vehicle relative to the center line of the road; L is the road width; when the vehicle crosses the road boundary, that is, |δd|≥L, r lk =-20 ends the current round and resets the environment;
[0183] During the driving process of the intelligent car, its action output should maintain a certain continuity to reduce the sudden change of action at adjacent moments. The control command reward r is defined. control for:
[0184] r control =-4||A cur -A per || 2 (19)
[0185] In the formula, A per , A cur are the action vectors at the previous and next moments respectively;
[0186] When the smart car makes necessary turns, it is necessary to establish a yaw comfort reward rω , guiding the vehicle's yaw rate to change smoothly:
[0187] r ω =-((ω-Ω ω ) / 5) 2 ,Ω ω <ω (20)
[0188] In the formula, Ω ω is the yaw angular velocity threshold;
[0189] Therefore, the driving behavior reward function R C for:
[0190] R C =r lk +r ω +r st +r control (twenty one)
[0191] Through the above design, we finally get the multi-objective reward function of the smart car:
[0192] R=R S +R T +R C (twenty two).
[0193] Step 2: Build a self-learning behavior risk classifier, as follows:
[0194] In order to achieve the strategy optimization goal of the behavior risk reinforcement learning algorithm and reduce the difficulty of labeling driving decision data, a self-learning behavior risk classifier based on a multi-layer perceptron was constructed; this classifier is used to evaluate the three behavior risk categories of safe, low-risk or high-risk driving decision behaviors. The specific training process is as follows Figure 3 As shown, it includes the following processes: data collection and annotation, classifier pre-training, unlabeled data prediction and classifier self-learning training;
[0195] The behavior risk classifier is constructed by a multi-layer perceptron. Its input is the state space s and the action space a. Its output is the probability of three behavior risk categories (safe, low risk, high risk). Its architecture is as follows: Figure 4 As shown in the figure, the specific training process of the classifier is as follows:
[0196] 1) Data collection and labeling: Based on the existing roundabout decision-making strategy, collect roundabout driving simulation data. Randomly sample state-action pairs {s, a} from the collected data, and label the behavior risk to evaluate the risk of its driving decision-making behavior. The three types of behavior risks are judged based on: high-risk behavior refers to behavior that will continue to deteriorate driving safety after execution under dangerous conditions such as collision, lane departure or high-speed turning; safe behavior refers to behavior such as following and changing lanes under safe conditions or behavior that is executed to avoid further deterioration of the working conditions under dangerous conditions; low-risk behavior refers to behavior that will deteriorate driving safety after execution under safe conditions.
[0197] 2) Pre-training: Use labeled data to pre-train the classifier based on supervised learning, so that it has the initial ability to distinguish different risk categories;
[0198] 3) Self-learning training: In the prediction phase, the classifier first predicts the unlabeled roundabout data. For those state-action pairs with a risk category prediction confidence of more than 90%, they are labeled as the corresponding prediction category and added to the next round of training sets; loop training, repeat the above self-learning process, and continuously update the classifier.
[0199] 4) Termination of training: When the classification accuracy of the classifier on the test set no longer improves, the training is terminated.
[0200] In the normal roundabout driving decision-making process, the frequency of high-risk behaviors is much lower than that of safe behaviors, resulting in an extreme imbalance between the number of categories, thereby affecting the network's ability to extract and learn features of a small amount of data categories. Therefore, the present invention uses a joint loss consisting of focal loss and label distribution-aware margin loss to deal with the problem of category imbalance and difficult sample mining during pre-training and self-learning training. The joint loss function is:
[0201]
[0202] In the formula, γ is the hyperparameter for adjusting the weight factor, p y ∈[0, 1] is the predicted value, α y =3∑ j n j / n y is the weight parameter for balancing the number of samples, where n j is the number of samples in each category. At the same time, as the self-learning training progresses, newly labeled data will be added to each round of training data, resulting in changes in the proportion of each category. Therefore, the parameter α related to the category proportion in the design loss function is y , n yAs the training changes dynamically, the loss function is adaptively adjusted, and the features of a small number of data categories are effectively extracted and the classification accuracy is improved.
[0203] Step 3: Model the behavioral risk reinforcement learning driving decision algorithm as follows:
[0204] 31) The present invention adopts a soft actor-critic reinforcement learning algorithm with high exploration efficiency and strong training convergence to generate driving decisions of intelligent vehicles including steering and acceleration and deceleration; the soft actor-critic is a reinforcement learning algorithm based on the Actor-Critic framework for policy iteration, which introduces the policy entropy H to stimulate the strategy to generate more diverse exploration behaviors:
[0205]
[0206] In the formula, π φ is the strategy function;
[0207] 32) The solution process of the reinforcement learning algorithm mainly includes two parts: strategy evaluation and strategy optimization. In the strategy evaluation stage, the reinforcement learning algorithm iteratively solves the Q-value function considering the strategy entropy H through the Bellman regression operator and time series difference, which can be expressed as:
[0208]
[0209] Correspondingly, the cost function of the algorithm value network is defined as:
[0210]
[0211] Where s, a, r, s′ are the current state, action, reward and next state of the sample respectively; α is the entropy weight coefficient; are the value network and the target network respectively; φ is the strategy network;
[0212] The algorithm needs to be under constraints Under this condition, the entropy weight coefficient α is automatically adjusted to balance the strategy entropy The ratio between α and the expected return, therefore, the cost function for designing the hyperparameter α is:
[0213]
[0214] In the formula, is the entropy constraint threshold;
[0215] 33) The algorithm strategy optimization stage focuses on using the updated Q value to guide the iteration process of the strategy function π; in order to increase the exploration ability of the reinforcement learning algorithm, the iteration goal is designed to minimize the information divergence between the strategy function π and the Q value function, that is, to maximize the expected reward and strategy entropy at the same time. It is expressed as:
[0216]
[0217] In the formula, π φ (s) = f φ (s, ξ) is the sampling value of the strategy distribution under the reparameterized skill; is the noise that obeys the standard normal distribution; K(s) is the partition function and does not participate in the strategy optimization;
[0218] The above strategy optimization is mainly based on iteration based on maximizing expected rewards. However, the manually designed reward function in complex traffic scenarios is not optimal, which may cause the strategy optimization process to deviate from the original intention of reward design. Therefore, the classifier is used to judge the risk of driving behavior on roundabouts to correct the direction of reward-guided strategy iteration, and the design introduces the optimization goal of behavioral risk strategy to make the strategy output safer driving behavior.
[0219] 34) The training process of human feedback reinforcement learning is introduced, including reward network training and policy function training, and the optimization target of the reinforcement learning behavior risk strategy is derived. In the reward network training process of human feedback reinforcement learning, it is optimized iteratively by maximizing the expected reward difference between good and bad behaviors under the same input, so that the reward network has the ability to identify better behavior strategies. The cost function is:
[0220]
[0221] In the formula, r ψ To reward the network; a + , a - are the superior and inferior behaviors in state s; σ is the sigmoid function;
[0222] 35) The strategy function training aims to maximize the expected reward, so that the strategy can output the dominant behavior that is more in line with the reward expectation, that is, the subjective expectation of human beings; at the same time, information divergence is added to limit the deviation between the new strategy and the original strategy. Its cost function is defined as:
[0223]
[0224] In the formula, π φ is the policy model for training; π ref is the initial model; β is the parameter that controls the degree of deviation between the two strategies; Z = ∑ a π ref (a|s)exp(r(s,a) / β) is the partition function, which is related only to the states s and π ref is related to the optimized strategy π φ Not relevant;
[0225] From the above formula, we can see that strategy πφ The optimization objective is equivalent to minimizing the difference between the numerator and the denominator; in addition, the optimal strategy π * Defined as:
[0226]
[0227] Further substituting formula (31) into formula (30), we get the strategy π φ The optimization goal is to minimize the training strategy π φ With the optimal strategy π * The deviation between the distributions is as follows:
[0228]
[0229] From formula (31), we get the reward network r ψ and the optimal strategy π * The relationship between them is:
[0230]
[0231] 36) Use the relationship between the reward network and the optimal strategy to eliminate the reward network r in formula (28) ψ , and based on the maximum likelihood optimization idea and the strategy π of formula (32) φ Optimize the objective by replacing the optimal policy with policy π φ , to achieve direct optimization strategy π using the dataset φ The purpose is to maximize the probability difference between the strategy outputting good and bad behaviors under the same state, as shown below:
[0232]
[0233] The above optimization objectives are introduced into the reinforcement learning algorithm strategy optimization process. At the same time, combined with the actual driving situation and human safety expectations for driving decisions, driving behaviors with greater risks are regarded as disadvantageous behaviors, thereby achieving the behavior risk strategy optimization objectives in the reinforcement learning algorithm and reducing the impact of insufficient reward function design on strategy optimization:
[0234]
[0235] In the formula, π φ′ is π φ The objective function of a + ,a - are respectively the superior and inferior behaviors based on subjective risk classification, and the target strategy function π φ′ Two random samplings are performed, and risk classification is performed through the behavioral risk classifier. The lower the driving risk, the better the behavior; β′=β+(C - -C +) / 4 is the behavioral risk category difference coefficient; C∈{0,1,2} are the three behavioral risk categories of safety, low risk and high risk respectively;
[0236] For behaviors with the same risk category, the value function Q is further adopted θ To improve the accuracy of behavior reward estimation, the present invention introduces a dual time scale update rule to Q θ The dual time scale update rule is optimized by increasing Q θ The update frequency can improve the accuracy of the evaluation.
[0237] After building the above-mentioned driving decision-making strategy based on behavioral risk reinforcement learning, it can be trained, tested and verified in the roundabout simulation scenario.
[0238] Embodiment 2
[0239] This embodiment uses a virtual driving simulator to build a roundabout simulation test platform, and tests and verifies the roundabout driving decision strategy of intelligent vehicles based on behavioral risk reinforcement learning. The conditions for the end of the simulation round are defined as follows: when the vehicle reaches the planned end point, collides, crosses the road boundary line, or is stationary for a long time, the simulation round ends and triggers the environment reset.
[0240] The safety of the roundabout driving decision-making strategy based on behavioral risk reinforcement learning was tested and verified in a roundabout simulation scenario. First, the basic reinforcement learning algorithm, the reinforcement learning algorithm with dual time scale update rules, and the behavioral risk reinforcement learning algorithm were trained in the same roundabout scenario to perform roundabout driving tasks such as merging, driving around the island, and driving out. The rewards during training are as follows: Figure 5 As shown, the curve represents the mean of the round reward, and the light-colored coverage area is the standard deviation of the round reward. All three reinforcement learning algorithms have reached a reward value of 700, indicating that the algorithm has the basic ability to complete the round-island driving task. From the rise of the mid-term reward curve, it can be seen that the dual-time scale update rule reinforcement learning algorithm and the behavioral risk reinforcement learning algorithm have achieved faster performance improvement than the basic reinforcement learning algorithm, indicating that the dual-time scale update rule can accelerate the convergence of the algorithm. In addition, the improvement of the behavioral risk reinforcement learning algorithm is more obvious than that of the dual-time scale update rule reinforcement learning algorithm, further indicating that the policy iteration direction of the behavioral risk reinforcement learning algorithm has been corrected, so that the strategy iterates faster in the direction of reward and subjective expectation. In the later stage of training, the behavioral risk reinforcement learning algorithm achieved higher rewards than other algorithms at convergence, reflecting that the algorithm optimization based on behavioral risk can help the algorithm avoid falling into the local optimum too early, thereby achieving safer and more efficient round-island driving decision-making performance.
[0241] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.
Claims
1. A decision-making method for intelligent car driving around an island based on behavioral risk reinforcement learning. It is characterized in that The following steps are involved: Step 1: Build a roundabout scene model; Step 2: Build a self-learning behavior risk classifier. Step 3: Model the behavioral risk reinforcement learning driving decision algorithm to enable smart cars to make decisions on roundabout driving.
2. According to claim 1, a smart car roundabout driving decision-making method based on behavioral risk reinforcement learning, It is characterized in that The specific method of step one is as follows: 11) Design action space; 12) Design state space; 13) Design a multi-objective reward function for intelligent cars.
3. According to claim 2, a smart car roundabout driving decision-making method based on behavioral risk reinforcement learning, It is characterized in that The specific method of step 11) is as follows: Combine intelligence and drive into the normalized signal p required for simulating vehicle longitudinal acceleration and deceleration control x ∈[-1, 1], and the front wheel active steering signal δ e ∈[-1, 1] to construct the horizontal and vertical coupling action space A of the intelligent vehicle e : A e =[p ex ,d e ] (1).
4. According to claim 2, a smart car roundabout driving decision-making method based on behavioral risk reinforcement learning, It is characterized in that The specific method of step 12) is as follows: Design a driving task vector V consisting of key tasks and driving tasks T ,as follows: Where ΔP END is the coordinate of the vehicle’s current position relative to the end point; and are the path points P of the vehicle relative to the front i The distance and orientation deviation, P i Progressive as the vehicle moves forward; Design the surrounding environment state vector V consisting of surrounding vehicle information E ; Through the region cutting coding method, the surrounding environment of the vehicle is divided into P 0 ~P 6 7 detection intervals; P 0 , P 1 , P 4 , P 5 The high interaction detection interval is expanded; when the external perception module detects other vehicles with the same driving direction as the self-vehicle and whose center point projection is located in the detection interval, it is marked as the perceived vehicle, and the corresponding vehicle information V i It is expressed as: V i =[F,ΔL,d -1 ,Δv,δθ,I],i={0,...,6}, (3) Where F∈{0,1} is the flag of whether there is a vehicle; ΔL,d,Δv=[Δv x , Δv y ] and δθ are the relative lane offset, distance, speed and relative orientation between the other vehicle and the vehicle itself; I∈[-1,1] is the possibility of the other vehicle changing lanes; When there is no other vehicle in the detection range, the corresponding setting Finally, we get the surrounding environment vector V E for: V E =[V 0 ,V 1 ,V 2 ,V 3 ,V 4 ,V 5 ,V 6 ]. (4) Design includes the vehicle speed information v e =[v ex ,v ey ], acceleration information a e =[a ex ,a ey ], driving direction information e =[cosθ e , sinθ e ] and yaw rate information ω e The vehicle state vector, with the future target point preview information added To comprehensively establish the state vector V e : V e =[v e ,a e ,h e ,oh e ,Dth e ,A e ], (5) In the formula, θ e is the heading angle of the vehicle; is the angle difference between the directions of the three future target points and the heading angle of the vehicle; In summary, the state space is formed: S=[V T ,V e ,V E ] T . (6)。 5. According to claim 2, a smart car roundabout driving decision-making method based on behavioral risk reinforcement learning, It is characterized in that The specific method of step 13) is as follows: Design collision reward function r c for: In the formula, min{d i } is the shortest distance between all vehicles and the self-vehicle in the detection interval; D c is the dangerous collision distance threshold; Through vertical and horizontal safety rewards f and r l Boot as follows: r f =-0.5|min(0,Δv fx )|(D f -d f ), (8) r l =-2.5|min(0,Δv ky )|(D l -d k ) 2 , (9) In the formula, Δv fx and d f ∈(D c , D f ) is the relative longitudinal speed and distance between the front vehicle and the vehicle; Δv ky and d k ∈(D c , D l ) is the relative lateral speed and distance between the nearest vehicle next to the vehicle and the vehicle; D f and D l The longitudinal and lateral safe vehicle distances; Rewards for designing roundabout entrance safety RE , as shown below: r RE =-0.25||Δv i ||(D RE -d i ), (10) Where, d i ∈(D c , D RE ) and Δv i are the distance and relative speed vector between the ego vehicle and the high-risk vehicle respectively; D RE Safe distance for high-risk interactions; The security reward function is obtained as follows: R S =r c +r f +r l +r RE , (11) Create a reward for reaching the end point r succ , as shown below: r succ =100,||ΔP End ||≤D P , (12) Where D P is the distance threshold between the vehicle and the path point P; Introducing path tracking reward r tag , as shown below: In the formula, is the driving distance of the vehicle between the two target points; Build Speed Bonus sp , as shown below: Design Course Rewards δθ Guide the vehicle to move in the specified road direction as shown below: Where V lim is the road speed limit; δθ e is the deviation between the vehicle heading and the road direction; Θ δθ1 , Θ δθ2 is the vehicle driving direction deviation threshold; Design the driving task reward function as follows: R T =r succ +r tag +r δθ +r sp (16) Design Turn Rewards st , as shown below: r st =-2|δ e |v x (17) Designing Lane Keeping Rewards lk , as shown below: Where δd is the lateral offset of the current position of the main vehicle relative to the center line of the road; L is the road width; when the vehicle crosses the road boundary, that is, |δd|≥L, r lk =-20 ends the current round and resets the environment; Define the control command reward r control , as shown below: r control =-4||A cur -A per || 2 (19) In the formula, A per ,A cur are the action vectors at the previous and next moments respectively; Establishing the yaw comfort bonus ω , guiding the vehicle's yaw rate to change smoothly, as shown below: r ω =-((ω-Ω ω ) / 5) 2 ,Oh ω <ω (20) In the formula, Ω ω is the yaw angular velocity threshold; Design driving behavior reward function R C , as shown below: R C =r lk +r ω +r st +r control (21) Finally, the multi-objective reward function of the intelligent car is obtained as follows: R=R S +R T +R C (22)。 6. According to claim 1, a smart car roundabout driving decision-making method based on behavioral risk reinforcement learning, It is characterized in that The specific method of step 2 is as follows: 21) Construct a multi-layer perceptron with inputs of state space s and action space a, and outputs the probabilities of three behavioral risk categories, namely safe, low risk, and high risk; 22) Train the classifier.
7. According to claim 2, a smart car roundabout driving decision-making method based on behavioral risk reinforcement learning, It is characterized in that The specific method of step 22) is as follows: 1) Data collection and labeling: Based on the existing roundabout decision-making strategy, collect roundabout driving simulation data; randomly sample state-action pairs {s, a} from the collected data, label them with behavioral risk categories, and evaluate the risk of their driving decision-making behaviors; the three types of behavioral risks are judged based on: high-risk behaviors refer to behaviors that will continue to deteriorate driving safety when executed under dangerous conditions such as collision, lane departure, or high-speed turning; Safe behavior refers to following or changing lanes under safe conditions or avoiding further deterioration of the conditions under dangerous conditions; low-risk behavior refers to behavior that will deteriorate driving safety if executed under safe conditions; 2) Pre-training: Use labeled data to pre-train the classifier based on supervised learning, so that it has the initial ability to distinguish different risk categories; 3) Self-learning training: In the prediction phase, the classifier first predicts the unlabeled roundabout data; High confidence screening: for those state-action pairs whose risk category prediction confidence exceeds 90%, they are labeled as the corresponding prediction category, and these data are added to the next round of training sets. The training cycle repeats the above self-learning process and continuously updates the classifier. 4) Termination of training: When the classification accuracy of the classifier on the test set no longer improves, the training ends; A joint loss consisting of focal loss and label distribution-aware margin loss is used to deal with the problems of class imbalance and difficult sample mining in the pre-training and self-learning training process. The joint loss function is constructed as follows: In the formula, γ is the hyperparameter for adjusting the weight factor; p y ∈[0,1] is the predicted value; α y =3∑ j n j / n y is the weight parameter for balancing the number of samples, where n j is the number of samples in each category; Design the parameter α related to the category proportion in the loss function y , n y As the training changes dynamically, the loss function is adaptively adjusted.
8. According to claim 1, a smart car roundabout driving decision-making method based on behavioral risk reinforcement learning, It is characterized in that The specific method of step three is as follows: 31) By introducing policy entropy To encourage the policy to produce more diverse exploration behaviors, as follows: In the formula, π φ is the strategy function; 32) The reinforcement learning algorithm iteratively solves the Q-value function considering the policy entropy H through the Bellman regression operator and temporal difference, as shown below: The cost function of the algorithm value network is defined as Where s, a, r, s′ are the current state, action, reward and next state of the sample respectively; α is the entropy weight coefficient; θ i , are the value network and the target network respectively; φ is the strategy network; Under the constraints Under this condition, the entropy weight coefficient α is automatically adjusted to balance the strategy entropy The ratio between the expected return and the design cost function of the hyperparameter α is as follows: In the formula, is the entropy constraint threshold; 33) The iteration objective is designed to minimize the information divergence between the policy function π and the Q-value function, that is, to maximize the expected reward and policy entropy at the same time As shown below: In the formula, π φ (s) = f φ (s, ξ) is the sampling value of the strategy distribution under the reparameterized skill; is the noise that obeys the standard normal distribution; K(s) is the partition function and does not participate in the strategy optimization; 34) Using the state-action dataset, the reward network is optimized iteratively by maximizing the expected reward difference between good and bad behaviors under the same input, so that the reward network has the ability to accurately evaluate good and bad strategies. The cost function is as follows: In the formula, r ψ To reward the network; a + , a - are the superior and inferior behaviors corresponding to the same state s; σ is the sigmoid function; 35) Information divergence is added to limit the deviation between the new strategy and the original strategy. The cost function is defined as: In the formula, π φ is the policy model for training; π ref is the initial model; β is the parameter that controls the degree of deviation between the two strategies; Z = ∑ a π ref (a|s)exp(r(s,a) / β) is the partition function, which is related only to the states s and π ref is related to the optimized strategy π φ Not relevant; Strategy π φ The optimization goal is equivalent to minimizing the difference between the numerator and the denominator. * Defined as: Further substituting formula (31) into formula (30), we get the strategy π φ The optimization goal is to minimize the training strategy π φ With the optimal strategy π * The deviation between the distributions is as follows: From formula (30), we get the reward network r ψ and the optimal strategy π * The relationship between them is: 36) Use the relationship between the reward network and the optimal strategy to eliminate the reward network r in formula (28) ψ , and based on the maximum likelihood optimization idea and the strategy π of formula (32) φ Optimize the objective by replacing the optimal policy with policy π φ , to directly optimize strategy π using human-annotated data φ The purpose is to maximize the probability difference between the strategy outputting good and bad behaviors under the same state, as shown below: By evaluating the risk of sampled driving behaviors, we can improve the direction of policy iteration and reduce the impact of insufficient reward function design on policy optimization, as shown below: In the formula, π φ′ is π φ The objective function of + ,a - They are respectively the advantages and disadvantages of behavior risk classification, and the target strategy function π φ′ Two random samplings are performed, and risk classification is performed through the behavioral risk classifier. The lower the driving risk, the better the behavior; β′=β+(C - -C + ) / 4 is the behavioral risk category difference coefficient; C∈{0,1,2} corresponds to the three risk behavior labels of safe, low-risk and high-risk, respectively.
Citation Information
Patent Citations
Automatic driving automobile decision planning method based on maximum entropy reinforcement learning
CN114580302A
Intelligent network connection automobile roundabout scene risk degree evaluation method based on risk field
CN117400930A
Automatic driving automobile decision-making method based on imitation learning and discrete reinforcement learning
CN117610681A
Automatic driving decision control method, device, system and equipment and storage medium
CN118393900A
Reinforcement learning automatic driving safety interpretable decision-making method based on risk estimation
CN118396131A
Cited By
Bionic memory-driven hybrid expert car-following decision-making method and system
CN121316900A
