A lane-changing decision control method for autonomous driving based on rule fusion reinforcement learning

By combining deep reinforcement learning and rule-based decision-making methods in the autonomous driving system, the safety evaluation mechanism is introduced, and the problem of insufficient reliability and interpretability of deep learning methods in complex traffic scenarios is solved, achieving higher decision-making safety and adaptability.

CN115257745BActive Publication Date: 2024-11-26TONGJI UNIV

Patent Information

Application Number
CN202210863035.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-07-21
Publication Date
2024-11-26
Estimated Expiration
2042-07-21

AI Technical Summary

Technical Problem

Among the existing autonomous driving decision-making methods, deep learning-based methods have reliability problems in complex traffic scenarios, which are difficult to trace and may make unreasonable actions, and lack interpretability and safety.

Method used

Adopt the method of rules-fusion reinforcement learning, combined with deep reinforcement learning and rule-based decision-making methods, and design a safety evaluation mechanism and reward function by establishing a vertical motion control model and a horizontal behavior decision-making model, introducing a safety evaluation mechanism to correct dangerous actions , and trained using dual-deep Q network and target value network.

Benefits of technology

It improves the decision reliability and safety of autonomous driving vehicles in complex road change scenarios, improves the interpretability and scenario adaptability of decision results, reduces the average road change risk value, and improves the success rate and driving efficiency of decision tasks.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115257745B_ABST
    Figure CN115257745B_ABST
Patent Text Reader

Abstract

The present invention relates to an automatic driving lane change decision control method based on rule fusion reinforcement learning, including: establishing a longitudinal motion control model of other vehicles and a lateral behavior decision model; configuring a highway driving environment, screening out unreasonable traffic scenes, and constructing an automatic driving decision training scene; establishing a state space and action space for driving behavior observation; building a deep learning network for strategy update; designing a lane change decision risk assessment method, and establishing a safety judgment mechanism for decision-making; setting a reward function; introducing a safety judgment mechanism into a deep learning network based on DDQN; performing fusion training based on the automatic driving decision training scene, correcting the dangerous actions decided, and setting two storage experience pools, combining the reward function to update the parameters of the target value network in the model, and the trained model is used for automatic driving lane change decision. Compared with the prior art, the present invention has a higher decision task success rate, driving efficiency and safety in the lane change scene.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving control technology, and in particular to an autonomous driving lane changing decision control method based on rule fusion reinforcement learning. Background Art

[0002] With the rapid development and close integration of current technologies such as automobiles, computers, radars, visual sensors, mapping, and robots, the advantages of autonomous driving technology are becoming more and more obvious. In the field of autonomous driving, the currently widely used decision-making technology routes are mainly divided into two categories: one is based on artificially formulated prior rules, and the other is a deep learning-based data-driven method.

[0003] Among them, rule-based decision-making methods can be traced back when problems occur to find the rule mechanism or corresponding parameters that caused the error. They have the advantage of explainability, and the execution results are stable and predictable. However, in actual traffic scenarios, the decision-making conditions faced by autonomous vehicles are often complex and changeable, and rule-based methods are difficult to adapt to. This has also made data-driven methods gradually become the focus of decision-making method research.

[0004] At present, in data-driven, data-driven methods based on deep learning are mainly divided into supervised learning and deep reinforcement learning. Reinforcement learning is collected by the agent's free trial and error in the simulation environment. The quality of the decision result is evaluated in the form of a reward function through the change in the state of the environment after the decision is executed. Therefore, its data comes from the real interaction results between the agent and the environment. The coverage of decision data is wider, and the labels of decision data are more real and objective. However, deep learning data-driven methods often rely on deep neural networks, which have poor interpretability and are difficult to trace when encountering problems. In addition, the update of neural networks is actually a process of iterative optimization to approximate functions. Even after the loss function converges, the deep neural network may still make unreasonable or even dangerous actions.

[0005] In summary, although the decision-making method based on deep reinforcement learning has strong scene adaptability, it still has reliability issues. Therefore, for autonomous driving vehicles, there is an urgent need for a driving decision-making control method that can take into account safety, reliability, strong explainability and good scene adaptability. Summary of the invention

[0006] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide an automatic driving lane changing decision control method based on rule-fused reinforcement learning. By combining deep reinforcement learning with a rule-based decision-making method during the training process, the reliability and safety of the decision results of the automatic driving vehicle in complex lane changing scenarios can be improved.

[0007] The purpose of the present invention can be achieved by the following technical solution: A method for controlling lane change decision of an automatic driving system based on rule fusion reinforcement learning, comprising the following steps:

[0008] S1. Establish the longitudinal motion control model and lateral behavior decision model of other vehicles;

[0009] S2. Configure the highway driving environment, add the perception range and perception noise of the vehicle, filter out unreasonable traffic scenes, and complete the construction of the autonomous driving decision-making training scene on structured roads;

[0010] S3. Based on the Markov decision model, establish the state space and action space of driving behavior observation; based on the DDQN (Double Deep Q Network) algorithm, build a deep learning network for strategy update;

[0011] S4. Calculate the predicted trajectory of other vehicles and its uncertainty, design a risk assessment method for lane-changing decisions, and then establish a safety assessment mechanism for decision-making;

[0012] S5. Consider the safety evaluation results and set the reward function;

[0013] Introducing the safety evaluation mechanism into the DDQN-based deep learning network;

[0014] Based on the constructed autonomous driving decision-making training scenario, fusion training is performed to correct the dangerous actions decided, and two storage experience pools are set up. The parameters of the target value network in the model are updated in combination with the reward function to obtain the trained model, which is used as the autonomous driving lane change decision model;

[0015] S6. Using the automatic driving lane changing decision model, the actual lane changing decision result of the automatic driving vehicle is output; according to the lane changing decision result, the vehicle planning module outputs the optimal trajectory of the corresponding vehicle so that the vehicle drives according to the optimal trajectory.

[0016] Furthermore, the longitudinal motion control model in step S1 is specifically established based on IDM (Intelligent Driver Model), and the motion control behavior of the longitudinal motion control model includes the acceleration of the other vehicle;

[0017] The lateral behavior decision model in step S1 is specifically established based on the MOBIL (Minimizing Overall Braking Induced by Lane changes) algorithm, and the lateral decision behaviors of the lateral behavior decision model include lane keeping and lane changing.

[0018] Furthermore, the highway driving environment in step S2 includes the number of scene lanes, the width of each lane, the total length of the road, the speed limit of each lane, the number of other vehicles, the lane set where the other vehicles are initialized, the initial speed of each vehicle of the other vehicles, the initial longitudinal position set of the other vehicles, and the speed expansion coefficient of the longitudinal motion control model;

[0019] The unreasonable traffic flow scenarios in step S2 include: canceling the left lane change when the vehicle is in the leftmost lane, and canceling the right lane change when the vehicle is in the rightmost lane;

[0020] When the vehicle speed reaches the maximum speed limit, the acceleration action is canceled; when the vehicle speed reaches the minimum speed limit, the deceleration action is canceled;

[0021] If the distance between the vehicle and the vehicle in front is less than the preset reference following distance, the acceleration command is cancelled.

[0022] Furthermore, the state space in step S3 includes the state information of the vehicle and the relevant state information of the four other vehicles closest to the vehicle in the curve coordinate system; the state information of the vehicle includes a flag bit that is always set to 1, the longitudinal coordinate s0 of the vehicle along the reference road, the lateral offset l0 of the road curve coordinate system, the speed component v of the vehicle along the reference road and perpendicular to the reference road 0x and v 0y ;

[0023] The relevant status information of the four other vehicles includes a flag bit of existence (1 if it exists, 0 otherwise), the curve distance Δs relative to the self-vehicle along the reference road, the distance Δl relative to the self-vehicle perpendicular to the reference road curve, and the relative speed Δv along the reference road curve. s , relative speed Δv in the direction perpendicular to the reference road curve l ;

[0024] The action space in step S3 is divided into discrete decisions of the target lane and discrete decisions of the target speed, including five discrete decisions: left lane change, right lane change, acceleration, deceleration, and IDLE (idling) without changing lanes or accelerating or decelerating.

[0025] Furthermore, the specific process of building a deep learning network in step S3 is:

[0026] First, build a deep learning network structure. The overall structure is a 25*256*256*5 fully connected neural network. The specific layers and settings are:

[0027] The observation information received by the input layer is a 25*1 vector;

[0028] Two hidden layers are set in the middle, each containing 256 neurons;

[0029] The output layer outputs information of 5*1;

[0030] The activation function uses ReLU and sets the learning rate;

[0031] Then set the network related configuration as follows:

[0032] Set the decision cycle of the network;

[0033] Set the sampling probability of each action so that the sampling probability of actions with high reward values ​​increases with the increase of exploration times;

[0034] The mean square error (MSE) is used to calculate the loss function;

[0035] Adam is used as the optimizer.

[0036] Furthermore, the risk assessment of the lane change decision in step S4 is specifically based on the collision probability between the vehicle and the other vehicle. Assuming that there are N trajectory points on the trajectory, the risk assessment of the lane change decision at time point T on the trajectory is uncertain [t] Gaussian distribution is sampled X times to obtain X possible positions and postures of the vehicle at time t. If they match the planned trajectory T plan [t] If overlap occurs, count one collision, otherwise count nothing;

[0037] Setting I c A function to determine whether the vehicle collides with another vehicle:

[0038]

[0039] The ratio of the number of collision samples to the total number of samples X is used as the collision probability P between the vehicle and the vehicle at time t. coll , which represents the collision probability, is calculated as follows:

[0040]

[0041] Among them, S self is the position rectangle of the planned trajectory of the vehicle at time t, S other is the rectangular frame of the surrounding vehicles’ pose obtained by the i-th sampling;

[0042] The safety evaluation mechanism in step S4 specifically performs a comprehensive evaluation based on three safety indicators: collision probability peak, peak time inverse, and number of high-risk trajectory points. The three safety indicators are normalized to calculate a comprehensive lane-changing risk value. If the comprehensive lane-changing risk value is greater than a set lane-changing risk threshold, it is determined to be a dangerous action, otherwise it is a safe action.

[0043] Furthermore, the reward function set in step S5 includes efficiency reward, safety reward and basic traffic etiquette reward, wherein the efficiency reward is specifically: when the autonomous driving vehicle collides with another vehicle, a negative reward is given and the environment is terminated and reset;

[0044] Safety rewards are specifically: negative rewards are given when a dangerous action is judged;

[0045] Traffic etiquette rewards are as follows: positive rewards are given to vehicles that leave the leftmost lane.

[0046] Furthermore, the step S5 specifically includes the following steps:

[0047] S51, respectively setting an efficiency reward, a safety reward, and a basic traffic etiquette reward, summing the efficiency reward, the safety reward, and the basic traffic etiquette reward, and normalizing the summation result to obtain a reward function;

[0048] S52. Based on the safety evaluation results, fusion training is performed in combination with the autonomous driving decision-making training scenario, and dangerous decisions whose comprehensive risks due to the safety evaluation results are greater than the set comprehensive danger threshold are corrected;

[0049] S53, setting two storage experience pools to update the parameters of the target value network in the model, wherein the two storage experience pools include a safety experience pool and a danger experience pool;

[0050] S54, using Bellman equation to calculate the target value of sampling data;

[0051] S55, calculate the loss function gradient, update the main value network, every N targ Step 1 updates the target value network parameter to θ targ .

[0052] Furthermore, the strategy for correcting the risk decision in step S52 is specifically:

[0053] If the dangerous maneuver is a left or right lane change,

[0054]

[0055] If the action judged to be dangerous is not an acceleration, constant speed, or deceleration action, then

[0056]

[0057] Furthermore, the step S53 specifically includes the following steps:

[0058] S531. After the dangerous action is corrected, two storage experience pools are set to store the safe and dangerous action experiences respectively:

[0059] In the dangerous action experience pool, a fixed penalty r is given to the action state (s, a) dangerous , store the state transfer information (s,a,*,r dangerous );

[0060] In the security experience pool, calculate the reward r and store the state transfer information (s,a,s′,r);

[0061] S532. When the safety experience pool and the danger experience pool have collected a set amount of data, experience playback is performed using a fixed-ratio random sampling method:

[0062] The sampling trajectory length is N t , when the security experience sampling ratio is k, kN are randomly selected from the security experience pool t state transition information; randomly extract (1-k)N from the dangerous experience pool t The two batches of data are then combined as objects for batch processing to calculate the loss function.

[0063] Compared with the prior art, the present invention proposes an automatic driving lane changing decision control method based on rule-fused reinforcement learning, which combines deep reinforcement learning with a rule-based decision method during the training process and introduces a safety evaluation mechanism. Compared with the traditional reinforcement learning method, the decision model trained by the present invention has better performance, which can not only cope with high-dimensional complex traffic scenes, but also ensure that the decision results have strong reliability, effectively improving the lane changing accuracy of the automatic driving vehicle.

[0064] In the process of reinforcement learning decision training based on rule fusion, the present invention sets the reward function based on the established safety evaluation mechanism on the one hand, so that the cumulative decision reward is higher; on the other hand, by correcting the dangerous actions decided, and setting two storage experience pools to store safe and dangerous state-action pairs respectively, the parameters of the target value network in the model are updated, thereby ensuring that the average lane-changing risk value is lower. This makes the reward increase speed, convergence value and safety index of the present invention better than traditional learning methods in the training process, and has a higher decision-making task success rate, driving efficiency and safety in lane-changing scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 It is a schematic diagram of the method flow of the present invention;

[0066] Figure 2 A schematic diagram of the process of constructing an automatic driving lane change decision model in an embodiment;

[0067] Figure 3 This is a diagram of a high-speed four-lane lane change scenario in the embodiment;

[0068] Figure 4 This is a diagram of the deep learning network structure;

[0069] Figure 5 Schematic diagram of the model training process of the automatic driving lane changing decision method based on rule fusion reinforcement learning in the embodiment. DETAILED DESCRIPTION

[0070] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0071] Example

[0072] like Figure 1 As shown, a lane-changing decision control method for autonomous driving based on rule fusion reinforcement learning includes the following steps:

[0073] S1. Establish the longitudinal motion control model and lateral behavior decision model of other vehicles;

[0074] S2. Configure the highway driving environment, add the perception range and perception noise of the vehicle, filter out unreasonable traffic scenes, and complete the construction of the autonomous driving decision-making training scene on structured roads;

[0075] S3. Based on the Markov decision model, establish the state space and action space of driving behavior observation; based on the DDQN algorithm, build a deep learning network for strategy updating;

[0076] S4. Calculate the predicted trajectory of other vehicles and its uncertainty, design a risk assessment method for lane-changing decisions, and then establish a safety assessment mechanism for decision-making;

[0077] S5. Consider the safety evaluation results and set the reward function;

[0078] Introducing the safety evaluation mechanism into the DDQN-based deep learning network;

[0079] Based on the constructed autonomous driving decision-making training scenario, fusion training is performed to correct the dangerous actions decided, and two storage experience pools are set up. The parameters of the target value network in the model are updated in combination with the reward function to obtain the trained model, which is used as the autonomous driving lane change decision model;

[0080] S6. Using the automatic driving lane changing decision model, the actual lane changing decision result of the automatic driving vehicle is output; according to the lane changing decision result, the vehicle planning module outputs the optimal trajectory of the corresponding vehicle so that the vehicle drives according to the optimal trajectory.

[0081] This embodiment applies the above technical solution to build an automatic driving lane change decision model. Figure 2 As shown, it mainly includes:

[0082] 1. Establish the longitudinal motion control model and lateral behavior decision model of other vehicles;

[0083] Specifically include:

[0084] 11) Based on the IDM model, the longitudinal behavior of the acceleration of the other vehicle is established. First, the reference value of the distance between the vehicle and the front vehicle is calculated at this moment:

[0085]

[0086] Wherein, d0 is the preset expected following distance, T is the preset expected headway, a is the maximum comfortable acceleration, b is the maximum comfortable deceleration, and Δv is the speed difference between the vehicle and the preceding vehicle;

[0087] Then calculate the acceleration based on whether there is a vehicle in front:

[0088] 111) If there is no car in front, the acceleration of the other car is

[0089]

[0090] Among them, v0 is the target speed of the other car;

[0091] 112) If there is a car in front, then the acceleration of the other car is

[0092]

[0093] Where d is the actual distance between the vehicle and the front vehicle;

[0094] 12) Based on the MOBIL algorithm, establish lateral behaviors including lane keeping and lane changing;

[0095] 121) If Lane keeping;

[0096] in, is the acceleration required by the vehicle behind the target lane, b safe The maximum braking acceleration that the rear vehicle can take under normal circumstances;

[0097] 122) Let p be the courtesy coefficient, if Then change lanes;

[0098] Among them, a c is the longitudinal acceleration of the vehicle and the vehicle in front of the current lane, a n is the longitudinal acceleration of the vehicle behind and ahead in the target lane, a o is the following acceleration of the vehicle behind the current lane and the vehicle itself, is the following acceleration of the vehicle and the vehicle in front of the target lane after the lane change is successful, is the following acceleration of the vehicle behind and the vehicle in the target lane, is the following acceleration of the vehicle behind and in front of the current lane, and p is the front courtesy coefficient.

[0099] Second, configure two driving environments: expressway and low-speed urban area, add the perception range and perception noise of the vehicle, filter out unreasonable traffic scenes, and complete the construction of the automatic driving decision simulation scene on structured roads;

[0100] Specifically:

[0101] 21) Configure the highway scene, the number of lanes in the scene is n lane , the width of each lane w, the total length of the road l, and the speed limit setting of each lane. In this embodiment, the number of lanes in the scene n lane is 4, the width of each lane w is 4m, the total length of the road l is 1000m, and the speed limit of each lane is set to v min =20m / s,v max =30m / s, the scenario after configuration is as follows Figure 3 As shown, Figure 3 The other cars are initialized from left to right;

[0102] 211) Set the number of other cars n;

[0103] Set traffic saturation Calculate the number of vehicles that the road can accommodate

[0104] Among them, d min is the minimum following distance. In this embodiment, the reference vehicle spacing Δs is taken init As the minimum following distance; d actual is the actual average following distance of adjacent vehicles;

[0105] 212) Using the uniform distribution method in the integer set containing the number of lanes, set the lane set where n other vehicles are initialized {lane_index init} n , in this embodiment, the uniform distribution of the integer set [1,2,3,4] is satisfied;

[0106] 213) Set the initial speed of each of the n other vehicles {v init} n , with 0.8 times the current road speed limit as the mean and 0.7 times the speed limit as the variance Gaussian distribution, random sampling generates the initial speed, which satisfies the equation:

[0107] v init ~N(0.8v max ,(0.7v max ) 2 )

[0108] 214) Generate n initial longitudinal position sets of other vehicles {s init} n ;

[0109] First calculate the reference vehicle spacing s max is the longitudinal position of the vehicle currently on the right;

[0110] Based on Δs init Calculate the initial generation position of his car: s init =s max +Δs init ;

[0111] 215) randomly generating a velocity expansion coefficient ξ of the IDM control model within a given range. In this embodiment, the given range is [3.5, 4.5];

[0112] 22) Add perception range and perception noise to simulate the limited perception of autonomous vehicles on real roads;

[0113] In the vehicle coordinate system, the sensing range in the x direction is from 20m behind the vehicle to 70m in front of the vehicle, and the sensing range in the y direction is from 20m on the left side of the vehicle to 20m on the right side of the vehicle.

[0114] The perceived uncertainty error is considered as a three-dimensional Gaussian distribution, with the observed state quantity To represent the relative position and heading angle of another vehicle in the rectangular coordinate system of the vehicle as perceived by the vehicle;

[0115] In the specific calculation, we first set the covariance matrix Σ to a 3×3 symmetric matrix:

[0116]

[0117] Among them, σ x , σ y , are the standard deviations of the three variables, ρ xy , and are the mutual correlation coefficients among the three variables;

[0118] The Gaussian probability density function of X is obtained as follows:

[0119]

[0120] Initial observation state Σ is the covariance, and the true value of other vehicle poses It is obtained by sampling the three-dimensional Gaussian distribution with the mean;

[0121] 23) Screen out unreasonable traffic scenes;

[0122] The following traffic scenarios are excluded: when the vehicle is in the leftmost lane, the left lane change is canceled; when the vehicle is in the rightmost lane, the right lane change is canceled; when the vehicle speed reaches the maximum speed limit, the acceleration action is canceled; when the vehicle speed reaches the minimum speed limit, the deceleration action is canceled; when the distance between the vehicle and the vehicle in front is less than the reference following distance d * , the acceleration command is canceled.

[0123] 3. Establish the state space and action space of driving behavior observation based on the Markov decision model, and build a deep learning network for strategy updating based on the DDQN algorithm;

[0124] 31) Set the state space of observation;

[0125] The state space includes the state information of the ego vehicle and the four vehicles closest to the ego vehicle in the curvilinear coordinate system;

[0126] 311) The vehicle status information includes a flag bit that is always set to 1, the longitudinal coordinate s0 of the vehicle along the reference road, the lateral offset l0 of the road curve coordinate system, the velocity component v of the vehicle along the reference road and perpendicular to the reference road 0x and v 0y ;

[0127] 312) The information of the 4 other vehicles includes a flag bit of whether it exists, which is 1 if it exists, otherwise 0; the curve distance Δs relative to the self-vehicle along the reference road, the distance Δl relative to the self-vehicle perpendicular to the reference road curve, and the relative speed Δv along the reference road curve. s , relative speed Δv in the direction perpendicular to the reference road curve l ;

[0128] The observed state matrix expression is:

[0129]

[0130] 313) Normalize the position Δs and velocity Δv in the observation matrix respectively, and compress the normalized observation matrix into a one-dimensional input vector s acceptable to the network. i nput :

[0131] Normalize the position quantity Δs:

[0132]

[0133] Where, Δs max , Δs min are the upper and lower limits of the normalized position quantity, satisfying Δs min =-5v max ;

[0134] Normalize the velocity Δv:

[0135]

[0136] Where Δv max , Δv min are the upper and lower limits of the normalized velocity, satisfying Δv min =-2v max and Δv max =2v max ;

[0137] The final one-dimensional observation information received by the network is:

[0138]

[0139] 32) Set the action space;

[0140] The five discrete decisions of the decision network model are divided into target lane and lane target Discrete decision and target speed v target The discrete decision of is as follows:

[0141] 321) Target lane target The discrete decisions are divided into left lane change, right lane change and no lane change. The lane numbers increase from left to right along the driving direction. In this embodiment, the lane numbers are 0-3 from left to right along the driving direction:

[0142]

[0143] Among them, lane current Indicates the lane number that the autonomous vehicle is currently in, lane max represents the maximum lane number, and clip(·) represents the truncation function, which limits the lanes beyond [0, lane max ] range number;

[0144] 322) Target speed v target The discrete decision of mainly determines the longitudinal behavior of the vehicle, including acceleration and deceleration; the corresponding target speed can be divided into acceleration, deceleration and constant speed:

[0145]

[0146] Among them, v current represents the current vehicle speed, Δv acc Δv is the target speed increment after the acceleration command is issued. dec It is the amount by which the target speed is reduced after the deceleration command is issued;

[0147] 323) Combined with the above discrete decision results, the decision action set is divided into the following five categories:

[0148] Left lane change, right lane change, acceleration, deceleration, and IDLE without changing lanes or accelerating or decelerating;

[0149] 33) Build and configure a deep learning network based on DDQN;

[0150] 331) The network structure built:

[0151] The overall structure is a 25*256*256*5 fully connected neural network, and the specific layers and settings are:

[0152] The observation information received by the input layer is a 25*1 vector;

[0153] Two hidden layers are set in the middle, each containing 256 neurons;

[0154] The output layer outputs information of 5*1;

[0155] The activation function uses ReLU and sets the learning rate. In this embodiment, the learning rate is set to 0.2.

[0156] The network structure after setting is as follows Figure 4 As shown;

[0157] 332) Set the network related configuration as follows:

[0158] Set the decision cycle of the network;

[0159] Set the sampling probability of each action so that the sampling probability of actions with high reward values ​​increases with the increase of exploration times. First, determine whether action a is a Q value. The action a with the highest Q value determined by the network is max = argmax a Q(s,a), and then perform the following corresponding calculations:

[0160]

[0161] Among them, ε starts from 1 and gradually decreases as the number of explorations increases. The calculation method is shown in the following formula:

[0162]

[0163] Where |A| is the number of discrete decision actions, ε initial is 1, ε final Set to 0, time is the number of decisions, and tau is the decay coefficient;

[0164] The mean square error (MSE) is used to calculate the loss function. The function expression used in this embodiment is: j -Q(s,a;θ)|| 2 ;

[0165] Adam is used as the optimizer.

[0166] 4. Calculate the predicted trajectory of other vehicles and its uncertainty, design a risk assessment method for lane-changing decisions, and then establish a safety assessment mechanism for decision-making;

[0167] 41) Carry out fusion prediction of other vehicle trajectories;

[0168] 411) Predicting other vehicles’ trajectories based on behavior models;

[0169] For the longitudinal discrete planning end point on the center line of the target lane, a lane change trajectory cluster is generated using a quintic polynomial curve, and the optimal path T is selected through the path evaluation function. man ;

[0170] In the planned path, the corresponding curve distance is interpolated to obtain the position of the other car at the next time t (x man (t),y man (t));

[0171] Prediction trajectory T based on behavior model man ={(x man (t),y man (t))};

[0172] 412) Predicting other vehicle trajectories based on kinematic models;

[0173] Using a kinematic model:

[0174]

[0175] Divided into two cases when ω=0 and ω≠0, calculate the position of the vehicle at the future time t (x(t), y(t));

[0176] When ω≠0:

[0177]

[0178] in is the initial position and initial heading angle of the vehicle at the current moment;

[0179] The predicted trajectory T based on the kinematic model is obtained mdl ={(x mdl (t),y mdl (t))};

[0180] 413) Use a cubic curve to transform T mdl With T man Fusion, to obtain the fusion prediction trajectory, this embodiment uses the cubic curve w(t) = a3t 3 +a2t2 +a1t+a0 will T mdl With T man Fusion, get the fusion prediction trajectory:

[0181] T fin (t) = w(t)·T mdl (t)+(1-w(t))·T man (t)

[0182] 42) Determine the uncertainty of the predicted trajectory;

[0183] 421) Use Gaussian distribution to calculate the horizontal coordinate x, vertical coordinate y, and heading angle of the other vehicle The uncertainty is described by:

[0184]

[0185] 43) Design an evaluation method based on the collision probability P between the vehicle and other vehicles coll , to assess the risk of lane-changing decisions;

[0186] 431) Assume that there are N points on the trajectory. For the point T at time t on the trajectory uncertain [t] Gaussian distribution is sampled X times to obtain X possible positions and postures of the vehicle at time t. If they match the planned trajectory T plan [t] overlap, then count 1 collision, otherwise do not count;

[0187] Set the function I to determine whether the vehicle collides with other vehicles c , I c Output Boolean value:

[0188]

[0189] The ratio of the collision sampling number to the total sampling number X is used as the collision probability P between the ego vehicle and the vehicle at time t. coll , to represent the collision risk, is calculated as follows:

[0190]

[0191] Where S self is the position rectangle of the planned trajectory of the vehicle at time t, S other is the rectangular frame of the surrounding vehicles’ pose obtained by the i-th sampling;

[0192] 44) Establish a safety assessment mechanism for decision-making based on the calculated lane-changing decision risk;

[0193] Specifically, the comprehensive collision probability of the lane-changing trajectory is calculated, and indicators such as the peak value of the collision probability, the inverse of the peak time, and the number of high-risk trajectory points are selected for comprehensive evaluation;

[0194] 441) Comprehensive collision probability P i The collision probability of the vehicle trajectory point at time i with all other vehicles is summed up to get:

[0195]

[0196] Among them, p ij is the collision probability between the vehicle and the jth vehicle at time i, and the sequence p is obtained by sorting them. ij ordered ;

[0197] 442) Select the safety index and calculate the comprehensive lane-changing risk to make a safety assessment; the safety index is the number of high-risk trajectory points C HR , peak collision probability C p and the inverse of the peak time C TTP , where C HR Indicates the number of collision risk values ​​P0 at each point of the vehicle lane change decision trajectory that are greater than the threshold;

[0198] C p =min(max(P1,P2,…P I-1 ,P I ),1),I represents the number of trajectory points on the predicted trajectory,P i Represents the comprehensive collision probability at each moment;

[0199] Peak time inverse C TTP =1 / T P , where T P is the collision peak time TTP;

[0200] The above three safety indicators are normalized to calculate the comprehensive lane-changing risk. If the risk is greater than 0.5, it is determined to be a dangerous action, otherwise it is a safe action. In this embodiment, C HR The normalized range is [0,20], C p is [0,1], C TTP is [0.05,20], and the comprehensive lane-changing risk risk is calculated as follows:

[0201]

[0202] If the risk is greater than 0.5, it is considered a dangerous action, otherwise it is a safe action.

[0203] 5. Consider the safety evaluation results and set the reward function;

[0204] 51) Set the reward function;

[0205] Rewards are set based on efficiency, safety, and basic traffic etiquette;

[0206] 511) Set efficiency rewards efficiency ;

[0207] When the autonomous vehicle collides with another vehicle, it will be given a large negative reward and the environment will be terminated and reset;

[0208] The calculation formula is:

[0209] where k speed is the discount factor;

[0210] 512) Set up security rewards safe , divided into dangerous actions and non-dangerous actions, and a negative reward r is given when it is judged as a dangerous action dangerous ;

[0211] The calculation formula is:

[0212] r dangerous It is a negative reward for dangerous actions;

[0213] 513) Set up traffic etiquette rewards rule , giving a small reward to vehicles that leave the leftmost lane:

[0214]

[0215] 514) Normalize and calculate the final reward;

[0216] The final reward is the sum of the three rewards, and the sum is normalized to get the final reward:

[0217]

[0218] 6. Introduce the safety evaluation mechanism of step 4 into the deep learning network based on DDQN, perform fusion training based on the scenario built in step 2, correct the dangerous actions decided, set up two storage experience pools, update the parameters of the target value network in the model, and use the trained model for autonomous driving lane change decision;

[0219] Figure 5 The entire training process of the autonomous driving lane change decision method based on rule fusion reinforcement learning is shown in the form of a flowchart. The specific steps are as follows:

[0220] 61) Based on the results of the safety assessment in step 4, the risk decisions with a comprehensive risk greater than 0.5 are revised, and the decisions with a comprehensive risk less than or equal to 0.5 are not revised;

[0221] 62) Conduct fusion training based on the scenario constructed in step 2 to correct the dangerous actions decided;

[0222] 621) The model is trained in a high-speed four-lane lane change scenario, with V / C set, the speed of the other vehicle is V0 m / s, and the speed of the self-vehicle is V e m / s. In this embodiment, V / C is set to 1.0, and the speed of the other vehicle is V o ∈[20,30]m / s, vehicle speed V e ∈[20,30]m / s, training until the cumulative reward converges;

[0223] 622) Establish a mechanism to correct lateral dangerous movements:

[0224] If the dangerous action is a left or right lane change,

[0225]

[0226] 623) Establish a vertical dangerous action correction mechanism:

[0227] If the action judged to be dangerous is not an acceleration, constant speed, or deceleration action, then

[0228]

[0229] 63) Set up two storage experience pools and update the parameters of the target value network in the model;

[0230] 631) After the dangerous action is corrected, two experience pools are set up to store the safe and dangerous action experience separately;

[0231] In the dangerous action experience pool, a fixed penalty r is given to the action state (s, a) dangerous , store the state transfer information (s,a,*,r dangerous );

[0232] In the security experience pool, calculate the reward r and store the state transfer information (s,a,s′,r);

[0233] 632) When the safety experience pool and the danger experience pool have collected enough data, experience replay is performed by using a fixed-proportion random sampling method;

[0234] Specifically, the sampling trajectory length is N t , when the security experience sampling ratio is k, kN are randomly selected from the security experience pool t state transition information, randomly extract (1-k)N from the dangerous experience pool t The state transition system information is combined into two batches of data as the object of batch processing loss function calculation;

[0235] 633) The Bellman equation is used to calculate the target value of the sampling data. The specific calculation formula is:

[0236]

[0237] 634) Calculate the loss function gradient and update the main value network. targ Step 1 updates the target value network parameter to θ targ .

[0238] In summary, this technical solution combines deep reinforcement learning with rule-based decision-making methods during the training process and introduces a safety evaluation mechanism. The decision model trained in this way is more reliable, obtains higher cumulative decision rewards, and has a lower average lane-changing risk value. It can simultaneously take into account the advantages of safety, reliability, strong interpretability, and good adaptability to complex traffic scenarios.

Claims

1. A lane-changing decision control method for autonomous driving based on rule fusion reinforcement learning. It is characterized in that The following steps are involved: S1. Establish the longitudinal motion control model and lateral behavior decision model of other vehicles; S2. Configure the highway driving environment, add the perception range and perception noise of the vehicle, filter out unreasonable traffic scenes, and complete the construction of the autonomous driving decision-making training scene on structured roads; S3. Based on the Markov decision model, establish the state space and action space of driving behavior observation; based on the DDQN algorithm, build a deep learning network for strategy updating; S4. Calculate the predicted trajectory of other vehicles and its uncertainty, design a risk assessment method for lane-changing decisions, and then establish a safety assessment mechanism for decision-making; S5. Consider the safety evaluation results and set the reward function; Introducing the safety evaluation mechanism into the DDQN-based deep learning network; Based on the constructed autonomous driving decision-making training scenario, fusion training is performed to correct the dangerous actions decided, and two storage experience pools are set up. The parameters of the target value network in the model are updated in combination with the reward function to obtain the trained model, which is used as the autonomous driving lane change decision model; S6. Using the automatic driving lane changing decision model, output the actual lane changing decision result of the automatic driving vehicle; according to the lane changing decision result, the vehicle planning module outputs the optimal trajectory of the corresponding vehicle, so that the vehicle drives according to the optimal trajectory; The risk assessment of the lane change decision in step S4 is specifically based on the collision probability between the vehicle and the other vehicle. Assuming that there are N trajectory points on the trajectory, at time point T uncertain [t] Gaussian distribution is sampled X times to obtain X possible positions and postures of the vehicle at time t. If they match the planned trajectory T plan [t] If overlap occurs, count one collision, otherwise count nothing; Setting I c A function to determine whether the vehicle collides with another vehicle: The ratio of the number of collision samples to the total number of samples X is used as the collision probability P between the vehicle and other vehicles at time t. coll , which represents the collision probability, is calculated as follows: Among them, S self is the position rectangle of the planned trajectory of the vehicle at time t, S other is the rectangular frame of the surrounding vehicles’ pose obtained by the i-th sampling; The safety evaluation mechanism in step S4 is specifically to perform a comprehensive evaluation based on three safety indicators: collision probability peak value, peak time inverse, and number of high-risk trajectory points. By normalizing the three safety indicators, a comprehensive lane-changing risk value is calculated. If the comprehensive lane-changing risk value is greater than a set lane-changing risk threshold, it is determined to be a dangerous action, otherwise it is a safe action. The reward function set in step S5 includes efficiency reward, safety reward and basic traffic etiquette reward, wherein the efficiency reward is specifically: when the autonomous driving vehicle collides with another vehicle, a negative reward is given and the environment is terminated and reset; Safety rewards are specifically: negative rewards are given when a dangerous action is judged; Traffic etiquette rewards are specifically: positive rewards are given to vehicles that leave the leftmost lane; The step S5 specifically comprises the following steps: S51, respectively setting an efficiency reward, a safety reward, and a basic traffic etiquette reward, summing the efficiency reward, the safety reward, and the basic traffic etiquette reward, and normalizing the summation result to obtain a reward function; S52. Based on the safety evaluation results, fusion training is performed in combination with the autonomous driving decision-making training scenario, and dangerous decisions whose comprehensive risks due to the safety evaluation results are greater than the set comprehensive danger threshold are corrected; S53, setting two storage experience pools to update the parameters of the target value network in the model, wherein the two storage experience pools include a safety experience pool and a danger experience pool; S54, using Bellman equation to calculate the target value of sampling data; S55, calculate the loss function gradient, update the main value network, every N targ Step 1 updates the target value network parameter to θ targ ; The strategy for correcting the risk decision in step S52 is specifically: If the dangerous maneuver is a left or right lane change, If the action judged to be dangerous is not an acceleration, constant speed, or deceleration action, then The step S53 specifically includes the following steps: S531. After the dangerous action is corrected, two storage experience pools are set to store the safe and dangerous action experiences respectively: In the dangerous action experience pool, a fixed penalty r is given to the action state (s, a) dangerous , store the state transfer information (s,a,*,r dangerous ); In the security experience pool, calculate the reward r and store the state transfer information (s,a,s′,r); S532. When the safety experience pool and the danger experience pool have collected a set amount of data, experience playback is performed using a fixed-ratio random sampling method: The sampling trajectory length is N t , when the security experience sampling ratio is k, kN are randomly selected from the security experience pool t state transition information; randomly extract (1-k)N from the dangerous experience pool t The two batches of data are then combined as objects for batch processing to calculate the loss function.

2. According to claim 1, an automatic driving lane changing decision control method based on rule fusion reinforcement learning, It is characterized in that The longitudinal motion control model in step S1 is specifically established based on the IDM, and the motion control behavior of the longitudinal motion control model includes the acceleration of the other vehicle; The lateral behavior decision model in step S1 is specifically established based on the MOBIL algorithm, and the lateral decision behaviors of the lateral behavior decision model include lane keeping and lane changing.

3. The automatic driving lane changing decision control method based on rule fusion reinforcement learning according to claim 1, It is characterized in that The highway driving environment in step S2 includes the number of scene lanes, the width of each lane, the total length of the road, the speed limit of each lane, the number of other vehicles, the lane set where the other vehicles are initialized, the initial speed of each vehicle of the other vehicles, the initial longitudinal position set of the other vehicles, and the speed expansion coefficient of the longitudinal motion control model; The unreasonable traffic flow scenarios in step S2 include: canceling the left lane change when the vehicle is in the leftmost lane, and canceling the right lane change when the vehicle is in the rightmost lane; When the vehicle speed reaches the maximum speed limit, the acceleration action is canceled; when the vehicle speed reaches the minimum speed limit, the deceleration action is canceled; If the distance between the vehicle and the vehicle in front is less than the preset reference following distance, the acceleration command is cancelled.

4. The automatic driving lane changing decision control method based on rule fusion reinforcement learning according to claim 1, It is characterized in that The state space in step S3 includes the state information of the vehicle and the state information of the four other vehicles closest to the vehicle in the curved coordinate system; the state information of the vehicle includes a flag bit that is always set to 1, the longitudinal coordinate s of the vehicle along the reference road, and the state information of the vehicle. 0 , the lateral offset l of the road curve coordinate system 0 , the velocity components v of the vehicle along the reference road and perpendicular to the reference road 0x and v 0y ; The relevant status information of the four other vehicles includes a flag bit of existence, which is 1 if it exists and 0 otherwise, the curve distance Δs relative to the self-vehicle along the reference road, the distance Δl relative to the self-vehicle perpendicular to the reference road curve, and the relative speed Δv along the reference road curve. s , relative speed Δv in the direction perpendicular to the reference road curve l ; The action space in step S3 is divided into discrete decisions of the target lane and discrete decisions of the target speed, including five discrete decisions: left lane change, right lane change, acceleration, deceleration, and IDLE idling without changing lanes or accelerating or decelerating.

5. The automatic driving lane changing decision control method based on rule fusion reinforcement learning according to claim 1, It is characterized in that The specific process of building a deep learning network in step S3 is: First, build a deep learning network structure. The overall structure is a 25*256*256*5 fully connected neural network. The specific layers and settings are: The observation information received by the input layer is a 25*1 vector; Two hidden layers are set in the middle, each containing 256 neurons; The output layer outputs information of 5*1; The activation function uses ReLU and sets the learning rate; Then set the network related configuration as follows: Set the decision cycle of the network; Set the sampling probability of each action so that the sampling probability of actions with high reward values ​​increases with the increase of exploration times; The loss function is calculated using mean square error; Adam is used as the optimizer.

Citation Information

Patent Citations

  • Driving behavior decision-making method for expressway

    CN113253739A

  • Method for realizing safety decision control of autonomous vehicle

    CN114644017A

  • Reinforcement learning based approach for SAE level-4 automated lane change

    US20200189597A1

Cited By

  • Space vector plane and self-control system for intelligent driving

    CN121411143A