Safe driving behavior planning method inspired by defensive driving experience

Abstract

The invention discloses a safe driving behavior planning method inspired by defensive driving experience, and particularly relates to the field of safe driving behavior planning, and the method comprises the steps: S1, initializing a traffic environment and setting a vehicle state based on a Gym library; s2, designing a state space; s3, designing an action space; s4, designing a reward function; and S5, training the algorithm until convergence. Soft safety rules such as defensive driving and road feeling and driving intuition of a human driver are applied to current automatic driving, so that the automatic driving function is more perfect, the motion planning capability of an intelligent vehicle is improved, and the automatic driving system can deal with different emergency road conditions more intelligently.

Description

Technical Field

[0001] The present invention relates to the field of safe driving behavior planning, and more specifically, to a safe driving behavior planning method inspired by defensive driving experience. Background Art

[0002] As the number of cars continues to increase, traffic congestion and traffic accidents are becoming increasingly serious. Therefore, improving traffic congestion and traffic safety issues by implementing intelligent driving strategies has become an important task in the current research of intelligent vehicle technology. Existing driving strategies are mainly based on manually designed rules and risk assessment methods. Although they can meet most conventional driving scenarios, they lack flexibility and applicability and are difficult to deal with emergencies. In recent years, with the rapid development of machine learning technology, learning-based decision-making methods have been widely used to solve intelligent driving strategy problems, but the driving strategies obtained by this method are still far behind human driving strategies.

[0003] Human drivers can rely on their intuition to avoid entering complex driving environments that they cannot handle during driving, and can rely on intuition to predict situations in advance. The "road sense" and "driving intuition" possessed by human drivers are capabilities currently lacking in intelligent vehicle motion planning. The main reason is that current autonomous driving decision-making technology is still dominated by rule-based methods, which make it difficult for such methods to take "road sense" and "driving intuition" into account. In addition, driving decision-making methods based on risk assessment are also used, but such methods mainly use hard safety rules as the main basis for risk assessment, and have not yet applied soft safety rules such as "defensive driving". Summary of the invention

[0004] In order to overcome the above-mentioned defects of the prior art, an embodiment of the present invention provides a safe driving behavior planning method inspired by defensive driving experience. By applying soft safety rules such as "defensive driving" and the "road sense" and "driving intuition" of human drivers to current autonomous driving, the autonomous driving function is further improved, the motion planning capability of intelligent vehicles is increased, and the autonomous driving system is made more intelligent to respond to different emergency road conditions.

[0005] A safe driving behavior planning method inspired by defensive driving experience includes the following steps:

[0006] S1: Initialize the traffic environment and set the vehicle status based on the Gym library;

[0007] S2: Design state space;

[0008] S3: Design action space;

[0009] S4: Design reward function;

[0010] S5: Train the algorithm until convergence.

[0011] Furthermore, in S1, the initialized traffic environment and the set vehicle state constitute an environment for interaction with the intelligent agent for training the intelligent vehicle reinforcement learning algorithm. The specific steps include the following:

[0012] S11: Build straight road scenes and intersection scenes based on the Gym library;

[0013] S12: According to the tasks of each scene, the road and vehicle initialization states of the scene are set, and the motion behavior control framework of the surrounding vehicles is set.

[0014] Furthermore, the motion behavior control framework of the surrounding vehicles in S12 is set up according to different mission scenarios as follows:

[0015] S121: In the straight-ahead scenario, two driving behavior models are used for the control of surrounding vehicles, namely the intelligent driver model and the overall braking minimization model;

[0016] S122: In the intersection scenario, the IDM model is used to control the surrounding vehicles. When constructing the intersection scenario, special attention should be paid to avoid lateral collisions between surrounding vehicles at the intersection. This method adopts the following simplified strategy in the process of constructing the scenario: By using a fixed speed model, each surrounding vehicle predicts the position of its adjacent vehicles in the next 3 seconds. When the risk of collision with an adjacent vehicle is found, it decides whether to give way based on the road priority and braking performance until the collision risk is eliminated.

[0017] Furthermore, in S2, the state space includes information about the main vehicle, information about other vehicles around the main vehicle, and environmental perception information, wherein the information about each vehicle includes location information and driving status information. Specifically:

[0018] S21: Suppose there is a set S, and the joint state information of the set S including the position information and driving state information of the autonomous vehicle s0 and the surrounding N other vehicles is expressed as:

[0019]

[0020] Among them, x and represent the lateral and longitudinal positions of the vehicle along the road direction, respectively; x and represent the lateral and longitudinal speeds of the vehicle along the road direction, respectively; a x and represent the lateral and longitudinal accelerations of the vehicle along the road, respectively; yaw represents the direction of vehicle travel, in order to ensure the uniformity of the designed state space form and the constant length of the output state information vector;

[0021] S23: suppose there is a set Sm representing road information, Sm represents the road information in the form of 0s and 1s in a grid;

[0022] S24: The designed state space is the set of the vehicle's position information, the joint state information of the driving state information and the road information in the environment. The environment state information at time t is expressed as follows:

[0023] s t ={s m ,s v}.

[0024] Furthermore, in S3, the action space is a discrete action, specifically including five possible decision actions: acceleration, deceleration, changing lanes to the left, changing lanes to the right, and maintaining the current state.

[0025] Furthermore, in S4, the reward function includes human-like driving reward, safe driving reward and driving efficiency reward, specifically:

[0026] S41: Humanoid driving reward function collects driving data of skilled human drivers, uses BP neural network for function approximation, and fits the state-action function of skilled human drivers. In this method, BP neural network uses a hidden layer containing 200 neurons, and the hidden layer neurons use Sigmoid transfer function;

[0027] Fit the driver state data to construct a closed graph. If the state of the training model is not within the range of the driver state during training, it indicates that the driving behavior of the model is quite different from that of human drivers, and a negative reward should be given. The farther the model state is from the graph boundary, the greater the gap between the model and the human driver's behavior, and the greater the absolute value of the negative reward that should be given. The square of the minimum distance from the state to the boundary point of the range is used as the reward value, specifically:

[0028] r = -lg{min(drel(t) - drel(i)) 2 +(vrel(t)-vrel(i) 2 )+(vx(t)-vx(i)) 2 ]+1}

[0029] S42: Safe driving reward uses the maximum reaction time T required for the vehicle to avoid collision with the vehicle in front as the driving risk indicator in the scenario, specifically:

[0030]

[0031] D represents the relative distance between the autonomous vehicle and the preceding vehicle, D e represents the distance traveled by the autonomous vehicle from the start of braking to the end of the collision avoidance process, D e represents the distance traveled by the leading vehicle during the collision avoidance process of the autonomous vehicle, and D0 is the expected safety distance.

[0032] Furthermore,

[0033] In this paper, two thresholds d_d and d_a need to be defined to represent the TM value thresholds for danger and potential risk respectively. When the TM value is less than d_d, the danger level is "Dangerous"; when d_d ≤ TM < d_a, the danger level is "Attentive"; when the TM value ≥ d_a, the danger level is "Safe". According to the three defined risk levels, the likelihood function regarding the risk index TM value can be obtained as follows:

[0034]

[0035] In the formula, x represents the value of the risk index T, and d_d represents the threshold of the "Dangerous" risk level. When the risk index is lower than this threshold, the driving risk level of the vehicle will be regarded as "Dangerous". However, the TM value of the risk index itself is only an estimate. Therefore, when constructing the likelihood function, we need to consider the uncertainties existing in the driving environment. In the Bayesian network, the probability inference for different risk levels in the straight - ahead scenario is defined as follows:

[0036]

[0037] In the formula, x represents the TM value of the risk index, σ_d represents the uncertainty in the driving scenario, and d_d and d_a represent the TM value thresholds of the "Dangerous" and "Attentive" risk levels respectively. When TM > 1.45s, there is no potential danger for the leading vehicle, and the leading vehicle has enough time to make decisions and take measures to deal with most possible situations. When 1.45s ≥ TM > 0.05s, the leading vehicle has potential risks and needs to take timely responses to avoid accidents. When 0.05s ≥ TM, the accident may be unavoidable, and only the severity of the accident can be reduced. Therefore, in this paper, the thresholds D_d and D_a are set to 0.05s and 1.45s respectively.

[0038] Furthermore, the safe - driving reward function is:

[0039]

[0040] S43: The reward function r_velocity for driving efficiency is specifically:

[0041]

[0042] Wherein, v_x represents the speed of the intelligent driving vehicle in the road direction; v_max and v_min represent the maximum expected speed and the minimum expected speed of the intelligent driving vehicle in the driving scenario studied in this paper, which are set to 40km / h and 60km / h respectively in the method;

[0043] S44: The immediate reward function is as follows:

[0044] r total =λ1r1+λ2r risk +λ3r velocity

[0045] Where r_1 represents the reward for human-like driving of the ego vehicle; r_risk is the reward for the driving risk of the intelligent vehicle based on the current traffic environment assessment; r_velocity is the reward for evaluating the driving efficiency of the intelligent vehicle; λ_1, λ_2 and λ_3 are the relative weight coefficients of the three partial rewards to the total immediate reward.

[0046] Furthermore, this method normalizes the total instant reward r_total from the value range [λ_1,λ_2+λ_3] to the range [0,1] to standardize the total reward value in each time step, specifically:

[0047]

[0048] The three reward weight coefficients λ_1, λ_2, and λ_3 are set to -1, 0.3, and 0.2 respectively.

[0049] Furthermore, in S5, the network structure design and hyperparameter settings are specifically as follows:

[0050] S51: Based on the DDDQN algorithm, the model in this study consists of a value function network, including a Q value network and a target Q value network. The structures of these two networks are exactly the same, such as Fig.10 As shown, the input of the network comes from the state space, is processed by three fully connected layers, and the output is the distribution of state-action values;

[0051] S52: This method uses the Adam optimizer to train the network and updates the network parameters with a learning rate η;

[0052] S53: In the DDDQN algorithm designed by this method, the output of the neural network is the probability distribution of the Q value of each action. The x-axis represents the standardized Q value, and the y-axis represents the probability that the agent obtains the corresponding Q value after taking the action. In order to model the Q value distribution, a discrete distribution represented by parameters N∈N and V_max, V_min∈R is used. In the process of parameterizing the distribution, the Q value range [V_min, V_max] is discretized into N fulcrums and divided into N-1 equal branch sets. Each branch set is represented as follows:

[0053] The support is fixed, and the neural network outputs the probability that the Q value of each action takes the current support value. The Q value of each action is calculated as follows:

[0054]

[0055] Since the neural network of the DDDQN algorithm outputs the estimated Q value distribution, the training target should also be obtained as a distribution. The Bellman update calculation for each support is as follows:

[0056] Tz i =r t +γz i

[0057] After the Bellman update calculation, the value range of the distribution may exceed the value range of the original distribution. In order to obtain the target probability distribution of each support (m_j,i={0,…,N-1}), it is necessary to project the distribution z_i onto the support of z_i. The specific process is as follows:

[0058]

[0059] In the DDDQN algorithm, since both the estimation network and the target network use distribution as output, this paper adopts a metric that measures the similarity of the two distributions as the loss. Therefore, the loss function of the DDDQN algorithm can be implemented by calculating the cross entropy term of the Kullback-Leibler divergence. Specifically, the loss function is calculated as follows:

[0060]

[0061] Technical effects and advantages of the present invention:

[0062] By applying soft safety rules such as "defensive driving" and the "road sense" and "driving intuition" of human drivers to current autonomous driving, the autonomous driving function will be improved, the motion planning capabilities of intelligent vehicles will be increased, and the autonomous driving system will be more intelligent in responding to different emergency road conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 The present invention is a flow chart of the method.

[0064] Figure 2 This is a diagram of the state information preprocessing process of the present invention.

[0065] Figure 3 This is a diagram of the road information preprocessing process of the present invention.

[0066] Figure 4 This is the environmental status information diagram of the present invention.

[0067] Figure 5 It is a schematic diagram of the BP neural network structure of the present invention.

[0068] Figure 6 It is a schematic diagram of driving data of a skilled human driver according to the present invention.

[0069] Figure 7 It is a schematic diagram of driver data range fitting of the present invention.

[0070] Figure 8 This is a logic diagram of the humanoid driving reward function of the present invention.

[0071] Fig. 9 It is a qualitative explanation diagram of the risk domain and prior probability distribution of the present invention.

[0072] Fig.10 This is a structural diagram of the fully connected neural network of the present invention.

[0073] Fig.11 This is a framework diagram of the DDDQN algorithm of the present invention.

[0074] Fig.12 It is a convergence diagram of the present invention. DETAILED DESCRIPTION

[0075] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0076] according to Figure 1-12 A safe driving behavior planning method inspired by defensive driving experience is shown, characterized in that it includes the following steps:

[0077] S1: Initialize the traffic environment and set the vehicle status based on the Gym library;

[0078] S2: Design state space;

[0079] S3: Design action space;

[0080] S4: Design reward function;

[0081] S5: Train the algorithm until convergence.

[0082] Furthermore, in S1, the initialized traffic environment and the set vehicle state constitute an environment for interaction with the intelligent agent for training the intelligent vehicle reinforcement learning algorithm. The specific steps include the following:

[0083] S11: Build straight road scenes and intersection scenes based on the Gym library;

[0084] S12: According to the tasks of each scene, the road and vehicle initialization states of the scene are set, and the motion behavior control framework of the surrounding vehicles is set.

[0085] Furthermore, the motion behavior control framework of the surrounding vehicles in S12 is set up according to different mission scenarios as follows:

[0086] S121: In the straight driving scenario, two driving behavior models are used to control surrounding vehicles, namely the Intelligent driver model (IDM) and the Minimizing Overall Braking Induced by Lane changes (MOBIL) model.

[0087] S122: In the intersection scenario, the IDM model is used to control the surrounding vehicles. When constructing the intersection scenario, special attention should be paid to avoid lateral collisions between surrounding vehicles at the intersection. This method adopts the following simplified strategy in the process of constructing the scenario: By using a fixed speed model, each surrounding vehicle predicts the position of its adjacent vehicles in the next 3 seconds. When the risk of collision with an adjacent vehicle is found, it decides whether to give way based on the road priority and braking performance until the collision risk is eliminated.

[0088] Furthermore, in S2, the state space includes information about the main vehicle, information about other vehicles around the main vehicle, and environmental perception information, wherein the information about each vehicle includes location information and driving status information. Specifically:

[0089] S21: Suppose there is a set S, and the joint state information of the set S including the position information and driving state information of the autonomous vehicle s0 and the surrounding N other vehicles is expressed as:

[0090]

[0091] Among them, x and represent the lateral and longitudinal positions of the vehicle along the road direction, respectively; x and represent the lateral and longitudinal speeds of the vehicle along the road direction, respectively; a x and represent the lateral and longitudinal accelerations of the vehicle along the road, respectively; yaw represents the direction of vehicle travel. In order to ensure the uniformity of the designed state space form and the constant length of the output state information vector, this paper preprocesses the selected state information. Specifically, Figure 2 As shown, the map information is gridded, and the vehicle status information in the environment is represented in the grid in the form of a tensor;

[0092] S23: Suppose there is a set Sm representing road information. Sm represents the road information in the form of 0 and 1 in a grid. Specifically, Figure 3 As shown;

[0093] S24: The designed state space is the set of the vehicle's position information, the joint state information of the driving state information and the road information in the environment. The environment state information at time t is expressed as follows:

[0094] s t ={s m ,s v}.

[0095] Furthermore, in S3, the action space is a discrete action, specifically including five possible decision actions: acceleration, deceleration, changing lanes to the left, changing lanes to the right, and maintaining the current state.

[0096] Furthermore, in S4, the reward function includes human-like driving reward, safe driving reward and driving efficiency reward, specifically:

[0097] S41: Humanoid driving reward function collects driving data of skilled human drivers, uses BP neural network for function approximation, and fits the state action function of skilled human drivers (the mapping function of the relative position, relative speed, vehicle speed, and vehicle acceleration of the vehicle and surrounding vehicles). In this method, the BP neural network uses a hidden layer containing 200 neurons, and the hidden layer neurons use Sigmoid transfer function, as shown in the following example: Figure 5 As shown;

[0098] Fit the driver state data to construct a closed graph such as Figure 7 As shown in Figure 2, if during the training process, the state of the training model is not within the range of the driver state (i.e., the state is within Figure 7), it indicates that the driving behavior of the model is quite different from that of human drivers, and a negative reward should be given. The farther the model state is from the boundary of the graph, the greater the difference between the model and human drivers’ behavior, and the greater the absolute value of the negative reward should be. The square of the minimum distance from the state to the boundary point of the range is used as the reward value, which is:

[0099] r = -lg{min[(drel(t) - drel(i)) 2 +(vrel(t)-vrel(i) 2 )+(vx(t)-vx(i)) 2 ]+1}

[0100] S42: Safe driving reward uses the maximum reaction time T required for the vehicle to avoid collision with the vehicle in front as the driving risk indicator in the scenario, specifically:

[0101]

[0102] D represents the relative distance between the autonomous vehicle and the preceding vehicle, D e represents the distance traveled by the autonomous vehicle from the start of braking to the end of the collision avoidance process, D e represents the distance traveled by the leading vehicle during the collision avoidance process of the autonomous vehicle, and D0 is the expected safety distance;

[0103] According to the motion state of the autonomous vehicle and other surrounding vehicles, D e and D f The calculation process is divided into three different cases:

[0104] (1) When the speed of the leading vehicle is lower than that of the ego vehicle and the leading vehicle continues to travel at a constant speed, in this scenario, the boundary condition for the two vehicles to avoid collision is that when the ego vehicle's speed decreases to the same speed as the leading vehicle, the relative distance between the two vehicles is greater than the expected safety distance. In this case, D e and D f The calculation of is as follows:

[0105]

[0106] In the formula, t bc Represents the braking coordination time, and its value varies depending on the vehicle model. In this method, t bc The value is 0.15s, t br Indicates the braking speed growth time, the value is 0.45s, a emax Indicates the maximum braking deceleration of the autonomous vehicle, which is taken as a emax =-0.6g;

[0107] (2) In the scenario where the vehicle ahead is decelerating, the boundary condition for avoiding a collision is to slow down the two vehicles to the same speed and ensure that the relative distance between the two vehicles is greater than the desired safety distance. Assume that in this scenario, the two vehicles have the same speed v p , the calculation process is as follows:

[0108] In the above formula, a f represents the braking deceleration of the vehicle ahead. In this case, if v f a emax >v e a f , the autonomous vehicle will brake to a stop first, and in this case, at the same speed v p >0, so in this case, D e and D f The calculation of is as follows:

[0109]

[0110] If v f a emax ≤v e a f , the vehicle in front should brake to a stop first, at the same speed v p =0, so in this case, D e and D f The calculation method is as follows:

[0111]

[0112] The front vehicle is traveling at a slower speed than the autonomous vehicle and is accelerating. In this case, to avoid a collision, the front vehicle needs to accelerate to the same speed as the autonomous vehicle, and the relative distance between the two vehicles needs to be greater than the desired safety distance. Assume that the speeds of the two vehicles in this case are v q , the calculation process is as follows;

[0113]

[0114] In addition to the above three situations, there are other situations such as the front vehicle's speed is greater than the autonomous vehicle's speed. However, in these situations, the front vehicle does not pose a threat to the autonomous vehicle's driving. Therefore, in this study, in addition to the above three situations, the autonomous vehicle is considered to be in a safe driving state.

[0115] In a straight - driving scenario, the autonomous vehicle analyzes the motion states of itself and surrounding vehicles to determine whether the current traffic situation meets one of the above - mentioned three situations. If not, it is considered that the autonomous vehicle is in a safe driving state; if so, it further evaluates the current traffic situation based on the motion states of itself and surrounding vehicles, and evaluates the potential driving risk of the autonomous vehicle by calculating the T - value.

[0116] To quantitatively evaluate the risk, three risk levels related to the T - value are defined to represent the degree of risk. Each level clearly indicates the correlation with the risk of possible collision, specifically:

[0117] z ∈ Z = {Dangerous, Attentive, Safe} = {D, A, S}

[0118] where z represents a random variable of the risk - degree level of the autonomous vehicle with respect to surrounding vehicles. The variable "Dangerous" represents danger, the variable "Attentive" represents potential risk, and "Safe" represents safety.

[0119] Furthermore, this paper needs to define two thresholds d_d and d_a to represent the TM - value thresholds of danger and potential risk respectively. As Fig. 9 shown, when the TM - value is less than d_d, the danger level is "Dangerous"; when d_d ≤ TM < d_a, the danger level is "Attentive"; when the TM - value ≥ d_a, the danger level is "Safe". According to the defined three risk levels, the likelihood function of the risk index TM - value can be obtained as follows:

[0120]

[0121] In the formula, x represents the value of the risk index T, and d_d represents the threshold of the "Dangerous" risk level. When the risk index is lower than this threshold, the driving risk level of the vehicle will be regarded as "Dangerous". However, the risk index TM - value itself is only an estimate. Therefore, when constructing the likelihood function, we need to consider the uncertainty existing in the driving environment. In the Bayesian network, the probability inference of different risk levels in the straight - driving scenario is defined as follows:

[0122]

[0123] Where x represents the risk index TM value, σ_d represents the uncertainty in the driving scenario, d_d and d_a represent the TM value thresholds of the "Dangerous" and "Attentive" risk levels, respectively. When TM>1.45s, the main vehicle is not in potential danger, and the main vehicle has enough time to make decisions and take measures to deal with most possible situations. When 1.45s≥TM>0.05s, the main vehicle is at potential risk and needs to respond in time to avoid accidents. When 0.05s≥TM, the accident may not be avoided, but the severity of the accident can only be reduced. Therefore, this paper sets the thresholds D_d and D_a to 0.05s and 1.45s, respectively.

[0124] Assuming that the prior probability P(z) of the vehicle risk level follows a uniform distribution, its posterior probability can be calculated as follows:

[0125]

[0126] In the formula, in order to quantify the risk level, the values ​​2, 1 and 0 are defined to represent the level of risk level in variable z. The specific risk level value ε is defined as follows:

[0127] ε∈Z={Dangerous, Attentive, Safe}={2,1,0}

[0128] Assuming that the prior probability P(z) of the vehicle risk level follows a uniform distribution, the expected risk level ε_risk of the autonomous vehicle at the current moment can be calculated as follows:

[0129]

[0130] Furthermore, the safe driving reward function is:

[0131]

[0132] S43: driving efficiency reward function r_velocity, specifically:

[0133]

[0134] Wherein, v_x represents the speed of the intelligent driving vehicle in the road direction; v_max and v_min represent the maximum expected speed and the minimum expected speed of the intelligent driving vehicle in the driving scenario studied in this paper, which are set to 40km / h and 60km / h respectively in the method;

[0135] S44: The immediate reward function is as follows:

[0136] r total =λ1r1+λ2r risk+λ3r vrlocity

[0137] Where r_1 represents the reward for human-like driving of the ego vehicle; r_risk is the reward for the driving risk of the intelligent vehicle based on the current traffic environment assessment; r_velocity is the reward for evaluating the driving efficiency of the intelligent vehicle; λ_1, λ_2 and λ_3 are the relative weight coefficients of the three partial rewards to the total immediate reward.

[0138] Furthermore, this method normalizes the total instant reward r_total from the value range [λ_1,λ_2+λ_3] to the range [0,1] to standardize the total reward value in each time step, specifically:

[0139]

[0140] The three reward weight coefficients λ_1, λ_2, and λ_3 are set to -1, 0.3, and 0.2 respectively.

[0141] Furthermore, in S5, the network structure design and hyperparameter settings are specifically as follows:

[0142] S51: Based on the DDDQN algorithm, the model in this study consists of a value function network, including a Q value network and a target Q value network. The structures of these two networks are exactly the same, such as Fig.10 As shown, the input of the network comes from the state space, is processed by three fully connected layers, and the output is the distribution of state-action values;

[0143] S52: This method uses the Adam optimizer to train the network and updates the network parameters with a learning rate η. All hyperparameters are detailed in Table 1;

[0144] Table 1 Network training hyperparameters

[0145]

[0146] S53: In the DDDQN algorithm designed by this method, the output of the neural network is the probability distribution of the Q value of each action, such as Fig.11 As shown in the figure, the x-axis represents the standardized Q value, and the y-axis represents the probability that the agent obtains the corresponding Q value after taking an action. In order to model the Q value distribution, we use a discrete distribution represented by parameters N∈N and V_max, V_min∈R. In the process of parameterizing the distribution, the Q value range [V_min, V_max] is discretized into N fulcrums and divided into N-1 equal branch sets. Each branch set is represented as follows:

[0147] The support is fixed, and the neural network outputs the probability that the Q value of each action takes the current support value. The Q value of each action is calculated as follows:

[0148]

[0149] Since the neural network of the DDDQN algorithm outputs the estimated Q value distribution, the training target should also be obtained as a distribution. The Bellman update calculation for each support is as follows:

[0150] Tz i =r t +γz i

[0151] After the Bellman update calculation, the value range of the distribution may exceed the value range of the original distribution. In order to obtain the target probability distribution of each support (m_j,i={0,…,N-1}), it is necessary to project the distribution z_i onto the support of z_i. The specific process is as follows:

[0152]

[0153] In the DDDQN algorithm, since both the estimation network and the target network use distribution as output, this paper adopts a metric that measures the similarity of the two distributions as the loss. Therefore, the loss function of the DDDQN algorithm can be implemented by calculating the cross entropy term of the Kullback-Leibler divergence. Specifically, the loss function is calculated as follows:

[0154]

[0155] Specifically, the training process of the DDDQN algorithm is shown in Table 2.

[0156] Table 2 Pseudocode of DDDQN algorithm

[0157]

Claims

1. A safe driving behavior planning method inspired by defensive driving experience, characterized by: The following steps are involved: S1: Initialize the traffic environment and set the vehicle status based on the Gym library; S2: Design state space; S3: Design action space; S4: Design reward function; S5: Train the algorithm until convergence.

2. The method for safe driving behavior planning inspired by defensive driving experience according to claim 1, characterized in that: In S1, the initialized traffic environment and the set vehicle state constitute an environment for interaction with the agent for training the intelligent vehicle reinforcement learning algorithm. The specific steps include the following: S11: Build straight road scenes and intersection scenes based on the Gym library; S12: According to the tasks of each scene, the road and vehicle initialization states of the scene are set, and the motion behavior control framework of the surrounding vehicles is set.

3. The method for safe driving behavior planning inspired by defensive driving experience according to claim 2, characterized in that: The motion behavior control framework of surrounding vehicles in S12 is set up according to different mission scenarios as follows: S121: In the straight-ahead scenario, two driving behavior models are used to control surrounding vehicles, namely the intelligent driver model and the overall braking minimization model; S122: In the intersection scenario, the IDM model is used to control the surrounding vehicles. When constructing the intersection scenario, special attention should be paid to avoid lateral collisions between surrounding vehicles at the intersection. This method adopts the following simplified strategy in the process of constructing the scenario: By using a fixed speed model, each surrounding vehicle predicts the position of its adjacent vehicles in the next 3 seconds. When the risk of collision with an adjacent vehicle is found, it decides whether to give way based on the road priority and braking performance until the collision risk is eliminated.

4. The method for safe driving behavior planning inspired by defensive driving experience according to claim 1, characterized in that: In S2, the state space includes the information of the main vehicle, the information of other vehicles around the main vehicle, and the environmental perception information, where the information of each vehicle includes the location information and driving status information. Specifically: S21: Suppose there is a set S, and the joint state information of the set S including the position information and driving state information of the autonomous vehicle s0 and the surrounding N other vehicles is expressed as: Among them, x and represent the lateral and longitudinal positions of the vehicle along the road direction, respectively; x and represent the lateral and longitudinal speeds of the vehicle along the road direction, respectively; a x and represent the lateral and longitudinal accelerations of the vehicle along the road, respectively; yaw represents the direction of vehicle travel, in order to ensure the uniformity of the designed state space form and the constant length of the output state information vector; S23: suppose there is a set Sm representing road information, Sm represents the road information in the form of 0s and 1s in a grid; S24: The designed state space is the set of the vehicle's position information, the joint state information of the driving state information and the road information in the environment. The environment state information at time t is expressed as follows: S t ={s m ,s v )。 5. The method for safe driving behavior planning inspired by defensive driving experience according to claim 1, characterized in that: In S3, the action space is a discrete action, specifically including five possible decision actions: acceleration, deceleration, changing lanes to the left, changing lanes to the right, and maintaining the current state.

6. The method for safe driving behavior planning inspired by defensive driving experience according to claim 1, characterized in that: In S4, the reward function includes human-like driving reward, safe driving reward and driving efficiency reward, specifically: S41: Humanoid driving reward function collects driving data of skilled human drivers, uses BP neural network for function approximation, and fits the state-action function of skilled human drivers. In this method, BP neural network uses a hidden layer containing 200 neurons, and the hidden layer neurons use Sigmoid transfer function; The driver state data is fitted to construct a closed graph. If, during the training process, the state of the training model is not within the range of the driver state, it indicates that the driving behavior of the model is quite different from that of a human driver, and a negative reward should be given. The farther the state of the model is from the boundary of the graph, the greater the gap between the behavior of the model and that of the human driver, and the greater the absolute value of the negative reward to be given. The square of the minimum distance from the state to the boundary point of the range is used as the reward value, specifically as follows: r=-lg{min[(drel(t)-drel(i)) 2 +(vrel(t)-vrel(i) 2 )+(vx(t)-vx(i)) 2 ]+1} S42: The safe driving reward uses the maximum reaction time T required for the host vehicle to avoid a collision with the vehicle ahead as the driving risk index in the scenario, specifically as follows: D represents the relative distance between the autonomous vehicle and the preceding vehicle, D e represents the distance traveled by the autonomous vehicle from the start of braking to the end of the collision avoidance process, D e represents the distance traveled by the leading vehicle during the collision avoidance process of the autonomous vehicle, and D0 is the expected safety distance.

7. A method for planning safe driving behavior inspired by defensive driving experience according to claim 1, characterized in that: In this paper, two thresholds d_d and d_a are defined to represent the TM value thresholds for danger and potential risk respectively. When the TM value is less than d_d, the danger level is "Dangerous"; when d_d ≤ TM < d_a, the danger level is "Attentive"; when the TM value ≥ d_a, the danger level is "Safe". According to the three defined risk levels, the likelihood function regarding the risk index TM value can be obtained as follows: In the formula, x represents the value of the risk index T, and d_d represents the threshold for the "Dangerous" risk level. When the risk index is lower than this threshold, the driving risk level of the vehicle will be regarded as "Dangerous". However, the risk index TM value itself is only an estimate. Therefore, when constructing the likelihood function, we need to consider the uncertainty existing in the driving environment. In the Bayesian network, the probability inference for different risk levels in the straight - ahead scenario is defined as follows: In the formula, x represents the risk index TM value, σ_d represents the uncertainty in the driving scenario, and d_d and d_a represent the TM value thresholds for the "Dangerous" and "Attentive" risk levels respectively. When TM > 1.45s, there is no potential danger for the host vehicle, and the host vehicle has enough time to make decisions and take measures to deal with most possible situations. When 1.45s ≥ TM > 0.05s, there is a potential risk for the host vehicle, and timely reactions need to be taken to avoid accidents. When 0.05s ≥ TM, the accident may be inevitable, and only the severity of the accident can be reduced. Therefore, in this paper, the thresholds D_d and D_a are set to 0.05s and 1.45s respectively.

8. The method for safe driving behavior planning inspired by defensive driving experience according to claim 1, characterized in that: The safe driving reward function is: S43: The reward function r_velocity for driving efficiency, specifically as follows: Among them, v_x represents the driving speed of the intelligent driving vehicle in the road direction; v_max and v_min represent the maximum expected vehicle speed and the minimum expected vehicle speed of the intelligent driving vehicle in the driving scenario studied in this paper, which are set to 40 km / h and 60 km / h respectively in the method; S44: The immediate reward function is specifically as follows: r total =λ1r1+λ2r risk +λ3r velocity Where r_1 represents the reward for human-like driving of the ego vehicle; r_risk is the reward for the driving risk of the intelligent vehicle based on the current traffic environment assessment; r_velocity is the reward for evaluating the driving efficiency of the intelligent vehicle; λ_1, λ_2 and λ_3 are the relative weight coefficients of the three partial rewards to the total immediate reward.

9. The method for safe driving behavior planning inspired by defensive driving experience according to claim 1, characterized in that: This method normalizes the total instant reward r_total from the value range [λ_1,λ_2+λ_3] to the range [0,1] to standardize the total reward value in each time step, specifically: The three reward weight coefficients λ_1, λ_2, and λ_3 are set to -1, 0.3, and 0.2 respectively.

10. The method for safe driving behavior planning inspired by defensive driving experience according to claim 1, characterized in that: In S5, the network structure design and hyperparameter settings are as follows: S51: Based on the DDDQN algorithm, the model in this study consists of a value function network, including a Q value network and a target Q value network. The structures of these two networks are exactly the same, as shown in Figure 10. The input of this network comes from the state space, which is processed by three fully connected layers, and the output is the distribution of state-action values; S52: This method uses the Adam optimizer to train the network and updates the network parameters with a learning rate η; S53: In the DDDQN algorithm designed by this method, the output of the neural network is the probability distribution of the Q value of each action. The x-axis represents the standardized Q value, and the y-axis represents the probability that the agent obtains the corresponding Q value after taking the action. In order to model the Q value distribution, a discrete distribution represented by parameters N∈N and V_max, V_min∈R is used. In the process of parameterizing the distribution, the Q value range [V_min, V_max] is discretized into N fulcrums and divided into N-1 equal branch sets. Each branch set is represented as follows: The support is fixed, and the neural network outputs the probability that the Q value of each action takes the current support value. The Q value of each action is calculated as follows: Since the neural network of the DDDQN algorithm outputs the estimated Q value distribution, the training target should also be obtained as a distribution. The Bellman update calculation for each support is as follows: Tz i =r t +γz i After the Bellman update calculation, the value range of the distribution may exceed the value range of the original distribution. In order to obtain the target probability distribution of each support (m_j,i={0,…,N-1}), it is necessary to project the distribution z_i onto the support of z_i. The specific process is as follows: In the DDDQN algorithm, since both the estimation network and the target network use distribution as output, this paper adopts a metric that measures the similarity of the two distributions as the loss. Therefore, the loss function of the DDDQN algorithm can be implemented by calculating the cross entropy term of the Kullback-Leibler divergence. Specifically, the loss function is calculated as follows:

Images