Decision-making planning method for autonomous driving vehicles based on value distribution reinforcement learning

Abstract

The present invention relates to a decision-making planning method for an autonomous driving vehicle based on value distribution reinforcement learning, and belongs to the field of autonomous driving vehicles. The method comprises: S1: constructing a scene of a non-signal intersection taking into account uncertainty; S2: constructing a fully parameterized quantile function model as a control model for an autonomous driving vehicle; S3: introducing conditional risk value based on the state-action reward distribution information learned in the fully parameterized quantile function model to generate a driving behavior with risk awareness. The present invention uses value distribution reinforcement learning to improve the safety and stability of the decision-making planning strategy of an autonomous driving vehicle in an environment with uncertainty.

Description

Technical Field

[0001] The present invention belongs to the field of autonomous driving vehicles and relates to a decision-making planning method for autonomous driving vehicles based on value distribution reinforcement learning. Background Art

[0002] Autonomous driving technology has made rapid progress in recent years, but safety has become a key issue facing autonomous driving technology. Safety is an important factor hindering the commercialization of autonomous vehicles and has also been a research hotspot in recent years. The autonomous driving decision-making planning module, as the "brain" of autonomous vehicles, has a significant impact on the safety of autonomous vehicles, especially in complex urban scenarios such as intersections. How to make autonomous and safe decisions has been widely studied in recent years.

[0003] The decision-making and planning module of an autonomous vehicle mainly generates the optimal driving behavior based on the current environmental state, so as to complete the driving task safely. The existing decision-making and planning methods are mainly divided into three categories: rule-based, optimization-based, and learning-based. Among them, the rule-based method is only applicable to specific scenarios; the optimization-based method has poor performance in real-time. Therefore, the learning-based method has been widely studied by academia and industry in recent years. Among them, reinforcement learning has been widely used in the decision-making and planning problems of autonomous vehicles. Thanks to the real-time and scene adaptability of reinforcement learning, the decision-making and planning method based on reinforcement learning can complete the driving task well. However, due to the increasingly complex driving environment faced by autonomous vehicles, incomplete perception caused by bad weather, building occlusion, etc., and the behavioral uncertainty of surrounding traffic participants, the safety of autonomous vehicles has brought great challenges. Traditional reinforcement learning algorithms can no longer meet the safety requirements of autonomous vehicles.

[0004] Since traditional reinforcement learning selects the optimal action by maximizing the expected value of the reward, the distribution information of the reward is largely lost, and therefore the impact of the inherent uncertainty in the environment on the decision-making strategy cannot be considered. Therefore, it is urgent to propose a new reinforcement learning algorithm to deal with the uncertainty in the environment to improve the safety of decision-making planning for autonomous vehicles. Summary of the invention

[0005] In view of this, the purpose of the present invention is to provide a decision-making planning method for an autonomous driving vehicle based on value distribution reinforcement learning, which can improve the safety and stability of the decision-making planning strategy of the autonomous driving vehicle in an uncertain environment.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] A decision planning method for an autonomous driving vehicle based on value distribution reinforcement learning specifically includes the following steps:

[0008] S1: Construct a non-signal intersection scenario considering uncertainty;

[0009] S2: Construct a fully parameterized quantile function (FQF) network model as a control model for autonomous vehicles;

[0010] S3: Based on the state-action reward distribution information learned in the fully parameterized quantile function (FQF) model, the conditional value-at-risk (CVaR) is introduced to generate risk-aware driving behaviors.

[0011] Furthermore, in step S1, a scene of a non-signalized intersection taking uncertainty into consideration is constructed, specifically including: establishing an occlusion model, determining a surrounding vehicle model, and establishing a surrounding vehicle type distribution.

[0012] Furthermore, in step S1, an occlusion model is established, specifically including: considering the occlusion on both sides of the intersection, by analyzing the relative position relationship between the surrounding vehicles and the vehicle and the center of the intersection, according to the geometric relationship, the critical distance d at which the surrounding vehicles can be observed by the vehicle is calculated, which is used as the critical condition for judging whether the surrounding vehicles are blocked:

[0013]

[0014] Where l is the width of each lane, d′ is the distance from the front of the vehicle to the center of the intersection, is the distance from the road boundary to the obstruction, and d is the distance from the front of the surrounding vehicles to the center of the intersection.

[0015] Further, in step S1, the surrounding vehicle model is determined, specifically including: in order to enable the surrounding vehicles to respond to active changes in the environment, it is stipulated that the behavior of the surrounding vehicles in the simulation environment is controlled by the intelligent driver model (IntelligentDriver Model):

[0016]

[0017]

[0018] Where a is the acceleration, a max is the maximum acceleration, v is the longitudinal velocity of the vehicle, v target is the expected longitudinal velocity of the vehicle, m is the acceleration parameter, d target is the expected longitudinal distance of the vehicle, d0 is the minimum longitudinal distance of the vehicle, T0 is the minimum collision time of the vehicle, and Δv is the relative speed to the vehicle in front.

[0019] Furthermore, in step S1, the distribution of surrounding vehicle types is established, specifically including: in the simulation environment, it is stipulated that the surrounding vehicles include three types: aggressive, conservative, and normal. Each type of vehicle is added to the environment with a certain probability at each time step. The surrounding vehicle type space is:

[0020]

[0021] Further, in step S2, a fully parameterized quantile function model is constructed, which specifically includes the following steps:

[0022] S21: Construct a Fraction proposal network: use state information as network input and output the optimal fraction τ corresponding to each state-action;

[0023] S22: Construct a quantile value network: Use the optimal quantile obtained by the quantile proposal network as the input of the quantile value network, and map the quantile function values ​​corresponding to each quantile in the current state;

[0024] S23: Construct state space S: The position, speed, heading angle of surrounding vehicles and the position, speed and heading angle of the self-vehicle are used as the observable state information of the self-vehicle, and the value distribution reinforcement learning is used to make the next decision planning based on the self-vehicle observation information;

[0025] S24: Constructing action space A: The action space is defined as the set of executable actions of the ego vehicle, which is the output value of the value distribution reinforcement learning network. Here, the action space of the ego vehicle includes three discrete action values: acceleration, cruising, and deceleration. The specific accelerations of the two actions of acceleration and deceleration are calculated by the intelligent driver model.

[0026] S25: Design reward function, the total reward is equal to the collision reward R collision , Reward for completing the task R success and timeout reward R timeout The sum of three parts;

[0027] S26: According to the current state S t , perform action A t , the training data (S t ,A t ,R t ,S t+1 ) is added to the experience pool;

[0028] S27: Fitted return distribution;

[0029] S28: Update the quantile proposal network: Update the quantile proposal network by minimizing the 1-Wasserstein distance to determine the optimal quantile τ so that the distribution it fits is closer to the true distribution;

[0030] S29: Update the quantile numerical network: The update goal of the quantile numerical network is to minimize the quantile regression Huber-loss, make the output of the quantile numerical network as close to the target value as possible, and update the quantile numerical network using the gradient descent method.

[0031] Further, step S27 specifically includes: fitting the distribution of returns through weighted values ​​of N mixed Dirac functions:

[0032]

[0033] Where N is the number of quantiles, τ i is the quantile point generated by the quantile proposal network, satisfying τ i-1 <τ i , and τ0=0, τ N =1,δ θi(s,a) is the parameter θ in the current state (s, a) i The Dirac function.

[0034] Further, step S28 specifically includes the following steps:

[0035] S281: 1-Wasserstein distance formula is:

[0036]

[0037] Where N is the number of quantiles, ω is the neural network parameter, is the quantile The corresponding quantile function value is,

[0038] S282: Due to the true quantile function In fact, it is impossible to obtain, so the quantile value function with the quantile network parameter ω2 is used As a function of the true quantile value in the current state;

[0039] S283: In order to avoid directly calculating the 1-Wasserstein distance, the quantile proposal network parameter ω1 is used to minimize the 1-Wasserstein distance using gradient descent:

[0040]

[0041] S284: The expected return of the fully parameterized quantile function is:

[0042] Further, step S29 specifically includes the following steps:

[0043] S291: Solving time difference equations:

[0044]

[0045] Among them, δ ij is TD-error, r t is the return at the current moment, γ is the attenuation factor, Z is the return distribution at the current moment, and Z′ is the return distribution at the next moment;

[0046] S292: Calculate the quantile regression Huber-loss:

[0047]

[0048]

[0049] in, is the quantile regression Huber-loss, is the Huber-loss function, κ is the threshold;

[0050] S293: Update the quantile value network using stochastic gradient descent:

[0051]

[0052] in, is the TD-error at time t.

[0053] Further, step S3 specifically includes the following steps:

[0054] S31: Based on the return distribution information obtained in the fully parameterized quantile function (FQF) model in step S2, the conditional value at risk (CVaR) corresponding to each distribution is calculated as:

[0055]

[0056] Among them, the value at risk (VaR): Z is the distribution of returns, α is the cumulative probability, and R is the return, which is a random variable;

[0057] S32: Select the best action to maximize the CVaR value and select the best risk-sensitive behavior:

[0058]

[0059] in, is the current state t The optimal action selected under , Z is the distribution of rewards, and α is the cumulative probability.

[0060] The beneficial effects of the present invention are:

[0061] 1) The present invention designs a simulation training environment for a non-signal intersection, taking into account the incomplete perception caused by occlusion in the environment and the behavioral uncertainty of surrounding traffic participants, making the scene more consistent with the real driving scene.

[0062] 2) The present invention designs a decision planning method based on value distribution reinforcement learning, which adopts a fully parameterized quantile function (FQF) to more accurately fit the value distribution and provide more accurate distribution information for the subsequent risk-aware decision-making behavior generation.

[0063] 3) The present invention designs a behavior generation method based on conditional value at risk (CVaR), which generates risk-aware driving behavior based on the obtained reward distribution information and taking into account the uncertainty in the environment.

[0064] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0066] Figure 1 This is the overall logical framework diagram of the autonomous driving vehicle decision planning method based on value distribution reinforcement learning of the present invention;

[0067] Figure 2 A logical framework diagram for building a simulation training environment;

[0068] Figure 3 This is the network structure diagram of the fully parameterized quantile function (FQF). DETAILED DESCRIPTION

[0069] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0070] See also Figure 1 to Figure 3 , the present invention provides a decision-making planning method for an autonomous driving vehicle based on value distribution reinforcement learning. Taking into account the uncertainty existing in the real driving environment, a simulation training environment for intersections without signal lights that simultaneously considers occlusion and different driver types is established. At the same time, taking into account the safety requirements of autonomous driving vehicle decision-making planning, a method based on value distribution reinforcement learning is proposed. The true distribution of returns is fitted by a fully parameterized quantile function (FQF), and then the conditional value at risk (CVaR) is introduced into the obtained distribution information to generate risk-aware driving behaviors, thereby improving the autonomous driving vehicle's ability to handle uncertainties in the environment. The method specifically includes the following steps:

[0071] Step S1: Construct a simulation training scene of an intersection without traffic lights, such as Figure 2 As shown, the specific steps include:

[0072] S11: Establish an occlusion model: Consider the occlusion on both sides of the intersection. By analyzing the relative position relationship between the surrounding vehicles, the vehicle itself, and the center of the intersection, the critical distance d at which the surrounding vehicles can be observed by the vehicle itself is calculated based on the geometric relationship. This is used as the critical condition for determining whether the surrounding vehicles are occluded:

[0073]

[0074] Where l is the width of each lane, d′ is the distance from the front of the vehicle to the center of the intersection, is the distance from the road boundary to the obstruction, and d is the distance from the front of the surrounding vehicles to the center of the intersection.

[0075] S12: Determine the surrounding vehicle model: In order to enable the surrounding vehicles to respond accordingly to changes in the environment, it is stipulated that the behavior of the surrounding vehicles in the simulation environment is controlled by the intelligent driver model (Intelligent Driver Model):

[0076]

[0077]

[0078] Where a is the acceleration, a max is the maximum acceleration, v is the longitudinal velocity of the vehicle, v target is the expected longitudinal velocity of the vehicle, m is the acceleration parameter, d target is the expected longitudinal distance of the vehicle, d0 is the minimum longitudinal distance of the vehicle, T0 is the minimum collision time of the vehicle, and Δv is the relative speed to the vehicle in front.

[0079] S13: Establish the distribution of surrounding vehicle types: In order to enable the ego vehicle to make different decisions according to different driver types, it is stipulated that in the simulation environment, the surrounding vehicles include three types: aggressive, conservative, and normal. Each type of vehicle has a probability of: P aggressive =0.2,P conservative =0.3,P normal = 0.5 is added to the environment, and the surrounding vehicle type space is:

[0080]

[0081] S14: Initialize the environment: Randomly initialize the initial speed, position and target speed of the surrounding vehicles.

[0082] S2: Construct a fully parameterized quantile function (FQF) model as the control model for the autonomous vehicle, such as Figure 3 As shown, the specific steps include:

[0083] S21: Construct a Fraction proposal network: Use state information as network input and output the optimal fraction τ corresponding to each state-action.

[0084] S22: Construct a quantile value network: Use the optimal quantile obtained by the quantile proposal network as the input of the quantile value network, and map the quantile function values ​​corresponding to each quantile in the current state.

[0085] S23: Construct state space S: The position, speed, heading angle of surrounding vehicles and the position, speed and heading angle of the own vehicle are used as the observable state information of the own vehicle, and value distribution reinforcement learning is used to make the next decision plan based on the observation information of the own vehicle.

[0086]

[0087] Among them, i=0 represents the ego vehicle, i∈[1,N] represents the surrounding vehicles, and x i ,y irepresents the lateral and longitudinal position of the vehicle, v xi ,v yi represents the lateral and longitudinal velocities of the vehicle, Represents the heading angle of the vehicle.

[0088] S24: Constructing action space A: The action space is defined as the set of executable actions of the ego vehicle, which is the output value of the value distribution reinforcement learning network. Here, the action space of the ego vehicle includes acceleration, cruising, and deceleration, where the specific acceleration of the two actions of acceleration and deceleration is calculated by the intelligent driver model:

[0089]

[0090]

[0091] Where a is the acceleration, a max is the maximum acceleration, v is the longitudinal velocity of the vehicle, v target is the expected longitudinal velocity of the vehicle, m is the acceleration parameter, d target is the expected longitudinal distance of the vehicle, d0 is the minimum longitudinal distance of the vehicle, T0 is the minimum collision time of the vehicle, Δv is the relative speed with the front vehicle, and the acceleration range is: a∈[-3, 1]m 2 / s .

[0092] S25: Design reward function: The reward function mainly considers safety R collision , success rate R success And the efficiency R timeout The sum of the three parts, namely:

[0093] R=R collision +R success +R timeout

[0094] The first term R collision For collision rewards, the ego vehicle is required not to collide with surrounding vehicles;

[0095] R collision =-10

[0096] The second term R success To receive the reward for completing the task, the vehicle is required to reach the target location without collision;

[0097] R success =10

[0098] The third item R timeout For the overtime reward, the car is required not to exceed the maximum number of steps in the specified round.

[0099] Rtimeout =-10

[0100] S26: According to the current state S t , perform action A t , the training data (S t , A t , R t , S t+1 ) is added to the experience pool.

[0101] S27: Fitting the return distribution: Fitting the distribution of returns by weighting N mixed Dirac functions:

[0102]

[0103] Where N is the number of quantiles, τ i is the quantile point generated by the quantile proposal network, satisfying τ i-1 <τ i , and τ0=0, τ N =1 and is the parameter θ in the current state (s, a) i The Dirac function.

[0104] S28: Update the quantile proposal network: Update the quantile proposal network by minimizing the 1-Wasserstein distance to determine the optimal quantile τ so that the fitted distribution is closer to the true distribution. The specific operations are as follows:

[0105] S281: 1-Wasserstein distance formula is:

[0106]

[0107] Where N is the number of quantiles, ω is the neural network parameter, is the quantile The corresponding quantile function value is,

[0108] S282: Due to the true quantile function In fact, it is impossible to obtain, so the quantile value function with the quantile network parameter ω2 is used As a function of the actual quantile value in the current state.

[0109] S283: In order to avoid directly calculating the 1-Wasserstein distance, the quantile proposal network parameter ω1 is used to minimize the 1-Wasserstein distance using gradient descent:

[0110]

[0111] in, is the quantile τ i The corresponding quantile function value is, ω2 is the quantile value network parameter.

[0112] S284: The expected return of the fully parameterized quantile function is:

[0113]

[0114] Where N is the number of quantiles, is the quantile τ i The corresponding quantile function value is, ω2 is the quantile value network parameter.

[0115] S29: Update the quantile numerical network: The update goal of the quantile numerical network is to minimize the quantile regression Huber-loss so that the output of the quantile numerical network is as close to the target value as possible. After obtaining the loss function, the quantile numerical network is updated using the gradient descent method. The specific operations are as follows:

[0116] S291: Solving time difference equations:

[0117]

[0118] Among them, r t is the return at the current moment, γ is the attenuation factor, ω1 is the neural network parameter, is the quantile τ i The corresponding quantile function value is, Z is the reward distribution at the current moment, and Z′ is the reward distribution at the next moment.

[0119] S292: Calculate the quantile regression Huber-loss:

[0120]

[0121] Among them, Huber-loss: δ ij is TD-error, and κ is the threshold.

[0122] S293: Update the quantile value network using stochastic gradient descent:

[0123]

[0124] Where N is the number of quantiles, is the quantile regression Huber-loss, is the TD-error at time t, κ is the threshold, τ iis the quantile point,

[0125] S3: Based on the reward distribution obtained in step S2, a conditional value at risk (CVaR) is introduced to generate a driving behavior with risk awareness, which specifically includes the following steps:

[0126] S31: Based on the return distribution information obtained in step S2, calculate the conditional value at risk (CVaR) corresponding to each distribution:

[0127]

[0128] Among them, the value at risk (VaR): Z is the distribution of returns, α is the cumulative probability, and R is the return, which is a random variable.

[0129] S32: Select the best action: To maximize the CVaR value, select the best risk-sensitive behavior:

[0130]

[0131] in, is the current state t The optimal action selected under , Z is the distribution of rewards, and α is the cumulative probability.

[0132] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A decision planning method for an autonomous driving vehicle based on value distribution reinforcement learning, characterized in that: The method specifically comprises the following steps: S1: Construct a non-signal intersection scenario considering uncertainty, including: establishing an occlusion model, determining the surrounding vehicle model, and establishing the surrounding vehicle type distribution; Establishing an occlusion model specifically includes: considering the occlusion on both sides of the intersection, analyzing the relative position relationship between the surrounding vehicles and the vehicle and the center of the intersection, and calculating the critical distance d at which the surrounding vehicles can be observed by the vehicle based on the geometric relationship, which is used as the critical condition for judging whether the surrounding vehicles are blocked: Where l is the width of each lane, d′ is the distance from the front of the vehicle to the center of the intersection, is the distance from the road boundary to the obstruction, and d is the distance from the front of the surrounding vehicles to the center of the intersection; Determine the surrounding vehicle model, including: The behavior of surrounding vehicles is controlled by the intelligent driver model: Where a is the acceleration, a max is the maximum acceleration, v is the longitudinal velocity of the vehicle, v target is the expected longitudinal velocity of the vehicle, m is the acceleration parameter, d target is the expected longitudinal distance of the vehicle, d0 is the minimum longitudinal distance of the vehicle, T0 is the minimum collision time of the vehicle, and Δv is the relative speed to the preceding vehicle; Establish the distribution of surrounding vehicle types: In order to enable the ego vehicle to make different decisions according to different driver types, it is stipulated that the surrounding vehicles in the simulation environment include three types: aggressive, conservative and normal. Each type of vehicle has a probability of: P aggressive =0.2,P conservative =0.3,P normal = 0.5 is added to the environment, and the surrounding vehicle type space is: S2: Construct a fully parameterized quantile function model as the control model for the autonomous driving vehicle, which specifically includes the following steps: S21: Construct a quantile proposal network: use state information as network input and output the optimal quantile τ corresponding to each state-action; S22: Constructing a quantile numerical network: Taking the optimal quantile obtained by the quantile proposal network as the input of the quantile numerical network, mapping the quantile function values ​​corresponding to each quantile in the current state; S23: Construct state space S: The position, speed, heading angle of surrounding vehicles and the position, speed and heading angle of the self-vehicle are used as the observable state information of the self-vehicle, and the value distribution reinforcement learning is used to make the next decision planning based on the self-vehicle observation information; S24: Constructing action space A: The action space is defined as the set of executable actions of the ego vehicle, which is the output value of the value distribution reinforcement learning network. Here, the action space of the ego vehicle includes three discrete action values: acceleration, cruising, and deceleration. The specific accelerations of the two actions of acceleration and deceleration are calculated by the intelligent driver model. S25: Design reward function, the total reward is equal to the collision reward R collision , Reward for completing the task R success and timeout reward R timeout The sum of three parts; S26: According to the current state S t , perform action A t , the training data (S t ,A t ,R t ,S t+1 ) is added to the experience pool; S27: Fitting the return distribution: Fitting the distribution of returns through the weighted values ​​of N mixed Dirac functions: Where N is the number of quantiles, τ i is the quantile point generated by the quantile proposal network, satisfying τ i-1 <τ i , and τ0=0, τ N =1, is the parameter θ in the current state (s, a) i Dirac function; S28: Update the quantile proposal network: Update the quantile proposal network by minimizing the 1-Wasserstein distance to determine the optimal quantile τ, which specifically includes the following steps: S281: 1-Wasserstein distance formula is: Where N is the number of quantiles, ω is the neural network parameter, is the quantile The corresponding quantile function value is, S282: Using the Quantile Value Function with Quantile Network Parameter ω2 As a function of the true quantile value in the current state; S283: By using gradient descent on the parameter ω1 of the quantile proposal network to minimize the 1-Wasserstein distance: S284: The expected return of the fully parameterized quantile function is: S29: Update the quantile value network: The update goal of the quantile value network is to minimize the quantile regression Huber-loss and update the quantile value network using the gradient descent method; S3: Based on the state-action reward distribution information learned in the fully parameterized quantile function model, conditional risk value is introduced to generate risk-aware driving behavior.

2. The autonomous driving vehicle decision-making planning method according to claim 1, characterized in that: Step S29 specifically includes the following steps: S291: Solving time difference equations: Among them, δ ij is TD-error, r t is the return at the current moment, γ is the attenuation factor, Z is the return distribution at the current moment, and Z′ is the return distribution at the next moment; S292: Calculate the quantile regression Huber-loss: in, is the quantile regression Huber-loss, is the Huber-loss function, κ is the threshold; S293: Update the quantile value network using stochastic gradient descent: in, is the TD-error at time t.

3. The autonomous driving vehicle decision planning method according to claim 1, characterized in that: Step S3 specifically includes the following steps: S31: Based on the return distribution information obtained by the fully parameterized quantile function model in step S2, the conditional value at risk CVaR corresponding to each distribution is calculated as: Among them, the risk value Z is the distribution of returns, α is the cumulative probability, and R is the return; S32: Select the best action to maximize the CVaR value and select the best risk-sensitive behavior: in, is the current state t The optimal action selected under , Z is the distribution of rewards, and α is the cumulative probability.

Images