Decision-making method for autonomous vehicles based on self-learning from driving experience

Abstract

This invention provides a self-learning method for autonomous vehicle decision-making based on driving experience. It has two basic steps: S1, designing an upper-level behavior planning layer; using MDP to make autonomous vehicle decisions, that is, using Markov decision processes to complete vehicle behavior planning; S2, designing a lower-level motion planning control layer. By introducing nonlinear model predictive control technology to impose rigid constraints on the target autonomous vehicle, the upper-level behavior planning layer will autonomously learn vehicle control instructions based on observation information collected from the driving environment. The vehicle control instructions are then passed to the model predictive control, that is, to the MPC, which solves the tracking optimization problem and provides low-level commands to the target autonomous vehicle. This invention achieves the integration of upper-level behavior planning and lower-level motion planning, significantly improving the training efficiency of autonomous vehicles and enhancing the performance and applicability of autonomous vehicles in real-world driving environments.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent transportation, and specifically relates to a decision-making method for an autonomous driving vehicle based on self-learning from driving experience. Background Art

[0002] The invention patent application with application publication number CN118396034A discloses a decision-making and planning method for autonomous driving vehicles in urban traffic scenarios based on reinforcement learning. The method includes the following steps:

[0003] Step 1: Multiple sensors are installed on the autonomous vehicle. The vehicle interacts with the urban traffic environment to obtain observation information, status information, and action information of itself and neighboring vehicles in the environment.

[0004] Step 2: Input the environmental information obtained by the visual sensor and lidar into the reinforcement learning algorithm, add a method based on improved priority replay experience, set a reward function based on urban traffic scenarios, and add an attention mechanism to the network;

[0005] Step 3: The reinforcement learning algorithm outputs the corresponding autonomous driving decision-making planning strategy, outputs the reward value of the strategy based on the reward function in the urban traffic scenario, and stores the corresponding state information, action information and reward value in the experience pool;

[0006] Step 4: Process and extract the data in the experience pool using a method based on improved priority replay experience, calculate the loss value, and update the parameters of the main network and the target network;

[0007] In step 2, the reinforcement learning algorithm includes DQN (Deep Q-Leaning Network, DQN) based on the attention mechanism. The specific steps are:

[0008] S1, multiple sensors are installed on the autonomous vehicle. In the initial state, the visual sensor and lidar obtain environmental information in the urban traffic scene, including observation information, action information, and information about neighboring vehicles;

[0009] S2 inputs all environmental information into the attention mechanism. By leveraging the attention mechanism's ability to give more attention to important information, it assigns higher weight to vehicle-related status information in the sensor data, thereby improving the ability to read effective information. The relevant processing process is as follows:

[0010] ① Compression: compress the feature map of each channel through global average pooling. Assume that the input visual sensor and lidar data feature map is X∈RC×H×W, where C is the number of channels, H is the height, and W is the width. The output of this stage is a channel vector z, which is calculated as follows:

[0011]

[0012] Where zc represents the vector of channel c, and Xcij represents the data feature map with height i and width j in channel c;

[0013] ② Incentive: Based on each channel vector z output in the compression phase, a fully connected layer is used to learn the channel weight to obtain the attention weight of each channel. Usually, a single fully connected layer is used to learn the channel weight. Given the weight and bias of the fully connected layer, the output channel attention weight s is calculated as follows:

[0014] s C =σ(W exc ·z C +b eXC )

[0015] Among them, sc represents the attention weight of channel c, Wexc represents the weight of the fully connected layer, and bexc represents the bias of the fully connected layer. The compressed feature map is mapped to a new dimension through the fully connected layer, and then the σ activation function is used to limit the value of this new dimension to an appropriate range;

[0016] ③ Scaling, applying the learned attention weights to the input feature map to achieve adaptive weighting of channels. For the channel attention weight vector obtained in the excitation phase, where C is the number of channels, the output feature map of the scaling phase is calculated as follows:

[0017] X′ cij =s c ·X cij

[0018] S3, according to the environmental information processed by the attention mechanism, follows the ε-greedy rule and selects the appropriate planning strategy

[0019] S4, execute the strategy at, output the reward r according to the reward function based on the urban traffic scenario, and reach the next environment state st+1;

[0020] S5, store (st, at, rt, st+1) in the above process into the experience pool D, and process the data in the experience pool using the method based on improved priority replay experience;

[0021] S6, repeat the above steps until the experience pool is full;

[0022] S7, when the experience pool is full, update the main network parameters, that is, start sampling from the experience pool using the minibatch strategy, extract (st, at, rt, st+1) as training data, send the sampled rt and st+1 into the target network, calculate yt=rt+γmaxa Q(st+1,a;θ), then calculate the loss value loss=[yt-Q(s,a;θ)]2, and perform gradient descent on the main network parameters θ to make Q(st,at) as close to yt as possible;

[0023] S8, after the main network parameters are updated for Z steps, all the updated parameters of the main network are directly copied to the target network.

[0024] In summary, the technical solution disclosed in invention patent application No. CN118396034A utilizes a DQN algorithm based on a channel attention mechanism for decision-making and planning, which can effectively reduce and avoid collisions during autonomous driving. In addition, by setting a reward function tailored to urban traffic scenarios, the invention can achieve safety, comfort, timeliness, and goal completion in autonomous driving, and improve the intelligence and autonomy of vehicle decision-making. However, the invention also has some technical drawbacks, which are specifically manifested in the following two aspects:

[0025] First, the invention uses a complex attention mechanism and deep neural network structure, which makes the model training process complex and computationally intensive, and places excessive demands on hardware resources. This flaw can affect the real-time and safety of autonomous driving, particularly in complex and volatile urban traffic environments, and can even prevent convergence to the optimal strategy in some scenarios.

[0026] Second, although the invention designs a reward function for urban traffic scenarios, it does not fully consider all relevant factors (such as passenger comfort, compliance with traffic rules, etc.), resulting in incomplete optimization objectives. An incomplete reward function will cause vehicle behavior to be inconsistent with human driving habits or traffic rules, reducing passenger satisfaction and driving safety. Summary of the Invention

[0027] The purpose of the present invention is to overcome the first and second technical deficiencies of the above-mentioned prior art.

[0028] In order to achieve the above object, the present invention adopts the following technical solutions:

[0029] A decision-making method for autonomous vehicles based on self-learning from driving experience includes the following two basic steps:

[0030] S1, upper behavior planning layer design; use MDP to complete vehicle autonomous driving decision-making, that is, use Markov decision process to complete vehicle behavior planning. MDP is represented as a tuple {S, A, R, T, γ}, where S is the observed state of the host vehicle and the target autonomous driving vehicle, including the speed, position and road topology of the host vehicle and the target autonomous driving vehicle, A is the action space, T is the transfer function, R is the reward function, and γ is the depreciation factor of the long-term reward;

[0031] S2, the design of the lower-level motion planning control layer; by introducing nonlinear model predictive control technology to impose rigid constraints on the target autonomous vehicle, the upper-level behavior planning layer will autonomously learn vehicle control instructions based on observation information collected from the driving environment, and then pass the vehicle control instructions to the model predictive control, that is, to the MPC, which solves the tracking optimization problem and provides low-level commands to the target autonomous vehicle.

[0032] On the basis of the above technical solutions, the present invention may add the following technical means to better or more specifically solve the technical problems to be solved by the present invention:

[0033] The S1 includes the following specific steps:

[0034] S101, reward function design;

[0035] S102, determine the observation space s′ and the action space a′;

[0036] S103, driving behavior decision based on Double DQN.

[0037] Furthermore, when executing step S101, the reward function is designed by comprehensively considering the safety index, efficiency index, and comfort index. The expression of the reward function is as follows:

[0038] r ego =r eff +α1r dtg +α2r lat +α3r exp +r terminal +r comf

[0039] Where r dtg and r lat They represent the penalty terms for the distance from the target and the lateral deviation, r exp represents the reward obtained from the continuously updated driving experience pool, and α1, α2, and α3 are balance coefficients;

[0040] r terminalCorresponding to the safety indicator, it means that if the vehicle successfully completes the target route, it will be rewarded, and if the vehicle crashes or exceeds the maximum mileage, it will be penalized;

[0041] r eff The corresponding efficiency index represents the efficiency reward, and the calculation formula is as follows:

[0042] r eff =r eff1 +r eff2

[0043]

[0044] Where r eff1 represents the reward obtained by the instantaneous speed, v lim and v ego Represent the road speed limit and current vehicle speed respectively, v eff To design the minimum speed to maintain efficient driving, c1 and c3 are penalty hyper parameters. ego >v lim When c1≤0, when v ego <v eff When c3≤0; c2 is the reward hyperparameter, 0≤c2≤1, r eff2 Indicates the reward given based on the average speed of the vehicle when completing the target. represents the average speed of the vehicle when it completes the goal, and c4 is the reward hyperparameter;

[0045] r comf Corresponding to the comfort index, it means that the vehicle acceleration and jerk (acceleration change rate) need to be limited. When the acceleration or jerk exceeds the threshold, a negative reward is given, r comf The calculation formula is as follows:

[0046]

[0047] Where, v a represents the vehicle acceleration, v j represents the vehicle acceleration, v acomf and v jcomf Represent the acceleration and jerk thresholds respectively. c5, c6, and c7 are all negative numbers, indicating negative reward parameters.

[0048] Furthermore, when executing step S102, the observation space s′ includes the host vehicle's own motion state, the target driving vehicle's speed information, and the road geometry topology. The host vehicle's own motion state includes the host vehicle's own speed v e , acceleration Yaw angle dθ e , yaw angular velocity The lateral deviation d between the main vehicle and the lane centercl , the action space a′ includes the reference velocity v ref and the reference heading θ ref The so-called ego vehicle refers to the vehicle currently making decisions and planning, that is, the vehicle controlled by the intelligent driving system.

[0049] Furthermore, when executing step S102, the observation space s′ also includes the relative lateral and longitudinal distances p between the target autonomous driving vehicle and the host vehicle. tar , and the linear velocity v of the target autonomous vehicle tar .

[0050] Furthermore, when executing step S103, the optimization objective of the autonomous driving decision task based on Double DQN is:

[0051]

[0052] Furthermore, when executing step S2, MPC uses the vehicle kinematic model for prediction, following the vehicle nonholonomic constraints.

[0053] Furthermore, when executing step S2, the relationship of the vehicle kinematic model is as follows:

[0054]

[0055] u=[v δ f ] Τ ;

[0056] Where x = [xy θ] Τ is the vehicle state, x and y are the vehicle positions in the XY global frame, θ is the vehicle localization, and u = [v δ f ] Τ is the vector of the control action, v is the speed of the main vehicle, δ f is the steering angle.

[0057] Furthermore, when executing step S2, hard constraints are designed for the optimization variables as follows:

[0058]

[0059] Furthermore, when executing step S2, the objective function J of MPC is defined as:

[0060]

[0061] Where z = [θv] Τ , Q∈R 2×2 , R∈R 2×2 , t0 is the initial time, T His the prediction time; the optimal control problem is formulated as follows:

[0062]

[0063] Compared with the prior art, the present invention has the following beneficial effects:

[0064] First, the present invention introduces a self-learning mechanism based on driving experience and adopts MDP (Markov decision process) combined with Double DQN algorithm for decision planning. Compared with the complex attention mechanism network, it realizes the integration of upper-level behavior planning and lower-level motion planning, greatly improving the training efficiency of vehicle autonomous driving.

[0065] Second, the present invention designs a more comprehensive reward function, which not only considers the penalty terms of distance from the target and lateral deviation, but also introduces multiple factors such as speed limit and driving task completion as balance coefficients.

[0066] Third, the present invention more fully considers the constraints in actual driving scenarios, including vehicle dynamics and ride comfort, thereby improving the performance and applicability of vehicle autonomous driving in real-world driving environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 This is a flow chart of the main working steps of an embodiment of the present invention;

[0068] Figure 2 is a schematic diagram of the decision-making scheme of the autonomous driving vehicle in this embodiment;

[0069] Figure 3 Schematic diagram of the vehicle dynamics model in this embodiment. DETAILED DESCRIPTION

[0070] In order to facilitate those skilled in the art to fully understand the technical solution of the present invention, an embodiment of the present invention is described below with reference to the accompanying drawings.

[0071] like Figure 1 Combined with Figure 2 As shown in FIG, a decision-making method for an autonomous vehicle based on self-learning from driving experience includes the following two basic steps:

[0072] S1, upper behavior planning layer design; use MDP to complete vehicle autonomous driving decision-making, that is, use Markov decision process to complete vehicle behavior planning. MDP is represented as a tuple {S, A, R, T, γ}, where S is the observed state of the host vehicle and the target autonomous driving vehicle, including the speed, position and road topology of the host vehicle and the target autonomous driving vehicle, A is the action space, T is the transfer function, R is the reward function, and γ is the depreciation factor of the long-term reward;

[0073] S2, the design of the lower-level motion planning control layer; by introducing nonlinear model predictive control technology to impose rigid constraints on the target autonomous vehicle, the upper-level behavior planning layer will autonomously learn vehicle control instructions based on observation information collected from the driving environment, and then pass the vehicle control instructions to the model predictive control, that is, to the MPC, which solves the tracking optimization problem and provides low-level commands to the target autonomous vehicle.

[0074] In this embodiment, S1 includes the following specific steps:

[0075] S101, reward function design;

[0076] S102, determine the observation space s′ and the action space a′;

[0077] S103, driving behavior decision based on Double DQN.

[0078] When executing step S101, this embodiment constructs a reward function that balances driving efficiency and safety. Specifically, for the upper-level behavior planning, the vehicle's desired driving behavior is to complete the driving task as efficiently as possible while ensuring safety, that is, staying within the lane and avoiding collisions. The reward function is designed according to the following rules:

[0079] (1) A comprehensive reward function design will motivate the vehicle to complete the task efficiently;

[0080] (2) Considering safety, speed thresholds are set for different road speed limits. Positive rewards are given to driving behaviors that keep the speed below the threshold on different roads, while negative rewards are given to those that exceed the threshold.

[0081] (3) Consider the lateral deviation of the lane center based on road topology constraints.

[0082] According to the above rules, this embodiment needs to comprehensively consider safety indicators, efficiency indicators, and comfort indicators to design the reward function. The expression of the reward function is as follows:

[0083] r ego =r eff +α1r dtg +α2r lat +α3r exp +r terminal

[0084] Where r dtg and r lat They represent the penalty terms for the distance from the target and the lateral deviation, r exp represents the reward obtained from the continuously updated driving experience pool, and α1, α2, and α3 are balance coefficients.

[0085] r terminal Corresponding safety index: r terminal It means that if the vehicle successfully completes the target route, it will be rewarded. If the vehicle collides or exceeds the maximum mileage, it will be punished, indicating that this decision result should be avoided. If it passes safely, a larger positive reward will be given.

[0086] r eff Corresponding efficiency index: In order to guide the vehicle to complete the goal efficiently, the vehicle speed should be rewarded. The vehicle should travel as fast as possible but its speed cannot exceed the limit v lim , r eff1 The calculation formula is as follows:

[0087]

[0088] Where, v lim and v ego Represent the road speed limit and current vehicle speed respectively, v eff To design the minimum speed to maintain efficient driving, c1 and c3 are penalty hyper parameters. ego >v lim When c1≤0, when v ego <v eff When , c3≤0; c2 is the reward hyperparameter, 0≤c2≤1.

[0089] On the other hand, the above r eff1 It can only represent the reward obtained by the instantaneous speed. In order to prevent the situation where the instantaneous speed is too high but the average speed is too low at some moments, r is designed. eff2 , rewards are given based on the average speed of the vehicle when it completes the target. The greater the average speed, the greater the reward.

[0090]

[0091] in represents the average speed of the vehicle when it completes the goal, and c4 is the reward hyperparameter.

[0092] Combining the above two efficiency indicators, the efficiency reward is r eff , the formula is as follows:

[0093] r eff =r eff1 +r eff2

[0094] r comf corresponds to the comfort index: excessive acceleration or deceleration or excessive change in acceleration (i.e., excessive jerk) will make the driver or passenger feel strongly uncomfortable, so it is necessary to limit the magnitude of acceleration and jerk. When the acceleration or jerk exceeds the threshold, a negative reward is given, r comf The calculation formula is as follows:

[0095]

[0096] Among them, v a represents the vehicle acceleration, v j represents the vehicle acceleration, v acomf and v jcomf Represent the acceleration and jerk threshold respectively. c5, c6, c7 are all negative numbers, indicating negative reward parameters.

[0097] With respect to the speed thresholds described above, different speed thresholds can be designed under different road conditions, as shown in the following table.

[0098] Road speed limit (in km / h) <![CDATA[Speed threshold (v eff , v lim (unit: km / h)]]> 120 (100,120] 100 (80,100] 80 (60,80] 60 (40,60] 40 (25,40] 30 (20,30] 20 (0,20]

[0099] For the acceleration and jerk thresholds described above, different thresholds can also be designed under different vehicle operating conditions, as shown in the following table.

[0100]

[0101]

[0102] When executing step S102, the observation space s′ includes the motion state of the subject vehicle itself (the so-called subject vehicle, corresponding to the English word Ego Vehicle, refers to the vehicle currently making decisions and planning, that is, the vehicle controlled by the intelligent driving system), the speed information of the target driving vehicle, and the road geometry topology. The motion state of the subject vehicle itself includes the subject vehicle's own speed v e , acceleration Yaw angle dθ e (deviation from lane center), yaw rate The lateral deviation d between the main vehicle and the lane center cl , the relative lateral and longitudinal distances p between the target autonomous driving vehicle and the host vehicle tar , and the linear velocity v of the target autonomous vehicle tar , the action space a′ includes the reference velocity v ref and the reference heading θ ref It should also be noted that the action taken previously t-1. The present invention can also record the driving experience space in the next state in the form of driving experience space playback, that is, constructing the driving experience space in a self-learning form. By continuously updating and querying the historical decision state, the concept of self-evolution is embedded in the autonomous driving decision-making process in a form similar to looking up a dictionary.

[0103] When executing step S103, the Double DQN algorithm estimates the The existing Change to That is, using a neural network Q ω The output of the action with the greatest value is selected, but when using the value of the action, another neural network is used Calculate the value of the action. It should be noted that the reason why this embodiment uses the existing Change to This is because the ordinary DQN algorithm usually leads to an overestimation of the Q value. The TD error target of the traditional DQN optimization is:

[0104]

[0105] in, The parameter is ω - The target network is calculated, and this formula can also be written as follows:

[0106]

[0107] The max operation can actually be broken down into two parts: first, the optimal action under state s′ is selected. Further calculate the value corresponding to the action When these two parts use the same Q network for calculation, each time the result is the maximum value of all action values ​​currently estimated by the neural network.

[0108] In this embodiment, the training network is directly used as the first neural network in the Double DQN algorithm to select actions, and the target network is used as the second neural network to calculate Q. Since the parameters of the training network are recorded as ω in the DQN algorithm, the parameters of the target network are recorded as ω. - Therefore, in this embodiment, the optimization goal of the autonomous driving decision-making task based on Double DQN is:

[0109]

[0110] See also Figure 3 In this embodiment, when executing step S2, MPC uses the vehicle kinematic model for prediction, following the vehicle nonholonomic constraints. The vehicle kinematic model is expressed as follows:

[0111]

[0112] u=[v δ f ] Τ ;

[0113] Where x = [x yθ] Τ is the vehicle state, x and y are the vehicle positions in the XY global frame, θ is the vehicle localization, and u = [v δ f ] Τ is the vector of the control action, v is the speed of the main vehicle, δ f is the steering angle.

[0114] It should also be noted that, in order to ensure the comfort and safety of the vehicle, this embodiment may design hard constraints on the optimization variables when executing step S2 as follows:

[0115]

[0116] In order to achieve optimal control of the vehicle, when executing step S2, the objective function J of the MPC can be defined as:

[0117]

[0118] Where z = [θv] Τ , Q∈R 2×2 , R∈R 2×2 , t0 is the initial time, T H is the prediction time; the optimal control problem is formulated as follows:

[0119]

Claims

1. A decision-making method for an autonomous vehicle based on self-learning from driving experience, comprising the following two basic steps: S1, upper behavior planning layer design; use MDP to complete vehicle autonomous driving decision-making, that is, use Markov decision process to complete vehicle behavior planning. MDP is represented as a tuple {S, A, R, T, γ}, where S is the observed state of the host vehicle and the target autonomous driving vehicle, including the speed, position and road topology of the host vehicle and the target autonomous driving vehicle, A is the action space, T is the transfer function, R is the reward function, and γ is the depreciation factor of the long-term reward; S2, design of lower motion planning and control layer; By introducing nonlinear model predictive control technology to impose rigid constraints on the target autonomous driving vehicle, the upper behavior planning layer will autonomously learn vehicle control instructions based on observation information collected from the driving environment, and then pass the vehicle control instructions to the model predictive control, that is, pass the vehicle control instructions to the MPC, which solves the tracking optimization problem and provides low-level commands to the target autonomous driving vehicle; characterized in that, S1 includes the following specific steps: S101, reward function design; S102, determine the observation space s′ and the action space a′; S103, driving behavior decision-making based on Double DQN; When executing step S102, the observation space s′ includes the host vehicle's own motion state, the target driving vehicle's speed information and the road geometry topology. The host vehicle's own motion state includes the host vehicle's own speed v e , acceleration Yaw angle dθ e , yaw angular velocity The lateral deviation d between the main vehicle and the lane center cl , the action space a′ includes the reference velocity v ref and the reference heading θ ref .

2. The autonomous driving vehicle decision-making method based on driving experience self-learning according to claim 1, characterized in that: When executing step S101, the reward function is designed by comprehensively considering the safety index, efficiency index, and comfort index. The expression of the reward function is as follows: r ego ＝r eff +α1r dtg +α2r lat +α3r exp +r terminal +r comf ； Where r dtg and r lat They represent the penalty terms for the distance from the target and the lateral deviation, r exp represents the reward obtained from the continuously updated driving experience pool, and α1, α2, and α3 are balance coefficients; r terminal Corresponding to the safety indicator, it means that if the vehicle successfully completes the target route, it will be rewarded, and if the vehicle crashes or exceeds the maximum mileage, it will be penalized; r eff The corresponding efficiency index represents the efficiency reward, and the calculation formula is as follows: r eff ＝r eff1 +r eff2 Where r eff1 represents the reward obtained by the instantaneous speed, v lim and v ego Represent the road speed limit and current vehicle speed respectively, v eff To design the minimum speed to maintain efficient driving, c1 and c3 are penalty hyper parameters. ego >v lim When c1≤0, when v ego <v eff When c3≤0; c2 is the reward hyperparameter, 0≤c2≤1, r eff2 Indicates the reward given based on the average speed of the vehicle when completing the target. represents the average speed of the vehicle when it completes the goal, and c4 is the reward hyperparameter; r comf Corresponding to the comfort index, it means that the vehicle acceleration and jerk need to be limited. When the acceleration or jerk exceeds the threshold, a negative reward is given. comf The calculation formula is as follows: Where, v a represents the vehicle acceleration, v j represents the vehicle acceleration, v acomf and v jcomf Represent the acceleration and jerk thresholds respectively. c5, c6, and c7 are all negative numbers, indicating negative reward parameters.

3. The autonomous driving vehicle decision-making method based on self-learning from driving experience according to claim 1, wherein: When executing step S102, the observation space s′ also includes the relative lateral and longitudinal distances p between the target autonomous driving vehicle and the host vehicle. tar , and the linear velocity v of the target autonomous vehicle tar .

4. The autonomous driving vehicle decision-making method based on self-learning from driving experience according to claim 1, wherein: When executing step S103, the optimization goal of the autonomous driving decision task based on Double DQN is:

5. The autonomous driving vehicle decision-making method based on self-learning from driving experience according to claim 1, wherein: When executing step S2, MPC uses the vehicle kinematic model to make predictions, following the vehicle nonholonomic constraints.

Images