A lane-changing trajectory planning method and system for an autonomous driving vehicle
Through deep reinforcement learning and game decision-making models, optimize the lane change trajectory planning of autonomous driving vehicles, solve the problems of complexity and data dependence in the existing technology, realize a safe, efficient and economical lane change trajectory, and improve the training speed and control accuracy.
Patent Information
- Application Number
- CN202210706690.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-06-21
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2042-06-21
AI Technical Summary
The existing self-driving vehicle lane change trajectory planning methods are complex and have strong data dependence, making it difficult to take into account safety, efficiency and fuel economy.
Adopt the lane change trajectory planning method of autonomous driving vehicles based on deep reinforcement learning, combining the game lane change decision model and horizontal and vertical discrete kinematic model, and optimize lane change by obtaining vehicle information and surrounding vehicle information, using the game revenue matrix and reward function. Decisions are made to reduce fuel consumption and improve driving efficiency, and achieve optimal lane change trajectory planning.
It realizes that while ensuring safety and comfort, the fuel consumption during lane change is reduced and driving efficiency is improved. The delay update strategy of the TD3 algorithm improves training speed and control accuracy, and reduces dependence on real data.
Smart Images

Figure CN114852105B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of vehicle control decision-making, and relates to a lane-changing trajectory planning method and system for an autonomous driving vehicle. Background Art
[0002] In recent years, autonomous vehicles have great potential in improving traffic safety, improving energy efficiency and alleviating traffic congestion, and have attracted much attention from academia and industry. Lane changing is a basic task in the vehicle driving process and plays a very important role in the safe driving of the vehicle. The current lane changing trajectory planning methods have problems such as high complexity and strong data dependence. With the improvement of the level of vehicle intelligence, vehicle lane changing trajectory planning that takes into account both safety and traffic efficiency has gradually become one of the hot spots in the research of autonomous vehicles. Summary of the invention
[0003] In order to solve the problems existing in the prior art, the purpose of the present invention is to provide a lane change trajectory planning method and system for an autonomous driving vehicle, which completes the lane change trajectory planning for an autonomous driving vehicle while taking into account safety, efficiency, comfort and fuel economy.
[0004] In order to achieve the above object, the present invention provides the following technical solutions:
[0005] A lane-changing trajectory planning method for an autonomous driving vehicle includes the following steps:
[0006] Obtaining vehicle information: The vehicle information includes: speed, acceleration and position information of the lane-changing vehicle, and speed, acceleration and position information of surrounding vehicles;
[0007] According to the vehicle information, the game lane-changing decision model between the lane-changing vehicle and surrounding vehicles, and the game payoff function considering safety and timeliness, and using the game payoff matrix, the optimal lane-changing decision of the lane-changing vehicle at the current moment is solved;
[0008] According to the optimal lane-changing decision, with the goal of reducing fuel consumption and improving driving efficiency, and using a lane-changing trajectory planning model for an autonomous driving vehicle based on deep reinforcement learning, the optimal sequential acceleration decision information for the entire lane-changing process is obtained;
[0009] The vehicle state of the lane-changing vehicle at each time point in the lane-changing process is calculated by using the vehicle's lateral and longitudinal discretized kinematic model and the optimal sequential acceleration decision information, and the lane-changing trajectory of the lane-changing vehicle is obtained according to the vehicle state of the lane-changing vehicle.
[0010] Preferably, the process of establishing the vehicle lateral and longitudinal discretized kinematic model includes:
[0011] In the XOY plane rectangular coordinate system, the X-axis direction is the longitudinal direction of the vehicle, and the Y-axis direction is the lateral direction of the vehicle. The vehicle kinematic model is decoupled and discretized with Δt as the sampling time. The longitudinal and lateral separated discretized vehicle kinematic model and constraint conditions are as follows:
[0012] v xt =v x(t-1) +a xt Δt
[0013] v yt =v y(t-1) +a yt Δt
[0014]
[0015] 0 <v xt <v x,max ,0 <v yt <v y,max
[0016]
[0017] Among them, v xt and v yt They represent the longitudinal velocity and lateral velocity of the vehicle at time t respectively; v x(t-1) and v y(t-1) They represent the longitudinal speed and lateral speed of the vehicle at time t-1 respectively; time t-1 is the moment before time t; x t and t Respectively represent the ordinate and abscissa of the vehicle at time t; x (t-1) and (t-1) They represent the ordinate and abscissa of the vehicle at time t-1 respectively; the longitudinal acceleration a xt and lateral acceleration a yt Obtained by the interaction between the vehicle and the algorithm at each time step Δt, v x,max and v y,max are the maximum longitudinal velocity and the maximum lateral velocity, respectively, x max and max are the maximum values of the longitudinal position and the horizontal position, respectively, t f is the completion time of the lane change;
[0018] The position and speed of the vehicle at the next moment are obtained through the kinematic model until the vehicle reaches the lane change target position or leaves the lane and the termination state ends.
[0019] Preferably, the game lane-changing decision model between the lane-changing vehicle and surrounding vehicles is as follows:
[0020] The participants of the game are the lane-changing vehicle M and the target lane following vehicle F.d and the vehicle in front of the target lane L d , the strategy set of lane-changing vehicle M is Φ1 = {m1, m2}, where m1 represents lane change and m2 represents no lane change; the target lane following vehicle F d and the vehicle in front of the target lane L d The strategy set is Φ2={d i1 , d i2}, i={F d ,L d}, where d i1 Indicates that vehicle i is allowed to change lanes, d i2 Indicates that vehicle i refuses to change lanes.
[0021] Preferably, the game profit function considering security and timeliness is as follows:
[0022] R M ,R D =α1*R safe +α2*R time .
[0023] Among them, R safe and R time They represent the benefits obtained by the decision-making vehicle considering safety and timeliness respectively; α1 and α2 are weight coefficients, α1+α2=1, representing the importance of different driving factors;
[0024]
[0025] Among them, v Mx and v Ldx They represent the longitudinal speed of the lane-changing vehicle and the longitudinal speed of the vehicle in front of the target lane, respectively. head is the distance between the two vehicles at the current moment, a Mx and is the longitudinal acceleration of the lane-changing vehicle and the vehicle in front of the target lane; P min is the minimum safety distance required in the current state; t0 represents the time required to reach the destination in the original state, x tar get is the longitudinal coordinate of the lane change target point; t f Represents the completion time of the lane change, which is obtained from the trajectory planning part.
[0026] Preferably, the game profit matrix is as shown in Table 1:
[0027] Table 1
[0028]
[0029] Among them, R M represents the game benefit that the lane-changing vehicle can obtain under the current strategy, R DIt represents the game benefits that the vehicle in the target lane can obtain under the current strategy;
[0030] The lane-changing vehicle plays games with the vehicle in front and the following vehicle in the target lane respectively, and four game results are obtained: the lane-changing vehicle changes lanes, and the vehicle in the target lane allows the lane change; the lane-changing vehicle changes lanes, and the vehicle in the target lane refuses to change lanes; the lane-changing vehicle does not change lanes, and the vehicle in the target lane allows the lane change; the lane-changing vehicle does not change lanes, and the vehicle in the target lane refuses to change lanes; the lane change is executed only when the lane-changing vehicle chooses a lane-changing strategy and the vehicle in front and the following vehicle in the target lane both make strategies to allow the lane change.
[0031] Preferably, the process of solving the optimal lane-changing decision of the lane-changing vehicle at the current moment includes:
[0032] If there is a pure strategy [d in ,m n ],i={F d ,L d}, n=1,2 so that the following equation holds true, then [d in ,m n ] is the pure strategy Nash equilibrium of the current game. The strategy is the optimal lane-changing decision made by the vehicle under the current environment. The lane-changing trajectory planning sub-problem to be solved is determined based on this decision. The lane-changing trajectory planning sub-problem includes: left lane changing, right lane changing or lane keeping;
[0033]
[0034] Where, d in ,i={F d ,L d}, n = 1 means that vehicle i in the target lane is allowed to change lanes, d in ,i={F d ,L d}, n = 2 means that vehicle i in the target lane refuses to change lanes, m n , n=1 means lane-changing vehicle M changes lanes, m n , n = 2 means that the lane-changing vehicle M does not change lanes, Φ1 represents the strategy set of the lane-changing vehicle M, m represents any strategy in the lane-changing vehicle strategy set Φ1, and Φ2 represents the target lane following vehicle F d and the vehicle in front of the target lane L d The strategy set, d i represents any strategy in the strategy set Φ2 of vehicle i in the target lane.
[0035] Preferably, the lane-changing trajectory planning model for an autonomous driving vehicle based on deep reinforcement learning includes a state space, an action space, and a reward function;
[0036] The state space contains all the information required for the automatic driving vehicle to change lanes. The vertical coordinate x of the current lane-changing vehicle is obtained at each moment. t , horizontal axis y t , longitudinal speed v xt and the lateral velocity v yt ; The state at each moment is represented by a four-tuple, s t =[x t ,v xt ;y t ,v yt ];
[0037] The action space defines the actions taken by the autonomous vehicle. Combined with the longitudinal and lateral decoupling state equations of vehicle kinematics, the longitudinal and lateral accelerations are used as the actions of the autonomous vehicle. The longitudinal acceleration a xt The value range is set to [-2m / s 2 ,2m / s 2 ]; lateral acceleration a yt The value range is [-0.2m / s 2 ,0.2m / s 2 ]; the action at each moment is represented by a two-tuple, a t =[a xt ; a yt ].
[0038] The reward function R t The reward R along the center line of the target lane y , Target Speed Reward R v , fuel consumption reward R e and end-of-mission reward R d It consists of four parts, as follows:
[0039] R t =w y R y +w v R v +w e R e +w d R d .
[0040] R y =-|y t -y target |
[0041] R v =-a xt (v xt -v target )
[0042] R e =ln(MOE)
[0043]
[0044] Where MOE is the instantaneous fuel consumption, including a combination of linear, quadratic and cubic velocity and acceleration terms, L k,q and M k,q represents the model coefficient of MOE at the kth power of velocity and the qth power of acceleration, L k,q and M k,q Using instantaneous fuel consumption as a penalty term, the vehicle learns an energy-saving lane-changing method; w y 、w v 、w e and w d are weight coefficients of different benefits, indicating their importance, C1 and C2 are constants, y target represents the lateral coordinate of the centerline of the target lane, v target It represents the target speed when the vehicle reaches the end of lane change, and e is the base of the natural logarithm.
[0045] Preferably, the training process of the lane change trajectory planning model for the autonomous driving vehicle based on deep reinforcement learning includes:
[0046] The initial state of the lane-changing vehicle is used as the input of the double-delay deep deterministic policy gradient learning algorithm, and the corresponding state of the lane-changing target is used as the final state of the lane-changing vehicle. The double-delay deep deterministic policy gradient learning algorithm is trained. During the training process, the decision-making behaviors and corresponding benefits of adding random noise are first collected and stored in the experience replay pool. After reaching the preset number, batch selection is performed for training until the benefits gradually stabilize and the double-delay deep deterministic policy gradient learning algorithm converges.
[0047] The loss of the evaluation network and the policy network of the double-delayed deep deterministic policy gradient learning algorithm during training is calculated as follows:
[0048]
[0049] Preferably, according to the optimal lane changing decision, with the goal of reducing fuel consumption and improving driving efficiency, and using the lane changing trajectory planning model of the autonomous driving vehicle based on deep reinforcement learning, the optimal sequential acceleration decision information of the entire lane changing process is obtained, and according to the converged double-delay deep deterministic policy gradient learning algorithm, the initial state of the current vehicle and the lane changing target state are input into the double-delay deep deterministic policy gradient learning algorithm to solve the lane changing trajectory planning model of the autonomous driving vehicle based on deep reinforcement learning, and the optimal sequential acceleration decision information of the entire lane changing process is obtained.
[0050] The present invention also provides a lane-changing trajectory planning system for an autonomous driving vehicle, comprising:
[0051] Acquiring environment information module: used to acquire vehicle information, the vehicle information including: speed, acceleration and position information of the lane-changing vehicle, and speed, acceleration and position information of surrounding vehicles;
[0052] Lane-changing decision module: used to solve the optimal lane-changing decision of the lane-changing vehicle at the current moment according to the vehicle information, the game lane-changing decision model between the lane-changing vehicle and the surrounding vehicles, and the game profit function considering safety and timeliness, and using the game profit matrix;
[0053] Lane-changing trajectory planning module: used to obtain the optimal sequential acceleration decision information of the entire lane-changing process based on the optimal lane-changing decision, with the goal of reducing fuel consumption and improving driving efficiency, and using the lane-changing trajectory planning model of the autonomous driving vehicle based on deep reinforcement learning;
[0054] Simulation module: used to calculate the vehicle state of the lane-changing vehicle at each time point in the lane-changing process by using the vehicle's lateral and longitudinal discretized kinematic model through the optimal sequential acceleration decision information, and obtain the lane-changing trajectory of the lane-changing vehicle according to the vehicle state of the lane-changing vehicle.
[0055] Compared with the prior art, the present invention has the following beneficial effects:
[0056] The present invention uses a Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm that can handle continuous control quantities, which can realize the control of vehicle speed and acceleration, solve the problem of loss of precision control caused by discretizing the action space using the DQN algorithm, and the problem of overvaluation of the benefit function that occurs when using the DDPG algorithm, and can more accurately evaluate the vehicle control behavior. At the same time, the delayed update strategy of the TD3 algorithm increases its average training speed by 10.5% compared with DDPG. The network is trained by storing historical experience data obtained through the interaction between the vehicle and the environment. It does not need to be trained through a large amount of real lane change data like machine learning, which saves pre-processing operations such as cleaning and screening of the data and improves flexibility. Instantaneous fuel consumption and target speed control are considered in the reward function, which reduces the fuel consumption during the lane change process, and finally obtains a safe, comfortable and green lane change trajectory. BRIEF DESCRIPTION OF THE DRAWINGS
[0057] Figure 1 The overall framework diagram of the lane change trajectory planning method for an autonomous driving vehicle of the present invention;
[0058] Figure 2 It is a schematic diagram of the TD3 network structure of the present invention;
[0059] FIG3(a) is a schematic diagram of a left lane change trajectory in Example 1 of the present invention, FIG3(b) is a comparison diagram of fuel consumption per round before and after training in Example 1 of the present invention, and FIG3(c) is a speed control curve in Example 1 of the present invention;
[0060] FIG4(a) is a schematic diagram of a right lane change trajectory in Example 2 of the present invention, FIG4(b) is a comparison diagram of fuel consumption per round before and after training in Example 2 of the present invention, and FIG4(c) is a speed control curve in Example 2 of the present invention;
[0061] FIG5(a) is a comparative experimental diagram of the left lane change training time of the TD3 algorithm used in the present invention and other (DDPG) algorithms, and FIG5(b) is a comparative experimental diagram of the right lane change training time of the TD3 algorithm used in the present invention and other (DDPG) algorithms. DETAILED DESCRIPTION
[0062] The present invention will be described in detail below with reference to the accompanying drawings.
[0063] Reference Figure 1 The overall framework diagram of lane change trajectory planning for autonomous driving vehicles of the present invention includes an environment information acquisition module, a lane change decision module, a lane change trajectory planning module, and a simulation module. In the environment perception layer, the autonomous driving vehicle is equipped with sensors such as on-board cameras, laser radars, and millimeter-wave radars to realize the perception of environmental information, and can obtain the speed, acceleration, and position information of surrounding vehicles in the LTE-V2X communication mode with low latency and no packet loss; the behavior decision layer vehicle uses a pure strategy game lane change decision model to determine the vehicle driving behavior under the condition of ensuring safety; the trajectory planning layer uses a lane change trajectory optimization algorithm based on deep reinforcement learning to construct a longitudinal and lateral lane change trajectory for autonomous driving vehicles that takes into account lane change fuel consumption and comfort.
[0064] The main steps include:
[0065] Step 1: During the driving process of the autonomous driving vehicle, the vehicle-mounted sensing and communication equipment is used to obtain the speed, acceleration, and position information of the lane-changing vehicle and surrounding vehicles with low latency and no packet loss.
[0066] Step 2: Decouple the lateral and longitudinal kinematic constraints of the autonomous driving vehicle and establish a discretized lateral and longitudinal kinematic model of the vehicle.
[0067] Step 3: Based on the vehicle information obtained in step 1, a lane-changing decision model between the lane-changing vehicle and surrounding vehicles is established, a game profit function that considers safety and timeliness is constructed, and the game profit matrix is used to solve the optimal lane-changing decision of the lane-changing vehicle at the current moment.
[0068] Step 4: Based on the optimal lane changing decision obtained in step 3, a lane changing trajectory planning module for autonomous driving vehicles based on deep reinforcement learning is established with the goal of reducing fuel consumption and improving driving efficiency, and the optimal sequential acceleration decision information for the entire lane changing process is obtained.
[0069] Step 5, using the vehicle's lateral and longitudinal discretized kinematic model in step 2, the acceleration decision information obtained in step 4 is used to calculate the vehicle state at each time point in the lane changing process. The vehicle state includes vehicle speed, lateral position, and longitudinal position, and finally the vehicle's lane changing trajectory is obtained.
[0070] Example 1: As shown in Figure 3(c), in this example, the simulation lane scene is set as follows: the initial position of the lane-changing vehicle is at the starting point of the center line of the right lane (1.75, 0), the initial speed is randomly initialized between 12m / s (43.2km / h) and 20m / s (72km / h), the driving speed of the surrounding vehicles is 16m / s (57.6km / h), the front distance between the lane-changing vehicle and the vehicle in front of the original lane is 50m, and the front distance between the vehicle in front and behind the target lane is 60m, the target position is at the center line of the left lane (-1.75, 100), and the weight coefficients α1 and α2 in the game payoff are 0.6 and 0.4 respectively.
[0071] Example 2: As shown in Figure 4(c), in this example, the simulation lane scene is set as follows: the initial position of the lane-changing vehicle is at the starting point of the center line of the left lane (-1.75m, 0m), the initial speed is randomly initialized from 15m / s (54km / h) to 25m / s (90km / h), the driving speed of the surrounding vehicles is 20m / s (72km / h), the front distance between the lane-changing vehicle and the vehicle in front of the original lane is 50m, and the front distance between the vehicle in front and behind the target lane is 60m, the target position is at the center line of the right lane (1.75, 120), and the weight coefficients α1 and α2 in the game payoff are 0.6 and 0.4 respectively.
[0072] Step 2 specifically includes the following steps:
[0073] Step 2.1, the X-axis direction is the longitudinal direction of the vehicle, and the Y-axis direction is the lateral direction of the vehicle. The vehicle kinematic model is decoupled and discretized with Δt as the sampling time. The constraints of the discretized vehicle kinematic model separated in the longitudinal and lateral directions are as follows:
[0074] v xt =v x(t-1) +a xt Δt] (1)
[0075] v yt =v y(t-1) +a yt Δt] (2)
[0076]
[0077] 0 <v xt <v x,max ,0 <v yt <v y,max (5)
[0078]
[0079] where v xt , v yt They represent the longitudinal velocity and lateral velocity of the vehicle at time t, respectively, t and t Respectively represent the longitudinal and transverse coordinates of the vehicle at time t. Longitudinal and lateral acceleration a xt ,a yt It is obtained by the interaction between the vehicle and the algorithm in each time step Δt, v x,max and v y,max are the maximum values of longitudinal and transverse velocities, x max and max are the maximum values of the vertical and horizontal positions, respectively, f is the completion time of lane change. Thus, the position and speed of the vehicle at the next moment are obtained through the kinematic model until the vehicle reaches the lane change target position or leaves the lane.
[0080] Step 3 specifically includes the following steps:
[0081] Step 3.1: The lane-changing decision model is established. The participants in the game are the lane-changing vehicle M and the target lane following vehicle F. d and the vehicle in front of the target lane L d , the strategy set of lane-changing vehicle M is Φ1 = {m1, m2}, which are two pure strategies, where m1 represents lane change and m2 represents no lane change; the target lane vehicle F d , L d The strategy set is Φ2={d i1 , d i2}, i={F d ,L d}, where d i1 Indicates that vehicle i is allowed to change lanes, d i2 Indicates that vehicle i refuses to change lanes.
[0082] Step 3.2: The game lane-changing decision-making benefit function is established. To ensure the safety of lane changing and improve the efficiency of lane changing, a benefit function that reflects safety and driving efficiency is established. The benefit functions of the lane-changing vehicle and the target lane vehicle are defined as follows:
[0083] R M ,R D =α1*Rsafe +α2*R time . (7)
[0084] Among them, R safe , R time It represents the benefit obtained by the decision-making vehicle considering safety and timeliness. Its calculation formula is as follows (8)~(12); α1, α2 are weight coefficients, α1+α2=1, which represents the importance of different driving factors.
[0085]
[0086] where v Mx and v Mx They represent the longitudinal speeds of the lane-changing vehicle and the vehicle in front of the target lane, respectively. head is the distance between the two vehicles at the current moment, a Mx and The longitudinal acceleration of the lane-changing vehicle and the vehicle ahead in the target lane. min is the minimum safety distance required in the current state (with the lane-changing vehicle M and the front vehicle L in the target lane d For example), when the two vehicles move at a constant speed, it can be written as formula (10); t0 represents the time required to reach the destination while maintaining the original state, x target is the longitudinal coordinate of the lane change target point; t f It represents the completion time of lane change, which can be obtained from the trajectory planning part.
[0087] Step 3.3, according to the game benefit calculation formula in step 3.2, the game benefit of each lane-changing strategy is obtained, and the game benefit matrix is listed. The strategy with the largest benefit in the game benefit matrix is the current lane-changing decision. The game benefit matrix is as follows:
[0088] Table 1
[0089]
[0090] In Table 1, the lane-changing vehicle plays games with the vehicle in front of the target lane and the following vehicle, respectively, and four game results are obtained: the lane-changing vehicle changes lanes, and the vehicle in the target lane allows the lane change; the lane-changing vehicle changes lanes, and the vehicle in the target lane refuses to change lanes; the lane-changing vehicle does not change lanes, and the vehicle in the target lane allows the lane change; the lane-changing vehicle does not change lanes, and the vehicle in the target lane refuses to change lanes. The lane change can only be successfully executed when the lane-changing vehicle selects the "lane change" strategy, and the vehicle in front of the target lane and the following vehicle both make the "allow lane change" strategy.
[0091] Calculate the benefits of the lane-changing vehicle and the target lane vehicle under each strategy, taking into account safety and timeliness, and obtain the game benefit matrix. If there is a pure strategy (d in ,m n ),i={Fd ,L d}, n = 1, 2 so that the following formula (13) holds true, then (d in ,m n ) is the pure strategy Nash equilibrium of the current game. That is, it is the optimal lane-changing decision made by the vehicle in the current environment. According to this decision, the lane-changing trajectory planning sub-problem to be solved is determined, such as left lane changing, right lane changing, and lane keeping (no lane changing).
[0092]
[0093] Step 4 specifically includes the following steps:
[0094] Step 4.1, based on the lane-changing decision obtained in step 3, determine the initial state and the final state of the lane-changing vehicle, including the speed and the horizontal and vertical position coordinates of the vehicle.
[0095] Step 4.2, establish a lane-changing trajectory planning model for intelligent connected vehicles based on deep reinforcement learning, including the design of state space, action space, and reward function.
[0096] a. The state space contains all the information needed for the autonomous vehicle to change lanes. At each moment, the ordinate x of the current lane-changing vehicle needs to be obtained. t , the horizontal axis y t , longitudinal velocity v xt , lateral velocity v yt The state at each moment is represented by a four-tuple, s t =[x t ,v xt ;y t ,v yt ].
[0097] b. The action space mainly defines the actions taken by the autonomous vehicle. Combined with the longitudinal and lateral decoupling state equations (1-4) of vehicle kinematics, the longitudinal and lateral accelerations are used as the actions of the autonomous vehicle. Considering the driving comfort, the longitudinal acceleration a xt The value range is set to [-2m / s 2 ,2m / s 2 ]; Considering the lateral safety constraint and lateral comfort of the lane-changing vehicle, the lateral acceleration a yt The value range is [-0.2m / s 2 ,0.2m / s 2 The action at each moment is represented by a two-tuple, a t =[a xt ; a yt ].
[0098] c. Reward function design, expecting the lane-changing vehicle to drive along the center line of the lane as much as possible during the lane-changing process, and eventually be able to maintain a safe distance with the vehicle in front of the target lane at the same speed; and considering the instantaneous fuel consumption during the lane-changing process to achieve the goal of energy-saving driving. Reward R t The reward R along the center line of the target lane y , target speed reward R v , fuel consumption bonus R e , and the end-of-mission reward R d It consists of four parts.
[0099] R t =w y R y +w v R v +w e R e +w d R d . (14)
[0100] R y =-|y t -y target | (15)
[0101] R v =-a xt (v xt -v target ) (16)
[0102] R e =ln(MOE) (17)
[0103]
[0104] Where MOE is the instantaneous fuel consumption, including a combination of linear, quadratic and cubic velocity and acceleration terms, L k,q and M k,q represents the model coefficient of MOE under the kth power of speed and the qth power of acceleration. This term uses instantaneous fuel consumption as a penalty term to enable the vehicle to learn an energy-saving lane-changing method; w y 、w v 、w e and w d are weight coefficients of different benefits, indicating their importance, C1 and C2 are constants, y target represents the lateral coordinate of the centerline of the target lane, v target It represents the target speed when the vehicle reaches the end of lane change, and e is the base of the natural logarithm.
[0105] Step 4.3, the initial state of the lane-changing vehicle is used as the double-delayed deep deterministic policy gradient learning algorithm (TD3 algorithm, such as Figure 2The input of the lane change target is as shown in the figure, and the corresponding state of the lane change target is used as the end state of the lane change vehicle for TD3 algorithm training. During the training process, some decision behaviors with random noise added and the corresponding benefits are first collected and stored in the experience replay pool. After reaching a certain number, batch selection is performed for training until the benefits gradually stabilize, that is, the vehicle can learn the corresponding lane change sequence decision. The loss calculation of the evaluation network and the policy network of the TD3 algorithm during the training process is as follows:
[0106]
[0107] Step 4.4, according to the converged TD3 algorithm obtained in step 4.3, the initial state of the current vehicle and the lane change target state are input into the TD3 algorithm, the lane change trajectory planning model established in step 4.2 is solved, and the optimal sequential acceleration decision information of the entire lane change process is obtained;
[0108] Step 5 specifically includes the following steps:
[0109] The optimal lane-changing trajectory is calculated by using the vehicle's lateral and longitudinal discretized kinematic model and the optimal sequential acceleration decision information, including the vehicle's speed trajectory and the vehicle's longitudinal and lateral position trajectory during the lane-changing process.
[0110] The lane-changing trajectory generated by the autonomous vehicle for a left lane change is shown in Figure 3(a), and the fuel consumption comparison before and after training and the speed change during the lane change process are shown in Figures 3(b) and 3(c). The lane-changing trajectory generated by the vehicle for a right lane change is shown in Figure 4(a), and the fuel consumption comparison before and after training and the speed change during the lane change process are shown in Figures 4(b) and 4(c).
[0111] From the experimental training process of left and right lane changes, it can be concluded that the vehicle agent is in the trial and error stage in the first 150 rounds. At this time, the vehicle does not know how to change lanes, and the round always ends early due to abnormal endings. After about 150 rounds, after collecting enough historical data, it began to gradually learn and improve, and the cumulative benefits obtained in each round began to increase, indicating that the strategy learned by the vehicle agent is getting better and better, from the initial inability to complete the lane change task to being able to gradually complete the lane change and continuously optimize (in order to make it easier to see that the strategy learned by the agent is being optimized, the benefits are smoothed using a sliding average curve. The fluctuation of the benefit curve is due to the different benefits obtained by completing the lane change at different initial speeds.) Finally, it gradually stabilizes within a range, indicating the optimization process of the vehicle's strategy. At this time, the vehicle agent can reach the set lane change end point, and the speed of the lane change vehicle is equal to the speed of the environment vehicle, and the lane change task can be completed safely.
[0112] The fuel consumption issue is considered in the lane changing process, and the fuel consumption is used as a penalty for the agent. The greater the fuel consumption, the greater the penalty. It is hoped that the vehicle agent can complete the lane changing task in an energy-saving manner. The comparison of the average fuel consumption of the vehicle before and after training in the left and right lane changing tasks is shown in Figure 3(b) and Figure 4(b):
[0113] As shown in Figure 3(b) and Figure 4(b), before learning the TD3 algorithm, the average fuel consumption per step in the left lane change process was 0.030L / s, and the average fuel consumption per step in the right lane change process was 0.032L / s. After the algorithm was improved, the average fuel consumption per step in the left lane change process was 0.011L / s, and the average fuel consumption per step in the right lane change process was 0.018L / s. The average fuel consumption in the left and right lane change processes was reduced by 63% and 44% respectively, achieving the goal of energy-saving driving.
[0114] The TD3 algorithm used in the present invention is compared with the DDPG algorithm. The comparison of the training time used in each round in the left lane change and right lane change experiments is shown in Figure 5(a) and Figure 5(b):
[0115] As can be seen from Figure 5(a) and Figure 5(b), the training time used by the TD3 algorithm in the left lane change experiment and the right lane change experiment is less than that of DDPG. In the left lane change experiment, the total training speed and the average training speed per round of TD3 are about 12% higher than that of DDPG. In the right lane change experiment, the total training speed and the average training speed per round of TD3 are about 9% higher than that of DDPG. In summary, compared with the DDPG algorithm, the average training speed of the algorithm used in this paper is increased by about 10.5%. And the time required to complete the complete trajectory planning of the left and right lane change scenarios using the trained model is within 1.3s, and the time required for single-step planning is within 10ms, which can meet the real-time requirements. The required time is related to the processor performance, and the time required will be shorter when using a high-performance processor.
[0116] The present invention uses a Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm that can handle continuous control quantities, which can realize the control of the speed and acceleration of the autonomous driving vehicle, solve the problem of loss of precision control caused by discretizing the action space using the DQN algorithm, and the problem of overvaluation of the benefit function that occurs when using the DDPG algorithm, and can more accurately evaluate the vehicle control behavior. The network is trained by storing historical experience data obtained through the interaction between the vehicle and the environment. It does not need to be trained through a large amount of real lane change data like machine learning, which saves pre-processing operations such as cleaning and screening of the data and improves flexibility. Instantaneous fuel consumption and target speed control are considered in the reward function, and a safe, comfortable and green lane change trajectory is finally obtained.
Claims
1. A lane-changing trajectory planning method for an autonomous driving vehicle, characterized in that: The process includes the following: Obtaining vehicle information: The vehicle information includes: speed, acceleration and position information of the lane-changing vehicle, and speed, acceleration and position information of surrounding vehicles; According to the vehicle information, the game lane-changing decision model between the lane-changing vehicle and surrounding vehicles, and the game payoff function considering safety and timeliness, and using the game payoff matrix, the optimal lane-changing decision of the lane-changing vehicle at the current moment is solved; According to the optimal lane-changing decision, with the goal of reducing fuel consumption and improving driving efficiency, and using a lane-changing trajectory planning model for an autonomous driving vehicle based on deep reinforcement learning, the optimal sequential acceleration decision information for the entire lane-changing process is obtained; The vehicle state of the lane-changing vehicle at each time point in the lane-changing process is calculated by using the vehicle lateral and longitudinal discretized kinematic model and the optimal sequential acceleration decision information, and the lane-changing trajectory of the lane-changing vehicle is obtained according to the vehicle state of the lane-changing vehicle; The process of establishing the vehicle lateral and longitudinal discretized kinematic model includes: In the XOY plane rectangular coordinate system, the X-axis direction is the longitudinal direction of the vehicle, and the Y-axis direction is the lateral direction of the vehicle. The vehicle kinematic model is decoupled and discretized with Δt as the sampling time. The discretized vehicle kinematic model and constraint conditions for the lateral and longitudinal separation are obtained as follows: v xt =v x(t-1) +a xt Δt v yt =v y(t-1) +a yt Δt 0<v xt <v x,max ,0<v yt <v y,max Among them, v xt and v yt They represent the longitudinal velocity and lateral velocity of the vehicle at time t respectively; v x(t-1) and v y(t-1) They represent the longitudinal speed and lateral speed of the vehicle at time t-1 respectively; time t-1 is the moment before time t; x t and t Respectively represent the ordinate and abscissa of the vehicle at time t; x (t-1) and (t-1) They represent the ordinate and abscissa of the vehicle at time t-1 respectively; the longitudinal acceleration a xt and lateral acceleration a yt Obtained by the interaction between the vehicle and the algorithm at each time step Δt, v x,max and v y,max are the maximum longitudinal velocity and the maximum lateral velocity, respectively, x max and max are the maximum values of the longitudinal position and the horizontal position, respectively, t f is the completion time of the lane change; The position and speed of the vehicle at the next moment are obtained through the kinematic model until the vehicle reaches the lane change target position or leaves the lane; The game lane-changing decision model between the lane-changing vehicle and surrounding vehicles is as follows: The participants of the game are the lane-changing vehicle M and the target lane following vehicle F. d and the vehicle in front of the target lane L d , the strategy set of lane-changing vehicle M is Φ1 = {m1, m2}, where m1 represents lane change and m2 represents no lane change; the target lane following vehicle F d and the vehicle in front of the target lane L d The strategy set is Φ2={d i1 , d i2 }, i={F d ,L d }, where d i1 Indicates that vehicle i is allowed to change lanes, d i2 Indicates refusal to change lanes; The game profit function considering security and timeliness is as follows: R M ,R D =α1*R safe +α2*R time Among them, R safe and R time They represent the benefits obtained by the decision-making vehicle considering safety and timeliness respectively; α1 and α2 are weight coefficients, α1+α2=1, representing the importance of different driving factors; Among them, v Mx and v ldx They represent the longitudinal speed of the lane-changing vehicle and the longitudinal speed of the vehicle in front of the target lane, respectively. head is the distance between the two vehicles at the current moment, a Mx and is the longitudinal acceleration of the lane-changing vehicle and the vehicle in front of the target lane; P min is the minimum safety distance required in the current state; t0 represents the time required to reach the destination in the original state, x target is the longitudinal coordinate of the lane change target point; t f represents the completion time of the lane change, which is obtained from the trajectory planning part; The game payoff matrix is shown in Table 1: Table 1 Among them, R M represents the game benefit that the lane-changing vehicle can obtain under the current strategy, R D It represents the game benefits that the vehicle in the target lane can obtain under the current strategy; The lane-changing vehicle plays games with the vehicle in front and the following vehicle in the target lane respectively, and four game results are obtained: the lane-changing vehicle changes lanes, and the vehicle in the target lane allows the lane change; the lane-changing vehicle changes lanes, and the vehicle in the target lane refuses to change lanes; the lane-changing vehicle does not change lanes, and the vehicle in the target lane allows the lane change; the lane-changing vehicle does not change lanes, and the vehicle in the target lane refuses to change lanes; the lane change is executed only when the lane-changing vehicle chooses a lane-changing strategy and the vehicle in front and the following vehicle in the target lane both make strategies to allow the lane change.
2. The method for planning lane change trajectory of an autonomous driving vehicle according to claim 1, characterized in that: The process of solving the optimal lane-changing decision of the lane-changing vehicle at the current moment includes: If there is a pure strategy [d in ,m n ],i={F d ,L d }, n=1,2 makes the following equation true, then [d in ,m n ] is the pure strategy Nash equilibrium of the current game. The pure strategy of the current game is the optimal lane-changing decision made by the vehicle under the current environment. The lane-changing trajectory planning sub-problem to be solved is determined according to this decision. The lane-changing trajectory planning sub-problem includes: left lane changing, right lane changing or lane keeping; Where, d in ,i={F d ,L d }, n = 1 means that vehicle i in the target lane is allowed to change lanes, d in ,i={F d ,L d }, n = 2 means that vehicle i in the target lane refuses to change lanes, m n , n=1 means lane-changing vehicle M changes lanes, m n , n = 2 means that the lane-changing vehicle M does not change lanes, Φ1 represents the strategy set of the lane-changing vehicle M, m represents any strategy in the lane-changing vehicle strategy set Φ1, and Φ2 represents the target lane following vehicle F d and the vehicle in front of the target lane L d The strategy set, d i represents any strategy in the strategy set Φ2 of vehicle i in the target lane.
3. The method for lane-changing trajectory planning of an autonomous driving vehicle according to claim 1, characterized in that: The lane-changing trajectory planning model for an autonomous driving vehicle based on deep reinforcement learning includes a state space, an action space, and a reward function; The state space contains all the information required for the automatic driving vehicle to change lanes. The vertical coordinate x of the current lane-changing vehicle is obtained at each moment. t , horizontal axis y t , longitudinal speed v xt and the lateral velocity v yt ; The state at each moment is represented by a four-tuple, s t =[x t ,v xt ;y t ,v yt ]; The action space defines the actions taken by the autonomous vehicle. Combined with the longitudinal and lateral decoupling state equations of vehicle kinematics, the longitudinal and lateral accelerations are used as the actions of the autonomous vehicle. The longitudinal acceleration a xt The value range is set to [-2m / s 2 ,2m / s 2 ]; lateral acceleration a yt The value range is [-0.2m / s 2 ,0.2m / s 2 ]; the action at each moment is represented by a two-tuple, a t =[a xt ; a yt ]; The reward function R t The reward R along the center line of the target lane y , Target Speed Reward R v , fuel consumption reward R e and end-of-mission reward R d It consists of four parts, as follows: R t =w y R y +w v R v +w e R e +w d R d R y =-|y t -y target | R v =-a xt (v xt -v target ) R e =ln(MOE) Where MOE is the instantaneous fuel consumption, including a combination of linear, quadratic and cubic velocity and acceleration terms, L k,q and M k,q represents the model coefficient of MOE at the kth power of velocity and the qth power of acceleration, L k,q and M k,q Using instantaneous fuel consumption as a penalty term, the vehicle learns an energy-saving lane-changing method; w y 、w v 、w e and w d are weight coefficients of different benefits, indicating their importance, C1 and C2 are constants, y target represents the lateral coordinate of the centerline of the target lane, v target It represents the target speed when the vehicle reaches the end of lane change, and e is the base of the natural logarithm.
4. The method for planning lane change trajectory of an autonomous driving vehicle according to claim 3, characterized in that: The training process of the lane change trajectory planning model for the autonomous driving vehicle based on deep reinforcement learning includes: The initial state of the lane-changing vehicle is used as the input of the double-delay deep deterministic policy gradient learning algorithm, and the corresponding state of the lane-changing target is used as the end state of the lane-changing vehicle. The double-delay deep deterministic policy gradient learning algorithm is trained. During the training process, the decision-making behaviors and corresponding benefits with added random noise are first collected and stored in the experience replay pool. After reaching the preset number, batch selection is performed for training until the benefits gradually stabilize and the double-delay deep deterministic policy gradient learning algorithm converges.
5. The method for planning lane change trajectory of an autonomous driving vehicle according to claim 4, characterized in that: According to the optimal lane changing decision, with the goal of reducing fuel consumption and improving driving efficiency, and using the lane changing trajectory planning model of the autonomous driving vehicle based on deep reinforcement learning, the optimal sequential acceleration decision information of the entire lane changing process is obtained. According to the converged double-delay deep deterministic policy gradient learning algorithm, the initial state of the current vehicle and the lane changing target state are input into the double-delay deep deterministic policy gradient learning algorithm to solve the lane changing trajectory planning model of the autonomous driving vehicle based on deep reinforcement learning, and the optimal sequential acceleration decision information of the entire lane changing process is obtained.
6. A lane-changing trajectory planning system for an autonomous driving vehicle, characterized in that: The method for planning lane change trajectories of an autonomous driving vehicle according to any one of claims 1 to 5 comprises: Acquiring environment information module: used to acquire vehicle information, the vehicle information including: speed, acceleration and position information of the lane-changing vehicle, and speed, acceleration and position information of surrounding vehicles; Lane-changing decision module: used to solve the optimal lane-changing decision of the lane-changing vehicle at the current moment according to the vehicle information, the game lane-changing decision model between the lane-changing vehicle and the surrounding vehicles, and the game profit function considering safety and timeliness, and using the game profit matrix; Lane-changing trajectory planning module: used to obtain the optimal sequential acceleration decision information of the entire lane-changing process based on the optimal lane-changing decision, with the goal of reducing fuel consumption and improving driving efficiency, and using the lane-changing trajectory planning model of the autonomous driving vehicle based on deep reinforcement learning; Simulation module: used to calculate the vehicle state of the lane-changing vehicle at each time point in the lane-changing process by using the vehicle's lateral and longitudinal discretized kinematic model through the optimal sequential acceleration decision information, and obtain the lane-changing trajectory of the lane-changing vehicle according to the vehicle state of the lane-changing vehicle.
Citation Information
Patent Citations
Lane changing decision-making method and system for autonomous vehicle based on rolling game
CN110297494A
Method for establishing automatic driving lane changing decision model in hybrid driving environment
CN110298131A
