An integrated decision-making method for autonomous driving behavior based on deep reinforcement learning

Through deep reinforcement learning algorithms, the lane advantage function and dangerous action shielding mechanism are built, which solves the problem of integrating lane transformation and follow-up behavior in intelligent driving, and improves driving safety and efficiency.

CN115320640BActive Publication Date: 2024-12-03SOUTHEAST UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202211077434.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-09-05
Publication Date
2024-12-03
Estimated Expiration
2042-09-05

AI Technical Summary

Technical Problem

The existing technology is difficult to effectively integrate lane transformation and vehicle follow-up behavior, resulting in insufficient complexity and safety of intelligent driving control.

Method used

Adopting an integrated decision-making method of autonomous driving behavior based on deep reinforcement learning, the lane transformation and follow-up behavior model is constructed through D3QN and DDPG algorithms, the lane advantage function and dangerous action shielding mechanism are introduced, and coupled training is carried out to realize lane transformation and follow-up behavior. comprehensive decision-making control.

Benefits of technology

It simplifies the complexity of the decision model, improves driving safety and efficiency, ensures coordinated cooperation between lane transformation and follow-up behavior, and enhances the generalization ability and authenticity of the model.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115320640B_ABST
    Figure CN115320640B_ABST
Patent Text Reader

Abstract

The present invention discloses an integrated decision-making method for autonomous driving behaviors based on deep reinforcement learning. Starting from the priority and logic of two basic driving behaviors, namely lane changing and following behaviors, the upper and lower layers of deep reinforcement learning models are first used to process large-scale mixed state spaces and ensure the composite action output of lane changing decisions and following decisions. The D3QN algorithm is used in the upper model to distinguish the potential value of the environment and the value of the lane changing decision. The DDPG algorithm is used in the lower model to process the following decision to output continuous vehicle speed control. After the two algorithms are coupled and trained using mixed standard driving conditions, the model is tested in a highly complex driving scenario reconstructed using NGSIM data. The integrated driving decision model proposed in the present invention improves the synergy of following and lane changing decisions, and integrates the lane advantage function and the dangerous action shielding mechanism into the decision model, which significantly improves the driving speed and is superior to other intelligent driving decision models.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to an integrated decision-making method for autonomous driving behavior based on deep reinforcement learning, and belongs to the technical field of intelligent driving decision-making development. Background Art

[0002] Intelligent vehicle decision-making systems can fulfill driving intentions more safely and efficiently. The key technology for vehicles to operate in complex traffic scenarios lies in the accurate and rapid response and execution of decision control. The task of decision control is to issue instructions for complex vehicle actions (such as following, obstacle avoidance, lane changing and overtaking) and pass them to the vehicle's action execution module. From a micro perspective, vehicle control is divided into longitudinal control and lateral control. The combination of following and lane changing can realize most driving intentions, such as following, overtaking and obstacle avoidance.

[0003] However, relatively few models have been developed for lane changing, given the randomness and complexity of lane changing behavior, which interacts with the longitudinal motion of the vehicle. Therefore, a comprehensive study of lane changing behavior and vehicle following behavior is crucial for intelligent driving control. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide an integrated decision-making method for autonomous driving behavior based on deep reinforcement learning, which uses a deep reinforcement learning algorithm to integrate lane changing and vehicle following behavior into a comprehensive decision-making control model, thereby simplifying the complexity of the comprehensive model and more effectively exploring the optimal decision in a model-free manner.

[0005] The present invention adopts the following technical solutions to solve the above technical problems:

[0006] An integrated decision-making method for autonomous driving behavior based on deep reinforcement learning includes the following steps:

[0007] Step 1: construct a vehicle lane-changing behavior model as the upper layer of the intelligent driving decision-making integrated model, introduce a lane advantage function and a dangerous action shielding mechanism, and use the D3QN algorithm to train the vehicle lane-changing behavior model to distinguish the potential value of the environment and the value of the lane-changing action when making a lane-changing decision;

[0008] Step 2, constructing a vehicle following behavior model as the lower layer of the intelligent driving decision integration model, using the DDPG algorithm to train the vehicle following behavior model, processing the following decision and outputting continuous vehicle speed control;

[0009] Step 3: Select a training environment under mixed standard driving conditions, perform coupled training on the upper and lower models, and integrate the upper and lower models.

[0010] As a preferred embodiment of the present invention, the specific process of step 1 is as follows:

[0011] 1.1, define the action of the Markov decision process:

[0012] action = {-1, 0, 1}

[0013] When the traffic scene is 3 lanes, -1 means selecting the right lane, 0 means selecting the middle lane, and 1 means selecting the left lane;

[0014] 1.2, define the reward of the Markov decision process:

[0015] reward=α*v r +β*lc+γ*c

[0016] Among them, α and β represent the reward coefficient of the host vehicle to the new speed state when changing lanes and the penalty parameter for frequent lane changes, respectively. r represents the speed of the main vehicle, lc represents whether a lane change occurs during the second of executing the current action, γ represents the penalty coefficient used to reduce the number of collisions, and c represents the number of collisions during the following process;

[0017] 1.3, define the state of the Markov decision process:

[0018]

[0019] Among them, a r Indicates the acceleration of the main vehicle, lane_index indicates the lane number of the lane the main vehicle is currently in, h i They represent the speed of the previous vehicle, the acceleration of the previous vehicle, and the headway in the i-th lane, pv i represents the lane advantage function value, N represents the total number of lanes;

[0020] 1.4, assuming that the main vehicle wants to catch up with the front vehicle as quickly as possible, ideally, the main vehicle goes through three stages in the process of following the front vehicle: accelerating to the maximum speed, maintaining the maximum speed, and reducing to the speed of the front vehicle; the following lane advantage function is introduced:

[0021]

[0022] in, represents the speed of the front vehicle in the i-th lane, They represent the duration of the three stages experienced by the main vehicle in the ith lane in the process of following the front vehicle, h represents the headway between the main vehicle and the front vehicle, LCF is the lane change feasibility of the current lane, and the future average speed obtained by the main vehicle in the ith lane based on the principle of catching up with the front vehicle as quickly as possible is used. As the lane advantage function value pvi;

[0023] 1.5, calculate the time t for the main vehicle to accelerate from the current speed to the maximum speed a , the time t for the main vehicle to decelerate from the maximum speed to the speed of the front vehicle d , and the distance S traveled by the host vehicle during acceleration and deceleration respectively a , S d :

[0024]

[0025]

[0026]

[0027]

[0028] Among them, v max 、a max 、a min Respectively represent the maximum speed, maximum acceleration, and minimum acceleration of the main vehicle, v f Indicates the speed of the vehicle ahead;

[0029] 1.6, calculate the distance s between the host vehicle and the vehicle in front during acceleration and deceleration error :

[0030] s error =(t a +t d )*v f +h-(S a +S d )

[0031] 1.7, define each duration:

[0032]

[0033]

[0034]

[0035] When error When it is less than 0, the maximum speed that the vehicle can travel is less than the maximum speed limit of the road;

[0036] 1.8, if the maximum speed of the vehicle is less than the maximum speed limit of the road, the maximum speed v′ max The calculation of is as follows:

[0037]

[0038]

[0039]

[0040] Among them, t′1 is the speed of the main vehicle v r Accelerate to the maximum feasible speed v′ max , t′3 is the time from the maximum feasible speed v′ max Slow down to the speed of the vehicle in front v f time;

[0041] Solve t′1, t′3 and substitute into the formula in 1.4 to get the advantage function value of the i-th lane;

[0042] 1.9, based on the advantage function, set up a dangerous action shielding mechanism. If a lane cannot meet the lane change conditions, the advantage function value is 0. When the model outputs the action of changing to this lane, it means that the action is dangerous.

[0043] 1.10, set the conditions for determining the feasibility of lane change:

[0044] v′ en >a safe And v′ fn >a safe

[0045] Among them, v′ en Indicates the deceleration that the host vehicle should take, v′ fn Indicates the deceleration that the new following vehicle should take after changing lanes, a safe Indicates the absolute value of the maximum safe deceleration of the vehicle on the road. When the absolute value of the deceleration of the main vehicle and the new following vehicle is greater than the absolute value of the maximum safe deceleration after the lane change action is taken, it means that the lane change process is dangerous.

[0046] 1.11, the dangerous action shielding mechanism is set to minimize the Q value corresponding to the dangerous action in the Dueling part structure of the D3QN network.

[0047] As a preferred solution of the present invention, in step 2, the target of the vehicle following behavior model is set to be consistent with the benefit target of the upper layer vehicle lane changing behavior model;

[0048] For the vehicle following behavior model, define the action:

[0049] action = {a∈[-3, 3]}

[0050] Among them, a represents action;

[0051] Define reward:

[0052]

[0053] Among them, α′ represents the speed reward coefficient of the vehicle in the following state, β′ represents the penalty coefficient for collision in the next second, γ′ represents the reward coefficient for improving driving comfort, and v r represents the speed of the main vehicle, c′ represents whether dangerous following behavior occurs within one second of executing the current action, Indicates the square value of acceleration;

[0054] Define the state:

[0055] {v r , v f ,h,a f}

[0056] Among them, v f represents the speed of the vehicle in front, h represents the distance between the main vehicle and the vehicle in front, a f Indicates the acceleration of the vehicle ahead.

[0057] As a preferred solution of the present invention, in step 3, multiple standard operating conditions are selected for combination, and the combined operating conditions are used as reference operating conditions for vehicle flow. The combined operating conditions are required to cover different vehicle driving scenarios of acceleration, deceleration and idling of the vehicle, including low-speed operation characteristics of urban roads, high-speed section operation characteristics, large instantaneous speed change characteristics, and relatively gentle change characteristics;

[0058] The leading vehicle in the traffic flow is allowed to accelerate and decelerate according to the combined operating conditions, and the remaining vehicles are allowed to drive freely according to the IDM model and the MOBIL model. When the main vehicle continues to overtake and becomes the leader of the traffic flow, the training object is replaced with the last vehicle in the traffic flow, and this process is repeated to obtain more training samples.

[0059] As a preferred solution of the present invention, in step 3, the intelligent driving decision-making integrated model is trained by coupling training. During the coupling training process, the upper and lower layers of the intelligent driving decision-making integrated model share the speed reward and collision penalty of the intelligent agent. The lower layer vehicle following behavior model adjusts the speed of the intelligent agent according to the lane change selection of the upper layer, and finally realizes the integration of the upper layer and the lower layer models.

[0060] Compared with the prior art, the present invention adopts the above technical solution and has the following technical effects:

[0061] 1. The present invention adopts a hierarchical reinforcement learning method to make the model more comprehensive. Different reinforcement learning algorithm designs are used for the upper and lower layers, taking into account the differences in influencing factors and output behaviors between the two tasks; a shared state space is constructed, and the integrated model of the hierarchical architecture is coupled with training to ensure the continuity of the complex driving operation process and solve the problem of poor coordination and collaboration.

[0062] 2. The present invention establishes a lane advantage function and a dangerous action shielding mechanism for the decision model, which is used for lane changes, thereby eliminating collisions and effectively ensuring driving safety.

[0063] 3. The present invention uses traffic environments under multiple standard working conditions to build a multi-lane training scenario, and combines SUMO software to perform traffic simulation, which improves the authenticity and complexity of the training environment and enhances the generalization ability of the model; its effectiveness is verified by the results of trajectory reconstruction based on NGSIM data. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 It is an architecture diagram of an integrated decision-making method for autonomous driving behavior based on deep reinforcement learning of the present invention;

[0065] Figure 2 Schematic diagram of lane-changing scenario of the present invention; (a) and (d) indicate that when the distance between the leading vehicles is sufficient, the time for the main vehicle to accelerate to the maximum speed, maintain the maximum speed, and drop to the speed of the leading vehicle is t1, t2, and t3 respectively; (b), (c), (e), and (f) indicate that after completing the acceleration, the main vehicle suddenly brakes to prevent a collision, that is, the deceleration process from t1 to t3;

[0066] Figure 3 It is a schematic diagram of the dangerous action shielding mechanism of the present invention;

[0067] Figure 4 This is a flow chart of the overall idea of ​​an integrated decision-making method for autonomous driving behavior based on deep reinforcement learning in the present invention. DETAILED DESCRIPTION

[0068] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention.

[0069] The present invention proposes a method for developing an integrated model of vehicle autonomous speed change and lane change decision based on deep reinforcement learning. The general framework is as follows: Figure 1 As shown in Figure 3, the decision-making process is divided into lane selection and longitudinal speed control process.

[0070] The present invention utilizes a traffic simulator to simulate the driving environment and the dynamic characteristics of the vehicle. When no collision occurs, it is assumed that the acceleration and deceleration performance of the vehicle can ensure safety.

[0071]

[0072] The action controlled by the car-following model is the speed change that the main vehicle needs to take at the current time step. It is a one-dimensional variable with a maximum value of the positive acceleration upper limit and a minimum value of the negative deceleration upper limit.

[0073] The present invention uses hierarchical reinforcement learning to establish a comprehensive decision model for lane change and car following, and decomposes the driving task into lane change decisions and vehicle longitudinal speed control decisions with time and logical priorities. Figure 4 As shown, the specific steps are as follows:

[0074] Step 1: Build a vehicle lane-changing behavior model as the upper layer of the intelligent driving decision-making integrated model. Use the D3QN (Dueling Double Deep Q Network) algorithm for training to distinguish the potential value of the environment and the value of choosing a lane-changing action when making a lane-changing decision, and innovatively introduce a lane advantage function and a dangerous action shielding mechanism;

[0075] Step 2: Construct a vehicle following behavior model as the lower layer of the intelligent driving decision-making integrated model. Use the DDPG (Deep Deterministic Policy Gradient) algorithm for training, process the following decision and output continuous vehicle speed control to ensure the efficiency and stability of vehicle driving;

[0076] Step 3: Select mixed standard driving conditions to build a highly complex training environment, improve the model's adaptability to real driving scenarios, couple the upper and lower layer models for training, and achieve effective integration of the two-layer model.

[0077] Step 1 specifically includes the following steps:

[0078] D3QN with Prioritized Experience Replay adjusts the value of each action by adding a competition network. The strategy of this algorithm is applicable to the lane changing process. When selecting a lane, each lane has its advantages. In addition, in this algorithm, this advantage can be expressed as the value of the state. The method of optimal selection by action value can be used in combination with the lane advantage function proposed in this invention.

[0079] The present invention mainly considers strategic lane change and forced lane change, pursues greater overall speed benefits, and does not consider path selection and ensuring lane change safety, and defines the lane selection model as follows:

[0080] Define the action:

[0081] action = {-1, 0, 1}

[0082] Take a 3-lane traffic scenario as an example: -1 in the formula means selecting the right lane, 0 means selecting the middle lane, and 1 means selecting the leftmost lane.

[0083] Define the reward:

[0084] reward=α*v r+β*lc+γ*c

[0085] Among them, α and β represent the reward coefficient of the host vehicle for the new speed state when changing lanes and the penalty parameter for frequent lane changes, respectively. γ refers to the penalty coefficient used to reduce the number of collisions. r represents the speed of the main vehicle, lc represents whether a lane change occurs in the second when the current action is executed, and c represents the collision caused by the lane change. After multiple tests, the present invention sets α to 0.9, β to -0.1, and γ to -20.

[0086] Define the state:

[0087]

[0088] Among them, v r represents the speed of the host vehicle, a r represents acceleration, lane_index represents the lane number of the lane the vehicle is currently in, h i They represent the speed of the previous vehicle, the acceleration of the previous vehicle, and the headway in lane i, pv i refers to the lane advantage function value, and N refers to the total number of lanes.

[0089] The present invention introduces a lane advantage function. Figure 2 As shown in the figure, according to the relationship between the speed of the main vehicle and the speed of the front vehicle and the distance between the front vehicles, the following process is divided into six states, v_f, v_r, v_max represent the speed of the front vehicle, the speed of the main vehicle and the maximum speed limit of the road respectively. The present invention assumes that the main vehicle wants to catch up with the front vehicle as quickly as possible. Ideally, in the process of following the front vehicle, the main vehicle will go through three stages: accelerating to the maximum speed, maintaining the maximum speed, and reducing to the speed of the front vehicle. Their durations are t1, t2, and t3 respectively. Figure 2 As shown in (a) and (d) in , when the distance between the leading vehicles is sufficient, the main vehicle will have the three processes described above. When the distance is reduced, the main vehicle can only accelerate to a certain speed and then brake suddenly to prevent collision with the leading vehicle, that is, Figure 2 v_max in (b) and (e), or Figure 2 After completing the acceleration, the host vehicle will brake suddenly to prevent a collision, that is, Figure 2 The deceleration process from t1 to t3 shown in (b), (c), (e), and (f).

[0090]

[0091]

[0092] in, refers to the speed of the front vehicle in the i-th lane, It refers to the duration of the three stages that the main vehicle in the ith lane will experience in the process of following the front vehicle, h refers to the headway of the current lane, and LCF is the feasibility of lane change in the current lane. The future average speed obtained by the main vehicle in the ith lane based on the principle of catching up with the front vehicle as quickly as possible is used. is the advantage value pv i .

[0093] Calculate the acceleration v′ that needs to be taken after the lane change to avoid collision between the host vehicle and the new follower _ego_new , v′ _follow_new , compare with the maximum acceleration and maximum deceleration to determine whether the above acceleration and deceleration are safe, and perform the following calculations:

[0094]

[0095]

[0096]

[0097]

[0098] Among them, v max 、a max 、a min Represent the maximum speed, maximum acceleration and minimum acceleration of the main vehicle respectively, t a is the time it takes for the host vehicle to accelerate from its current speed to its maximum speed, t d is the time it takes for the main vehicle to decelerate from its maximum speed to the speed of the preceding vehicle, S a , S d Refers to the distance traveled during acceleration and deceleration, respectively.

[0099] Calculate the distance s between the host vehicle and the vehicle in front during acceleration and deceleration error :

[0100] s error =(t a +t d )*v f +h-(S a +S d )

[0101] Wherein, h represents the actual headway.

[0102] Define the time periods:

[0103]

[0104]

[0105]

[0106] Among them, when s error When it is less than 0, the main vehicle has caught up with the front vehicle. At this time, the main vehicle cannot meet the complete process of accelerating to the maximum speed and decelerating to the speed of the front vehicle. The process of maintaining the maximum speed does not exist. The maximum speed that the vehicle can travel will be less than the maximum speed limit of the road.

[0107] If the maximum speed of the vehicle is less than the maximum speed limit of the road, calculate the maximum speed v′ max The steps are as follows:

[0108]

[0109]

[0110]

[0111] Where t′1 is the main vehicle speed v r Accelerate to the maximum feasible speed v′ max The time required, t′3, is the maximum feasible speed v′ max Slow down to the speed of the vehicle in front v f The time required, h is the headway between the vehicles.

[0112] Substituting t′1, t′3 into the formula of the advantage function, we can obtain the advantage function of a specific lane i.

[0113] Based on the lane advantage function, the present invention introduces a dangerous action shielding mechanism into the decision model. Figure 3 As shown in the figure, it is set as follows: when a lane cannot meet the lane change conditions, the advantage value is 0. When the model outputs the action of changing to this lane, it means that the action is dangerous. When a dangerous action in the current state is detected, the corresponding value of the action value network in the D3QN network is minimized. This dangerous action will not be selected.

[0114] Set simple lane change feasibility determination conditions:

[0115] v′ en , v′ fn >a safe

[0116] In the formula, v′ en Indicates the deceleration that the host vehicle should take, v′ fn Indicates the deceleration rate that the new follower should take after changing lanes. safe It indicates the maximum absolute safe deceleration of the vehicle on the road. When the absolute value of the deceleration of the main vehicle and the new rear vehicle after the lane change action is greater than the absolute value of the safe deceleration, it indicates that the lane change process is dangerous.

[0117] The dangerous action shielding mechanism is set to minimize the Q value corresponding to the dangerous action in the Dueling part structure of the D3QN network.

[0118] Step 2 specifically includes the following steps:

[0119] When setting up hierarchical reinforcement learning, considering that the upper and lower algorithms need to be coupled for training, the goal setting of the car-following model is consistent with the goal of the upper lane-changing model. The car-following model studied in the present invention mainly pursues greater speed benefits under the premise of ensuring safety.

[0120] The car-following model is defined as follows:

[0121] Define the action:

[0122] action = {a∈[-3, 3]}

[0123] Define the reward:

[0124]

[0125] Among them, α′ represents the speed reward coefficient of the vehicle in the following state, β′ represents the penalty coefficient for a collision in the next second, and γ′ refers to the reward coefficient used to improve driving comfort, which aims to improve comfort through a smaller acceleration value. r is the speed of the main vehicle, c′ indicates whether dangerous following behavior occurs within one second of executing the current action. It represents the square value of acceleration. The larger the value, the worse the comfort. After many tests, α′ is 0.9, β′ is -20, and γ′ is -2.25.

[0126] Define the state:

[0127] {v r , v f ,h,a f}

[0128] where v r Indicates the speed of the host vehicle, a f represents the acceleration of the front vehicle, and h represents the distance between the main vehicle and the front vehicle.

[0129] For different decision-making tasks, the present invention uses the D3N algorithm and the DDPG algorithm as the upper and lower layers of hierarchical decision-making, shares the reward values ​​of the two algorithms in terms of efficiency improvement and security, and then performs coupled training. The pseudo code of the algorithm is shown in Table 1.

[0130] Table 1 Pseudo code of the two-layer reinforcement learning algorithm framework

[0131]

[0132]

[0133] Step three specifically includes the following steps:

[0134] Sumo simulation environment construction:

[0135] In order to improve the authenticity of the model, the present invention uses SUMO software as the traffic simulation platform. The computer configuration and environment configuration are Windows 10, python 3.6, tensorflow 1. The CPU is AMD R7-5800H, the GPU is RTX3060, and the present invention uses the Traci interface in python to control the following and lane changing process of the intelligent agent. The road network configuration parameters of the traffic simulation environment are shown in Table 2.

[0136] Table 2 Configuration parameters of traffic simulation environment

[0137]

[0138]

[0139] The present invention selects China_city working condition, HWFET working condition, UDDS working condition and US06_2 standard working condition for combination, and takes the combined working condition as the reference working condition for vehicle flow. The working condition is described as follows:

[0140] The combined working condition is 45,153.53 meters long and takes 3,961 seconds to run, covering a variety of vehicle driving scenarios such as acceleration, deceleration, and idling. It includes low-speed operation characteristics on urban roads, high-speed operation characteristics, large instantaneous speed changes, and relatively gentle changes.

[0141] In order to enhance the complexity of the environment, the present invention accelerates and decelerates the leading vehicle in the traffic flow according to the combined working conditions, and the remaining vehicles drive freely according to the IDM model and the Mobil model. When the main vehicle continues to overtake and becomes the leader of the traffic flow, the training object is replaced with the last vehicle in the traffic flow, and this process is repeated to obtain more training samples.

[0142] With respect to the dual-layer decision controller of the present invention, the present invention adopts a coupled training method to train the dual-layer reinforcement learning model. During the coupled training process, the dual-layer controller shares the speed reward and collision penalty of the agent. The lower-layer following model will continuously adjust the speed of the agent according to the lane change selection of the upper layer, and finally achieve a good coupled optimization of the dual-layer decision controller.

[0143] Due to the difficulty of coupled training, the present invention uses the IDM car-following model with the upper lane-changing model for 4,000 kilometers of training, and finally conducts state-sharing car-following and lane-changing coupled training for 6,000 kilometers, where the car-following model involved in the training is a deep reinforcement learning car-following model that has been trained for 773,000 time steps. The overall network structure configuration is shown in Table 3.

[0144] Table 3 Overall network structure configuration

[0145]

[0146]

[0147] The simulation results in SUMO show that the double-layer model can increase the driving speed of the original data by 23.99%, which is more efficient than other models.

[0148] The present invention demonstrates the superiority of the dual-layer model using coupled training. While both the dual-layer model and the model using D3QN and IDM can achieve the optimal speed, the dual-layer model can reduce the number of lane changes. It can be seen that optimizing the dual-layer model can make lane change and car following work better together.

[0149] The above embodiments are only for illustrating the technical idea of ​​the present invention, and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the present invention.

Claims

1. A method for integrated decision-making of autonomous driving behavior based on deep reinforcement learning, characterized by comprising the following steps: Step 1: construct a vehicle lane-changing behavior model as the upper layer of the intelligent driving decision-making integrated model, introduce the lane advantage function and the dangerous action shielding mechanism, and use the D3QN algorithm to train the vehicle lane-changing behavior model to distinguish the potential value of the environment and the value of the lane-changing action when making a lane-changing decision. The specific process is as follows: 1.1, define the action of the Markov decision process: action = {-1,0,1} When the traffic scene is 3 lanes, -1 means selecting the right lane, 0 means selecting the middle lane, and 1 means selecting the left lane; 1.2, define the reward of the Markov decision process: reward=α*v r +β*lc+γ*c in, α and β represent the reward coefficient of the host vehicle for the new speed state when changing lanes and the penalty parameter for frequent lane changes, respectively. r represents the speed of the main vehicle, lc represents whether a lane change occurs during the second of executing the current action, γ represents the penalty coefficient used to reduce the number of collisions, and c represents the number of collisions during the following process; 1.3, define the state of the Markov decision process: Among them, a r represents the acceleration of the main vehicle, lane_index represents the lane number of the lane the main vehicle is currently in, v fi , h i They represent the speed of the previous vehicle, the acceleration of the previous vehicle, and the headway in the i-th lane, pv i represents the lane advantage function value, N represents the total number of lanes; 1.4, assuming that the main vehicle wants to catch up with the front vehicle as quickly as possible, ideally, the main vehicle goes through three stages in the process of following the front vehicle: accelerating to the maximum speed, maintaining the maximum speed, and reducing to the speed of the front vehicle; the following lane advantage function is introduced: in, represents the speed of the front vehicle in the i-th lane, They represent the duration of the three stages experienced by the main vehicle in the ith lane in the process of following the front vehicle, h represents the headway between the main vehicle and the front vehicle, LCF is the lane change feasibility of the current lane, and the future average speed obtained by the main vehicle in the ith lane based on the principle of catching up with the front vehicle as quickly as possible is used. As the lane advantage function value pv i ; 1.5, calculate the time t for the main vehicle to accelerate from the current speed to the maximum speed a , the time t for the main vehicle to decelerate from the maximum speed to the speed of the front vehicle d , and the distance S traveled by the host vehicle during acceleration and deceleration respectively a , S d : Among them, v max 、a max 、a min Respectively represent the maximum speed, maximum acceleration, and minimum acceleration of the main vehicle, v f Indicates the speed of the vehicle ahead; 1.6, calculate the distance s between the host vehicle and the vehicle in front during acceleration and deceleration error : s error =(t a +t d )*v f +h-(S a +S d ) 1.7, define each duration: When error When it is less than 0, the maximum speed that the vehicle can travel is less than the maximum speed limit of the road; 1.8, if the maximum speed of the vehicle is less than the maximum speed limit of the road, the maximum speed v′ max The calculation of is as follows: Among them, t′1 is the speed of the main vehicle v r Accelerate to the maximum feasible speed v′ max t′3 is the time from the maximum feasible speed v′ max Slow down to the speed of the vehicle in front v f time; Solve t′1, t′3 and substitute into the formula in 1.4 to get the advantage function value of the i-th lane; 1.9, based on the advantage function, set up a dangerous action shielding mechanism. If a lane cannot meet the lane change conditions, the advantage function value is 0. When the model outputs the action of changing to this lane, it means that the action is dangerous. 1.10, set the conditions for determining the feasibility of lane change: v' en > a safe and v' fn > a safe Among them, v′ en Indicates the deceleration that the host vehicle should take, v′ fn Indicates the deceleration that the new following vehicle should take after changing lanes, a safe Indicates the absolute value of the maximum safe deceleration of the vehicle on the road. When the absolute value of the deceleration of the main vehicle and the new following vehicle is greater than the absolute value of the maximum safe deceleration after the lane change action is taken, it means that the lane change process is dangerous. 1.11, set the dangerous action shielding mechanism to minimize the o value corresponding to the dangerous action in the Dueling part structure of the D3QN network; Step 2, constructing a vehicle following behavior model as the lower layer of the intelligent driving decision integration model, using the DDPG algorithm to train the vehicle following behavior model, processing the following decision and outputting continuous vehicle speed control; Step 3: Select a training environment under mixed standard driving conditions, perform coupled training on the upper and lower models, and integrate the upper and lower models.

2. The autonomous driving behavior integrated decision-making method based on deep reinforcement learning according to claim 1, characterized in that: In the step 2, the goal of the vehicle following behavior model is set to be consistent with the benefit goal of the upper layer vehicle lane changing behavior model; For the vehicle following behavior model, define the action: action = {a∈[-3, 3]} Among them, a represents action; Define reward: Among them, α′ represents the speed reward coefficient of the vehicle in the following state, β′ represents the penalty coefficient for collision in the next second, γ′ represents the reward coefficient for improving driving comfort, and v r represents the speed of the main vehicle, c′ represents whether dangerous following behavior occurs within one second of executing the current action, Indicates the square value of acceleration; Define the state: {in r ,in f ,h,a f } Among them, v f represents the speed of the vehicle in front, h represents the distance between the main vehicle and the vehicle in front, a f Indicates the acceleration of the vehicle ahead.

3. The autonomous driving behavior integrated decision-making method based on deep reinforcement learning according to claim 1, characterized in that: In step 3, multiple standard operating conditions are selected for combination, and the combined operating conditions are used as reference operating conditions for vehicle flow. The combined operating conditions are required to cover different vehicle driving scenarios such as acceleration, deceleration and idling of the vehicle, including low-speed operation characteristics on urban roads, high-speed operation characteristics, large instantaneous speed change characteristics, and relatively gentle change characteristics; The leading vehicle in the traffic flow is allowed to accelerate and decelerate according to the combined operating conditions, and the remaining vehicles are allowed to drive freely according to the IDM model and the MOBIL model. When the main vehicle continues to overtake and becomes the leader of the traffic flow, the training object is replaced with the last vehicle in the traffic flow, and this process is repeated to obtain more training samples.

4. The autonomous driving behavior integrated decision-making method based on deep reinforcement learning according to claim 1, characterized in that: In step 3, the intelligent driving decision-making integrated model is trained by coupling training. During the coupling training, the upper and lower layers of the intelligent driving decision-making integrated model share the speed reward and collision penalty of the intelligent agent. The lower layer vehicle following behavior model adjusts the speed of the intelligent agent according to the lane change selection of the upper layer, and finally realizes the integration of the upper layer and the lower layer models.