A self-driving strategy learning method based on deep reinforcement learning

Abstract

The present invention provides an autonomous driving strategy learning method based on deep reinforcement learning, which belongs to the field of reinforcement learning and autonomous driving, and includes the following steps: 1) building an autonomous driving numerical simulation environment based on the Gym library that can interact with the reinforcement learning algorithm; 2) designing a reward function for the autonomous driving strategy learning task, the reward function including a following reward, a lane change reward, a safety reward, and a lane keeping reward; 3) designing a state space, the state space of the reinforcement learning algorithm including information about the main vehicle and the four vehicles surrounding the main vehicle; 4) designing an action space including one discrete action and two continuous actions; 5) adjusting the network structure and hyperparameters of the P-DQN algorithm to train the algorithm until convergence. Compared with the prior art, the present invention uses the P-DQN algorithm with a hybrid action space to learn autonomous driving strategies, without simplifying the original action space. Therefore, the P-DQN algorithm has better stability and higher learning efficiency.

Description

Technical Field

[0001] The present invention belongs to the field of reinforcement learning and autonomous driving technology, and in particular relates to an autonomous driving strategy learning method based on deep reinforcement learning. Background Art

[0002] In recent decades, autonomous driving technology has garnered widespread attention and intensive research. This technology has the potential to improve road safety, resolve traffic congestion, and enhance fuel efficiency. Currently, various perception tasks in autonomous driving systems rely on deep learning techniques to achieve high accuracy. However, the decision-making and control functions of autonomous driving are not well suited to deep learning. The decision-making and control of autonomous vehicles is essentially a sequential decision-making problem. Autonomous vehicles need to continuously learn about their ever-changing environment and make a decision at each step, making reinforcement learning a suitable solution. Reinforcement learning explores the question of how an intelligent agent can maximize its reward in complex and uncertain environments. Reinforcement learning has been proven to be effective in handling sequential decision-making problems with partial observability, long-term planning, and high dimensionality.

[0003] Existing reinforcement learning research mostly models driving behaviors such as lane changing and car following separately. However, in real-world driving scenarios, drivers do not clearly distinguish between lane changing and car following. Drivers often adjust one dimension of their driving behavior to better achieve another. Simultaneously modeling both car following and lane changing for intelligent vehicles is theoretically possible with both traditional and reinforcement learning methods. However, both approaches currently have significant limitations, making it difficult to learn effective driving strategies in practice.

[0004] (1) Traditional methods, such as the MPC method, can model the entire process of intelligent vehicles driving on the road. However, such traditional methods have a major limitation. In scenarios such as lane change decisions, lane change trajectory planning, and car-following, many different constraints need to be set, many of which are nonlinear and non-convex, making the optimization problem difficult to solve. In addition, the clear constraint setting also means that the model can only be applied to simple and stable scenarios. Such traditional methods are difficult to generalize across different scenarios.

[0005] (2) Classic reinforcement learning models can also solve this problem. For example, the DQN algorithm can convert multiple output variables into discrete actions. The DQN algorithm only needs to select and execute a certain action pair, and each action pair includes the values ​​of the discrete actions that have been determined. This obviously makes the action design very rough, and it is difficult to train to achieve ideal results in some complex situations. Applying DDPG to solve this problem directly ignores the higher-level lane change decision action and only outputs the lateral and longitudinal acceleration. Summary of the Invention

[0006] The present invention aims to provide a method for learning an autonomous driving strategy based on deep reinforcement learning, which is characterized by comprising the following steps:

[0007] S1: Building a numerical simulation environment for autonomous driving based on the Gym library to interact with reinforcement learning algorithms;

[0008] S2: Based on the autonomous driving numerical simulation environment built in S1, design the reward function for the autonomous driving strategy learning task.

[0009] S3: Design state space;

[0010] S4: Design action space;

[0011] S5: Adjust the network structure and hyperparameters of the P-DQN algorithm, and then train the algorithm until convergence.

[0012] Furthermore, in S1, the autonomous driving numerical simulation environment is used to simultaneously train the reinforcement algorithm to learn vehicle following strategy, lane keeping strategy, safety strategy, and free lane change strategy. Specifically, the following steps are included:

[0013] S11: Six numerical simulation experiments were built based on the Gym library. These experiments included car-following, lane-keeping, safety maintenance, and lane-changing scenarios.

[0014] S22: According to the tasks of each scenario, set the road and vehicle initialization states of the scenario, and set the vehicle state transition equation;

[0015] S13: Register the prepared Gym environment and call the environment.

[0016] Furthermore, in S2, the reward function includes a following reward, a lane-changing reward, a safety reward, and a lane-keeping reward. The following reward is specifically: the main vehicle and the preceding vehicle maintain a constant headway; the lane-changing reward is specifically: the lateral speed of the main vehicle is the same as the lane-changing direction; the safety reward is specifically: avoiding a collision between the main vehicle and surrounding vehicles and the main vehicle does not drive out of the road boundary line; the lane-keeping reward is specifically: the main vehicle drives on the center line of the lane when not changing lanes.

[0017] Furthermore, the calculation formula of the following reward is expressed as:

[0018] R following =max(-2,9-(v expected -v long ) 2 )

[0019] In the calculation formula of the following reward, v expected is the desired longitudinal velocity of the vehicle; vlong is the true longitudinal velocity of the vehicle; in order to avoid all rewards being negative, a constant term is added to the reward;

[0020] The calculation formula of lane keeping reward is expressed as:

[0021] R lanekeeping =3-{x-[(ni-1)*3.75+1.875]} 2

[0022] In the calculation formula for lane keeping reward, ni is the lane number of the main vehicle, x is the lateral position of the main vehicle, and 3.75 represents the lane width;

[0023] In the safety reward, the calculation formula for avoiding collisions between the main vehicle and surrounding vehicles is expressed as:

[0024]

[0025] In the calculation formula for avoiding collision between the main vehicle and surrounding vehicles, lo is the longitudinal distance between the main vehicle and surrounding vehicles; la is the lateral distance between the main vehicle and surrounding vehicles; 14 is the longitudinal safety distance; 2.8 is the lateral safety distance;

[0026] In the safety reward, the calculation formula for preventing the main vehicle from driving out of the road boundary is expressed as:

[0027]

[0028] In the calculation formula to prevent the main vehicle from driving out of the road boundary line, dl is the distance between the main vehicle and the left boundary line of the road; dr is the distance between the main vehicle and the right boundary line of the road; 1.5 is the safe distance between the main vehicle and the road boundary line;

[0029] The pseudo code for the free lane change reward is as follows:

[0030]

[0031]

[0032] Furthermore, in S3, the state space includes information about the main vehicle and the four vehicles around the main vehicle, wherein the information about each vehicle includes the vehicle's lateral and longitudinal position, lateral and longitudinal speed, lateral and longitudinal acceleration, vehicle number, and lane number.

[0033] Furthermore, S3 is specifically as follows: there is a set S, which contains information of the four vehicles around the main vehicle and the information of the main vehicle, and the information of the surrounding vehicles is arranged in order. The set S is expressed as: S = [OV1, OV2, OV3, OV4, EGO];

[0034] The information of surrounding vehicles is as follows:

[0035] The information of the main vehicle is expressed as: EGO = [x, y, al a ,vl a ,al o ,vl o ,n vehicle ,n lane ];

[0036] Extract the input information of the reinforcement learning algorithm from the set S I = [D1, D2, D3, D4, L]; D i Indicates the EGO of the main vehicle and the OV of the surrounding vehicles i Relative information The calculation method of each information in the relative information Di is:

[0037] la i =xx i

[0038] lo i =yy i

[0039]

[0040]

[0041]

[0042] The distance between the main vehicle and the lane boundary line and the ideal lane number are expressed as: L = [dl, dr, di, ni]; where dl is the distance between the main vehicle and the left boundary line of the road; dr is the distance between the main vehicle and the right boundary line of the road; di is the distance between the main vehicle and the center line of the ideal lane; and n is the number of the ideal lane.

[0043] Furthermore, in S4, the action space is a mixed action space, including one discrete action and two continuous actions. The discrete action is the lane change decision action, specifically: whether to change lanes; the continuous action is the lateral and longitudinal acceleration of the main vehicle.

[0044] Furthermore, S4 is specifically:

[0045] Let the discrete action be k, then the decision of whether to change lanes is k∈[0,1,2], where k=0 means changing lanes to the left; k=1 means staying in the lane; and k=2 means changing lanes to the right.

[0046] Set the continuous action to longitudinal acceleration and lateral acceleration The longitudinal direction specifically refers to: along the direction of the road; the transverse direction specifically refers to: perpendicular to the direction of the road.

[0047] Furthermore, in S5, by adding a convolutional neural network layer to both the actor and commentator networks of P-DQN, the network structure and hyperparameters of the P-DQN algorithm were adjusted; by adjusting the hyperparameters of the P-DQN algorithm and the coefficients of various rewards, the algorithm converged.

[0048] Furthermore, in S5, making the algorithm converge specifically includes the following steps:

[0049] S51: Adjust the calculation method and coefficients of the reward function;

[0050] S52: Adjust the hyperparameters of the algorithm;

[0051] In S52, the hyperparameters include the discount factor, the learning rate of the neural network, the soft update parameter, the noise type, the network weight initialization method, the experience replay cache capacity, the data batch size, and the target network update frequency.

[0052] Compared with the prior art, the beneficial effects of the present invention are mainly reflected in:

[0053] 1. This paper establishes a numerical experimental simulation environment based on the Gym platform that can simultaneously model car-following behavior, free lane changes, lane keeping, and crash avoidance. Compared to simulation software such as SUMO, numerical simulation experiments can better demonstrate the details of modeling various microscopic traffic behaviors, helping researchers better understand autonomous driving strategies.

[0054] 2. This paper uses the P-DQN algorithm to simultaneously learn free lane-changing, car-following, lane-keeping, and safety strategies in a single experiment. Previous studies often modeled these strategies using traditional models combined with reinforcement learning. This approach effectively separates the learning of each strategy and ignores the interactions between them.

[0055] 3. The present invention uses the P-DQN algorithm with a hybrid action space to learn the autonomous driving strategy, without simplifying the original action space. Therefore, the P-DQN algorithm has better stability and higher learning efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 This is a flow chart of an autonomous driving strategy learning method based on deep reinforcement learning in the present invention.

[0057] Figure 2 Schematic diagram of the interaction between reinforcement learning and digital simulation in the present invention.

[0058] Figure 3 Schematic diagram of the experimental scene of the present invention.

[0059] Figure 4This is a schematic diagram of the convergence of the reinforcement learning algorithm rewards of the present invention.

[0060] Figure 5 Schematic diagram of the trajectory and speed of the smart car after the algorithm of the present invention converges. DETAILED DESCRIPTION

[0061] The following is a more detailed description of an autonomous driving strategy learning method based on deep reinforcement learning of the present invention in conjunction with a schematic diagram, which shows a preferred embodiment of the present invention. It should be understood that those skilled in the art can modify the present invention described herein while still achieving the beneficial effects of the present invention. Therefore, the following description should be understood as being widely known to those skilled in the art and not as a limitation of the present invention.

[0062] The present invention proposes a reinforcement learning method P-DQN with a hybrid action space to solve the problem of free lane changing for intelligent vehicles. P-DQN can place three variables: lane change decision (discrete), longitudinal acceleration control (continuous), and lateral acceleration control (continuous) into its action space, without reducing continuous actions to discrete actions or ignoring discrete actions. P-DQN can realize free lane change decision-making and lane change trajectory planning for intelligent vehicles, and can also achieve stable following and lane keeping behavior between the intelligent vehicle and the preceding vehicle before and after the lane change, without the need to model each behavior separately.

[0063] The present invention includes Figure 1 The five steps shown are further explained in conjunction with the embodiments and diagrams, as follows:

[0064] Step 1: Build an autonomous driving digital simulation environment based on the Gym library that can interact with the reinforcement learning algorithm. The autonomous driving digital simulation environment is used to simultaneously train the reinforcement algorithm to learn vehicle following strategy, lane keeping strategy, safety strategy, and free lane change strategy.

[0065] Step 1.1: Set the road information and all vehicle information. The parameters of the road and environment are shown in Table 1:

[0066]

[0067] Table 1 Road and vehicle parameters

[0068] Step 1.2: Set up the vehicle's state transition equations as follows:

[0069]

[0070]

[0071]

[0072]

[0073]

[0074]

[0075]

[0076]

[0077]

[0078]

[0079] Step 2: Set the reward function for the autonomous driving policy learning task, which includes the following reward, lane change reward, safety reward, and lane keeping reward.

[0080] Step 2.1: The calculation method of the following reward is:

[0081] R following =max(-2,9-(v expected -v long ) 2 )

[0082] Among them, v expected represents the desired longitudinal velocity of the vehicle, v long Represents the true longitudinal speed of the vehicle. In order to avoid all rewards being negative, a constant term is added to the reward;

[0083] Step 2.2: The calculation method for avoiding collisions with surrounding vehicles in the safety bonus is:

[0084]

[0085] Wherein, lo represents the longitudinal distance between the main vehicle and surrounding vehicles, la represents the lateral distance between the main vehicle and surrounding vehicles, 14 represents the longitudinal safety distance, and 2.8m represents the lateral safety distance;

[0086] Step 2.3: The calculation method for avoiding driving beyond the road boundary in the safety reward is:

[0087]

[0088] Where dl represents the distance between the main vehicle and the left boundary line of the road, dr represents the distance between the main vehicle and the right boundary line of the road, and 1.5m represents the safe distance between the main vehicle and the road boundary line;

[0089] Step 2.4: The lane keeping reward is calculated as:

[0090] Rlanekeeping =3-{x-[(ni-1)*3.75+1.875]} 2

[0091] Among them, ni is the lane number of the main vehicle, x is the lateral position of the main vehicle, and 3.75m represents the lane width.

[0092] Step 2.5: Lane change reward R efficiency The calculation method is shown in Table 2:

[0093]

[0094]

[0095] Table 2 Free lane change reward pseudo code

[0096] Step 3: Set the input information of P-DQN;

[0097] Step 3.1: Extract the input information of the reinforcement learning algorithm from the set S I = [D1, D2, D3, D4, L], D i Indicates the EGO of the main vehicle and the OV of the surrounding vehicles i Relative information The calculation method of each information is:

[0098] la i =xx i

[0099] lo i =yy i

[0100]

[0101]

[0102]

[0103] Step 3.2: L = [dl, dr, di, ni] represents the distance between the host vehicle and the lane boundary and the ideal lane number; where dl represents the distance between the host vehicle and the left side of the road boundary, dr represents the distance between the host vehicle and the right side of the road boundary, di represents the distance between the host vehicle and the centerline of the ideal lane, and ni represents the number of the ideal lane.

[0104] Step 4: Set the action space of P-DQN. The action information of P-DQN is shown in Table 3:

[0105]

[0106] Table 3 P-DQN algorithm action space design

[0107] Step 5: Set the hyperparameters of P-DQN and the coefficients of each reward;

[0108] Step 5.1: The hyperparameters of P-DQN are shown in Table 4:

[0109]

[0110] Table 4 Hyperparameters of P-DQN

[0111] Step 5.2: The coefficients for each reward are set as shown in Table 5:

[0112]

[0113] Table 5 P-DQN algorithm reward coefficient settings

[0114] Step 5.3: Train the P-DQN algorithm until convergence. The reward convergence of the P-DQN algorithm is as follows: Figure 4 As shown in the figure, P-DQN reaches convergence after 1000 rounds of training, and its convergence speed is faster than that of the traditional DDPG model. After P-DQN converges, the trajectory and speed of the smart car change as shown in the figure. Figure 5 As shown in the figure, the intelligent car first changes lanes to lane 2, then changes lanes to lane 1, and then maintains lane 1. P-DQN simultaneously completes the learning of tasks such as following, keeping lanes, maintaining safety, and free lane changes.

[0115] The above description is merely a preferred embodiment of the present invention and does not limit the present invention in any way. Any person skilled in the art who, without departing from the scope of the present invention, makes any equivalent substitution, modification, or other changes to the technical solution and technical content disclosed in the present invention shall be deemed to be within the scope of the present invention and still fall within the scope of protection of the present invention.

Claims

1. A method for learning autonomous driving strategies based on deep reinforcement learning, characterized in that: The steps include: S1: Building a numerical simulation environment for autonomous driving based on the Gym library to interact with reinforcement learning algorithms; S2: Based on the autonomous driving numerical simulation environment built in S1, design the reward function for the autonomous driving strategy learning task. S3: Design state space; S4: Design action space; S5: Adjust the network structure and hyperparameters of the P-DQN algorithm, and then train the algorithm until convergence; In S2, the reward function includes a following reward, a lane-changing reward, a safety reward, and a lane-keeping reward. The following reward is specifically: the host vehicle and the preceding vehicle maintain a constant headway; the lane-changing reward is specifically: the host vehicle's lateral speed is consistent with the lane-changing direction; the safety reward is specifically: avoiding collisions between the host vehicle and surrounding vehicles and the host vehicle not driving beyond the road boundary; the lane-keeping reward is specifically: the host vehicle stays on the center line of the lane when not changing lanes; The calculation formula of the following reward is expressed as: ； In the calculation formula of the following reward, is the desired longitudinal velocity of the vehicle; is the true longitudinal velocity of the vehicle; in order to avoid all rewards being negative, a constant term is added to the reward; The calculation formula of the lane keeping reward is expressed as: ； In the calculation formula of the lane keeping reward, The lane number where the main vehicle is located, is the lateral position of the main vehicle, Indicates lane width; In the safety reward, the calculation formula for avoiding collision between the host vehicle and surrounding vehicles is expressed as: ； In the calculation formula to avoid collision between the main vehicle and surrounding vehicles; The longitudinal distance between the main vehicle and surrounding vehicles; The lateral distance between the main vehicle and surrounding vehicles; is the longitudinal safety distance; is the lateral safety distance; In the safety reward, the calculation formula for preventing the main vehicle from driving out of the road boundary line is expressed as: ； In the calculation formula to prevent the main vehicle from driving out of the road boundary line, The distance between the main vehicle and the left boundary line of the road; The distance between the main vehicle and the right boundary line of the road; The safe distance between the main vehicle and the road boundary line; The pseudo code of the lane change reward is specifically: If the lane where the main vehicle is located = the ideal lane ； ； 0 else: 0 0 If the ideal lane is greater than the host vehicle's lane: if : ； elseif : ； else: ； else: if : ； elseif : ； else: ； In S3, the state space includes information of the main vehicle and information of four vehicles around the main vehicle, wherein the information of each vehicle includes the vehicle's lateral and longitudinal position, lateral and longitudinal speed, lateral and longitudinal acceleration, vehicle number, and lane number; In S4, the action space is a mixed action space, including one discrete action and two continuous actions. The discrete action is a lane change decision action, specifically: whether to change lanes; the continuous action is the lateral and longitudinal accelerations of the main vehicle.

2. The autonomous driving strategy learning method based on deep reinforcement learning according to claim 1, characterized in that: In S1, the autonomous driving numerical simulation environment is used to simultaneously train the reinforcement algorithm to learn vehicle following strategy, lane keeping strategy, safety strategy and free lane change strategy, and specifically includes the following steps: S11: Building six numerical simulation experiments based on the Gym library, including a car-following scenario, a lane-keeping scenario, a safety-maintaining scenario, and a lane-changing scenario; S22: According to the tasks of each scenario, set the road and vehicle initialization states of the scenario, and set the vehicle state transition equation; S13: Register the prepared Gym environment and call the environment.

3. The autonomous driving strategy learning method based on deep reinforcement learning according to claim 1, characterized in that: The S3 is specifically: there is a set , the set Contains the information of the four vehicles around the main vehicle and the main vehicle. The information of the surrounding vehicles is arranged in order. It manifests as: ; The information of surrounding vehicles is as follows: ; The information of the main vehicle is represented as: ; From the collection Extracting input information from reinforcement learning algorithms ; Indicates the main vehicle With surrounding vehicles Relative information ; Relative information The calculation method of each information is as follows: ； ； ； ； ； The distance between the host vehicle and the lane boundary line and the ideal lane number are expressed as: ;in, The distance between the main vehicle and the left boundary line of the road; The distance between the main vehicle and the right boundary line of the road; is the distance between the main vehicle and the center line of the ideal lane; is the number of the ideal lane.

4. The autonomous driving strategy learning method based on deep reinforcement learning according to claim 1, characterized in that: The S4 is specifically: Set the discrete action to , then, decide whether to change lanes ,in, To change lanes to the left; To maintain your lane, Indicates changing lanes to the right; Set the continuous action to longitudinal acceleration and lateral acceleration The longitudinal direction is specifically along the direction of the road; the transverse direction is specifically perpendicular to the direction of the road.

5. The autonomous driving strategy learning method based on deep reinforcement learning according to claim 1, characterized in that: In S5, by adding a convolutional neural network layer to both the actor and commentator networks of the P-DQN algorithm, the network structure and hyperparameters of the P-DQN algorithm are adjusted; and by adjusting the hyperparameters of the P-DQN algorithm and the coefficients of various rewards, the algorithm converges.

6. The autonomous driving strategy learning method based on deep reinforcement learning according to claim 5, characterized in that: In S5, making the algorithm converge specifically includes the following steps: S51: Adjust the calculation method and coefficients of the reward function; S52: Adjust the hyperparameters of the algorithm; In S52, the hyperparameters include a discount factor, a learning rate of a neural network, a soft update parameter, a noise type, a network weight initialization method, an experience replay cache capacity, a data batch size, and a target network update frequency.

Images