Lane keeping method for autonomous driving vehicles based on TD3 algorithm improved by exploration strategy

Abstract

The present invention discloses a lane keeping method for an autonomous driving vehicle based on an improved TD3 algorithm with an exploration strategy, abstracts and refines the decision-making control problem in the autonomous driving intelligent control task, and describes the control of the unmanned vehicle in the task as a model-free Markov decision process; based on the TD3 algorithm, the input state of the autonomous driving vehicle and its sensor information are calculated, specific actions are output, and the vehicle action is controlled in real time; the Ornstein-Uhlenbeck process is used to add noise to the action output by the double-delayed deep deterministic policy gradient algorithm, and the Ornstein-Uhlenbeck process noise is weightedly corrected based on the path tracking method. The present invention combines the path tracking method to perform weighted correction on the action noise in the exploration strategy of the TD3 algorithm, so that the exploration of the unmanned vehicle is biased towards the correct direction, and the proportion of inefficient samples in the experience replay pool is reduced, which ultimately makes the algorithm converge faster and the strategy has better expressiveness.

Description

Technical Field

[0001] The present invention belongs to the field of deep reinforcement learning and autonomous driving, and specifically relates to a lane keeping method for an autonomous driving vehicle based on an improved TD3 algorithm. Background Art

[0002] An autonomous vehicle is an outdoor wheeled mobile robot with autonomous driving capabilities. Most autonomous vehicles use a modular system, and its core functions can be summarized as: perception, planning, decision-making, and vehicle control. The advantage of a modular system is that it decomposes challenging autonomous driving tasks into a set of simpler problems, but this modular design makes the system prone to error propagation. In recent years, end-to-end structures have begun to emerge as an alternative to modular methods. In end-to-end driving, vehicle control actions are directly calculated and output based on sensor input. Compared with traditional modular methods, this method has the advantages of simple ideas and sufficient environmental information perception. It can integrate environmental information and autonomous driving vehicle information to achieve human-like driving behavior. Deep reinforcement learning (DRL) is a main method for achieving end-to-end driving by combining deep learning (DL) and reinforcement learning (RL). RL is a very common framework for solving artificial intelligence problems. DL can not only bring convenience to end-to-end optimization for RL, but also make RL no longer limited to low-dimensional space, greatly expanding the scope of RL application. In practical applications, RL is divided into model-based RL and model-free RL according to whether the unmanned vehicle can predefine the environmental dynamic model through data obtained by interacting with the environment. Model-based RL will learn the optimal strategy based on the dynamic model of the environment. If the model can accurately predict the dynamic changes of the environment, then when a state and an action are input, there is no need to interact with the environment in real time, and the new state and action reward can be predicted directly, which greatly improves the utilization of data. However, when the problem has a complex state space and action space, it is a big challenge to accurately estimate the dynamic model of the environment, and using inaccurate models for learning can easily lead to compound errors. Model-free RL refers to the unmanned vehicle interacting and exploring the environment in real time, learning directly from the acquired experience data, and ultimately maximizing the cumulative benefits or other specific goals.

[0003] This paper mainly studies the lane keeping task in autonomous driving, which is the premise for autonomous vehicles to complete other complex tasks. If it is not done well, safety problems such as collisions may occur. In the past few years, many scholars have conducted in-depth research on this issue. The main algorithm they used is the Deep Deterministic Policy Gradient Algorithm (DDPG). From their experimental results, DDPG can indeed guide unmanned vehicles to complete lane keeping tasks, but when we further reproduced their experiments, we found the following problems:

[0004] (1) The convergence of the DDPG algorithm depends largely on the setting of hyperparameters. Bad hyperparameters can easily lead to non-convergence of the algorithm and poor performance of the algorithm. Specifically, when the driverless vehicle guided by the DPPG algorithm performs the lane keeping task, it sways from side to side, just like drunk driving. Although the task is completed, there are great safety risks.

[0005] (2) DDPG is a deterministic policy algorithm that outputs only a single action at a time. However, the nature of model-free algorithms determines that it needs to learn the optimal policy by exploring the environment. Therefore, we usually use the method of directly adding random noise to the output action to help the unmanned vehicle explore. Commonly used noises include Gaussian noise and OU noise, which are all formed based on the actions output by the algorithm. In the initial stage of training, the network structure in the algorithm is imperfect, the output action is unreliable, and the initial weight of the noise is large. In this case, the unmanned vehicle will do a lot of invalid exploration, which greatly reduces the training efficiency. Summary of the invention

[0006] Purpose of the invention: The present invention proposes a lane keeping method for autonomous driving vehicles based on the TD3 algorithm improved by the exploration strategy, which improves the quality of training samples obtained during the interaction between the unmanned vehicle and the environment, enhances the performance of the algorithm, and enables the exploration of the autonomous driving vehicle to tend in the right direction.

[0007] Technical solution: The present invention provides a lane keeping method for an autonomous driving vehicle based on a TD3 algorithm improved by an exploration strategy, comprising the following steps:

[0008] (1) Abstract and refine the decision-making and control problems in the autonomous driving intelligent control task, and combine it with the deep reinforcement learning algorithm to describe the autonomous driving vehicle control in the task as a model-free Markov decision process;

[0009] (2) Based on the double-delayed deep deterministic policy gradient algorithm, it can calculate the input autonomous driving vehicle state and its sensor information, output specific actions, and control the vehicle's behavior in real time;

[0010] (3) During the training phase of the double-delayed deep deterministic policy gradient algorithm, in order to fully explore the state space, the Ornstein-Uhlenbeck process is used to add noise to the actions output by the double-delayed deep deterministic policy gradient algorithm, and the Ornstein-Uhlenbeck process noise is weightedly corrected based on the path tracing method to reduce the invalid exploration of the autonomous driving vehicle during the training process;

[0011] (4) A large number of experiments were conducted on the TORCS simulation platform. The autonomous driving vehicle was guided by a double-delayed deep deterministic policy gradient algorithm improved by the exploration strategy. A large amount of training data was collected by interacting with the environment, and the algorithm was updated using this data for learning, and finally converged to the optimal strategy.

[0012] Furthermore, the implementation process of step (2) is as follows:

[0013] There are six networks in the TD3 algorithm, namely Actor network, Critic1 network_1, Cirtic2 network_2, Actor target network, Critic target network_1 and Critic2 target network_2; among them, the Actor network is a policy network, which receives [the state of the autonomous driving vehicle] as input and outputs [the control action for the vehicle]; the Critic network is an evaluation network, which takes [the vehicle state, the action generated by the Actor network] as input and outputs [the value of the action generated by the Actor network], which is used to evaluate the Actor network.

[0014] Furthermore, the path tracking method in step (3) is:

[0015] For the lane keeping task in autonomous driving, the TD3 algorithm receives the state of the unmanned vehicle as input and outputs three actions: steering, acceleration, and braking. In the algorithm training phase, in order to fully explore the state space, the OU process is used to add noise to the actions output by the TD3 algorithm. The exploration strategy of the autonomous driving vehicle is expressed as:

[0016] a t =μ(s t |θ)+N t

[0017] Among them, μ(s t |θ) is the policy action output by the TD3 algorithm, which is a three-dimensional vector: [steer, accel, brake], N t The noise generated by the OU process is also a three-dimensional vector: [steer_noise, accel_noise, brake_noise];

[0018] The motion noise generated by the OU process is weightedly corrected based on the path tracking method, and the angle between the vehicle direction and the road axis is minimized while considering the vehicle speed. And shorten the distance trackPos between the center of mass of the vehicle and the center line of the road. The steering control action given by the path tracking method is:

[0019]

[0020] Among them, η1, η2, η3 are proportional coefficients.

[0021] Furthermore, the weighted correction process of the Ornstein-Uhlenbeck process noise based on the path tracing method in step (3) is as follows:

[0022] The steering noise generated by the OU process is weighted and corrected to reduce the invalid exploration of the unmanned vehicle:

[0023] steer_noise=(1-ε)*steer_noise_OU+ε*steer_action_PT,

[0024] The initial value of ε is 0.5, and it gradually decreases during the training process. When the number of training steps reaches 100,000, it decreases to 0. After weighted correction of the steering noise, the steering action performed by the unmanned vehicle is:

[0025] steer=original_steer+steer_noise

[0026] Among them, original_steer is the original steering action output by the Actor network.

[0027] Furthermore, the step (4) comprises the following steps:

[0028] 1) First initialize the Actor network π θ , And initialize the experience replay pool D; start the kth round of simulation (1≤k≤M), where M is the maximum number of simulation rounds set;

[0029] 2) Initialization state s t ;

[0030] 3) Start the tth time step (1≤t≤T) in the kth round of simulation, where T is the maximum number of time steps set for a round of simulation;

[0031] 4) Input the state of the autonomous driving vehicle into the Actor network and the path tracking algorithm to obtain the original action original_a output by the TD3 algorithm t And the steering action steering_action_PT output by the path tracking algorithm;

[0032] 5) Change the action original_a t Input OU process, output OU noise N t ;

[0033] 6) Set steering_action_PT and N tCombined to generate weighted modified noise N' t ;

[0034] 7) Generate the final action a t =μ θ (s t )+N' t ;

[0035] 8) Execute the action a t , get reward r t , and enter the next state s' t ;

[0036] 9)Tuple(s) t ,a t ,r t ,s' t ) is stored in the experience replay pool D;

[0037] 10) If there is enough data in the experience replay pool D, sample N data (s, a, r, s');

[0038] 11) Use the Actor target network to generate actions in state s': in is the action output by the Actor target network, ε~Normal(0,σ) is added to improve the robustness of the algorithm The noise based on normal distribution is σ, σ is the covariance of normal distribution, and clip(ε,-C,C) is used to clip the noise. The noise range after clipping by the clip function is between (-C,C);

[0039] 12) Calculate the update target: where γ is the discount factor, is the action value generated by the Critic target network;

[0040] 13) Update the parameters of the Critic network:

[0041] 14) Update the parameters of the Actor network every two time steps using the gradient ascent method: Update the parameters of the three target networks simultaneously: Where τ is the parameter for sliding smoothing update;

[0042] 15) After a time step is completed, return to step (3) to proceed to the next time step;

[0043] 16) After one round of simulation is completed, return to step (1) to perform the next round of simulation until the number of simulation rounds k reaches the set upper limit M.

[0044] Beneficial effects: Compared with the prior art, the beneficial effects of the present invention are: the improved TD3 algorithm proposed in the present invention reduces the sensitivity of the algorithm to hyperparameters, makes the algorithm training process more stable, and the algorithm performance is better; makes the exploration in the unmanned vehicle training process tend to the correct direction, which not only improves the training efficiency, but also improves the quality of training samples obtained during the interaction between the unmanned vehicle and the environment, thereby further improving the performance of the algorithm; makes the exploration of the autonomous driving vehicle tend to the correct direction. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 Schematic diagram of the TD3 algorithm framework for the improved exploration strategy proposed in the present invention;

[0046] Figure 2 It is a schematic diagram of TD3 algorithm;

[0047] Figure 3 It is a schematic diagram of the traditional path tracing method;

[0048] Figure 4 Schematic diagram of weighted correction of motion noise. DETAILED DESCRIPTION

[0049] The present invention is further described in detail below with reference to the accompanying drawings.

[0050] The present invention proposes a lane keeping method for an autonomous driving vehicle based on a TD3 algorithm improved by an exploration strategy, which specifically includes the following steps:

[0051] Step 1: Abstract and refine the decision-making and control problems in the autonomous driving intelligent control task, and combine it with the deep reinforcement learning algorithm to describe the autonomous driving vehicle control in the task as a model-free Markov decision process.

[0052] Step 2: Based on the double-delay deep deterministic policy gradient algorithm (TD3), the input autonomous driving vehicle state and its sensor information are calculated, specific actions are output, and vehicle behavior is controlled in real time.

[0053] There are six networks in the TD3 algorithm, namely Actor network, Critic1 network_1, Cirtic2 network_2, Actor target network, Critic target network_1 and Critic2 target network_2. Figure 2As shown in the figure, the Actor network is a policy network that receives [the state of the autonomous vehicle] as input and outputs [the control action for the vehicle]; the Critic network is an evaluation network that takes [the state of the vehicle, the action generated by the Actor network] as input and outputs [the value of the action generated by the Actor network] to evaluate the Actor network. In particular, using two Critic networks can effectively avoid overestimating the value of the action; and the three Target networks are for the purpose of jointly outputting the update target of the algorithm. The Actor network parameters are updated every two steps, which reduces the variance of the action value function.

[0054] Step 3: During the training phase of the double-delayed deep deterministic policy gradient algorithm, in order to fully explore the state space, the Ornstein-Uhlenbeck process is used to add noise to the actions output by the double-delayed deep deterministic policy gradient algorithm, and the Ornstein-Uhlenbeck process (OU process) noise is weightedly corrected based on the path tracing method to reduce the invalid exploration of the autonomous driving vehicle during the training process.

[0055] For the lane keeping task in autonomous driving, the TD3 algorithm receives the state of the driverless vehicle as input and outputs three actions: steering, acceleration, and braking. In the algorithm training phase, in order to fully explore the state space, the OU process is used to add noise to the actions output by the TD3 algorithm. The exploration strategy of the autonomous driving vehicle can be expressed as:

[0056] a t =μ(s t |θ)+N t

[0057] Among them, μ(s t |θ) is the policy action output by the TD3 algorithm, which is a three-dimensional vector: [steer, accel, brake], N t The noise generated by the OU process is also a three-dimensional vector: [steer_noise, accel_noise, brake_noise].

[0058] However, the initial weight value of the noise is large and highly random, which makes the unmanned vehicle often make some unreasonable behaviors in the early stage of training. The present invention uses the path tracking method to perform weighted correction on the action noise generated by the OU process to reduce the invalid exploration of the autonomous driving vehicle during the training process: The path tracking method is as follows Figure 3 As shown, the goal is to minimize the angle between the vehicle direction and the road axis while taking into account the vehicle speed. And shorten the distance trackPos between the center of mass of the vehicle and the center line of the road. The steering control action given by the path tracking method is:

[0059]

[0060] Among them, η1, η2, η3 are proportional coefficients, which are determined as follows: Assume that the vehicle is at the edge of the road, the speed direction is perpendicular to the road axis, and the speed is 100km / h. Under this extreme condition, the magnitude of the steering control action is 0.3, and η1*trackPos and The values ​​are equal.

[0061] like Figure 4 As shown in Figure 1, the weighted corrected noise uses the steering action generated by the path tracking method to perform weighted correction on the steering noise generated by the OU process (the acceleration noise and braking noise remain unchanged) to reduce the invalid exploration of the unmanned vehicle:

[0062] steer_noise=(1-ε)*steer_noise_OU+ε*steer_action_PT

[0063] The initial value of ε is 0.5, and it gradually decreases during the training process. When the number of training steps reaches 100,000, it decreases to 0. After weighted correction of the steering noise, the steering action performed by the unmanned vehicle is:

[0064] steer=original_steer+steer_noise

[0065] Among them, original_steer is the original steering action output by the Actor network.

[0066] Step 4: Conduct a large number of experiments on the TORCS simulation platform. The autonomous driving vehicle is guided by a double-delayed deep deterministic policy gradient algorithm improved by the exploration strategy. A large amount of training data is collected by interacting with the environment, and the data is used for learning, the algorithm is updated, and finally converges to the optimal strategy.

[0067] like Figure 1 The TD3 algorithm with improved exploration strategy is shown ( E xploration S strategy I The basic framework, algorithm flow and specific implementation steps of the improved TD3 are described as follows:

[0068] First initialize the Actor network π θ , And initialize the experience replay pool D.

[0069] Step (1): Start the kth round of simulation (1≤k≤M), where M is the maximum number of simulation rounds set;

[0070] Step (2): Initialize state s t ;

[0071] Step (3): start the tth time step (1≤t≤T) in the kth round of simulation, where T is the maximum number of time steps set for a round of simulation;

[0072] Step (4): Input the state of the autonomous driving vehicle into the Actor network and path tracing algorithm to obtain the original action original_a output by the TD3 algorithm t And the steering action steering_action_PT output by the path tracking algorithm;

[0073] Step (5): Change action original_a t Input OU process, output OU noise N t ;

[0074] Step (6): Set steering_action_PT and N t Combined to generate weighted modified noise N' t ;

[0075] Step (7): Generate the final action a t =μ θ (s t )+N' t ;

[0076] Step (8): Execute the action a t , get reward r t , and enter the next state s' t ;

[0077] Step (9): Substitute the tuple (s t ,a t ,r t ,s' t ) is stored in the experience replay pool D;

[0078] Step (10): If there are enough data in the experience replay pool D, sample N data (s, a, r, s');

[0079] Step (11): Use the Actor target network to generate actions in state s': in is the action output by the Actor target network, ε~Normal(0,σ) is added to improve the robustness of the algorithm The noise based on normal distribution is σ, σ is the covariance of normal distribution, and clip(ε,-C,C) is used to clip the noise. The noise range after clipping by the clip function is between (-C,C);

[0080] Step (12): Calculate the updated target: where γ is the discount factor, is the action value generated by the Critic target network;

[0081] Step (13): Update the parameters of the Critic network:

[0082] Step (14): Update the parameters of the Actor network every two time steps using the gradient ascent method: Update the parameters of the three target networks simultaneously: Where τ is the parameter for sliding smoothing update;

[0083] Step (15): After one time step is completed, return to step (3) to proceed to the next time step;

[0084] Step (16): After one round of simulation is completed, return to step (1) to perform the next round of simulation until the number of simulation rounds k reaches the set upper limit M.

[0085] In order to solve the problem that the DDPG algorithm is sensitive to hyperparameters, resulting in poor performance of the final trained strategy, the present invention improves the DDPG algorithm into a double-delayed deterministic policy gradient algorithm. TD3 adopts the value network structure of Double Q-learning, limits overestimation by selecting the minimum value between a pair of value networks, and further improves the algorithm performance by delaying the update of the policy network to reduce the error of each update. In order to solve the problem that unmanned vehicles perform a large amount of invalid exploration in the early stage of training, resulting in low training efficiency, we propose the TD3 algorithm with improved exploration strategy. In this algorithm, we combine the traditional path tracking method to perform weighted correction on the action noise in the exploration strategy, so that the exploration of the unmanned vehicle tends to the right direction.

Claims

1. A lane keeping method for an autonomous driving vehicle based on the TD3 algorithm improved by the exploration strategy, characterized in that: The following steps are involved: (1) Abstract and refine the decision-making and control problems in the autonomous driving intelligent control task, and combine it with the deep reinforcement learning algorithm to describe the autonomous driving vehicle control in the task as a model-free Markov decision process; (2) Based on the double-delayed deep deterministic policy gradient algorithm, it can calculate the input autonomous driving vehicle state and its sensor information, output specific actions, and control the vehicle's behavior in real time; (3) During the training phase of the double-delayed deep deterministic policy gradient algorithm, in order to fully explore the state space, the Ornstein-Uhlenbeck process is used to add noise to the actions output by the double-delayed deep deterministic policy gradient algorithm, and the Ornstein-Uhlenbeck process noise is weightedly corrected based on the path tracing method to reduce the invalid exploration of the autonomous driving vehicle during the training process; (4) Conduct a large number of experiments on the TORCS simulation platform. The autonomous driving vehicle is guided by a double-delayed deep deterministic policy gradient algorithm that improves the exploration strategy. It collects a large amount of training data by interacting with the environment, and uses this data to learn and update the algorithm, eventually converging to the optimal strategy. The path tracking method in step (3) is: For the lane keeping task in autonomous driving, the TD3 algorithm receives the state of the unmanned vehicle as input and outputs three actions: steering, acceleration, and braking. In the algorithm training phase, in order to fully explore the state space, the OU process is used to add noise to the actions output by the TD3 algorithm. The exploration strategy of the autonomous driving vehicle is expressed as: a t =μ(s t |θ)+N t Among them, μ(s t |θ) is the policy action output by the TD3 algorithm, which is a three-dimensional vector: [steer, accel, brake], N t is the noise generated by the OU process, which is also a three-dimensional vector: [steer_noise, accel_noise,brake_noise]; The motion noise generated by the OU process is weightedly corrected based on the path tracking method, and the angle between the vehicle direction and the road axis is minimized while considering the vehicle speed. And shorten the distance trackPos between the center of mass of the vehicle and the center line of the road. The steering control action given by the path tracking method is: Among them, η1, η2, η3 are proportional coefficients; The step (4) comprises the following steps: 1) First initialize the Actor network π θ , Critic network_1 Critic Network_2 Actor Target Network Critic target network_1 Critic target network_2 And initialize the experience replay pool D; start the kth round of simulation, where 1≤k≤M, M is the maximum number of simulation rounds set; 2) Initialization state s t ; 3) Start the tth time step in the kth round of simulation, where 1≤t≤T, and T is the maximum number of time steps set for a round of simulation; 4) Input the state of the autonomous driving vehicle into the Actor network and the path tracking algorithm to obtain the original action original_a output by the TD3 algorithm t And the steering action steering_action_PT output by the path tracking algorithm; 5) Change the action original_a t Input OU process, output OU noise N t ; 6) Set steering_action_PT and N t Combined to generate weighted modified noise N' t ; 7) Generate the final action a t =μ θ (s t )+N' t ; 8) Execute the action a t , get reward r t , and enter the next state s' t ; 9)Tuple(s) t ,a t ,r t ,s' t ) is stored in the experience replay pool D; 10) If there is enough data in the experience replay pool D, sample N data (s, a, r, s'); 11) Use the Actor target network to generate actions in state s': in is the action output by the Actor target network, ε~Normal(0,σ) is added to improve the robustness of the algorithm The noise based on normal distribution is σ, σ is the covariance of normal distribution, and clip(ε,-C,C) is used to clip the noise. The noise range after clipping by the clip function is between (-C,C); 12) Calculate the update target: where γ is the discount factor, is the action value generated by the Critic target network; 13) Update the parameters of the Critic network: 14) Update the parameters of the Actor network every two time steps using the gradient ascent method: Update the parameters of the three target networks at the same time: θ - ←τθ+(1-τ)θ, Where τ is the parameter for sliding smoothing update; 15) After a time step is completed, return to step (3) to proceed to the next time step; 16) After one round of simulation is completed, return to step (1) to perform the next round of simulation until the number of simulation rounds k reaches the set upper limit M.

2. The lane keeping method for an autonomous driving vehicle based on the TD3 algorithm improved by the exploration strategy according to claim 1, characterized in that: The implementation process of step (2) is as follows: There are six networks in the TD3 algorithm, namely Actor network, Critic1 network_1, Cirtic2 network_2, Actor target network, Critic target network_1 and Critic2 target network_2; among them, the Actor network is a policy network, which receives [the state of the autonomous driving vehicle] as input and outputs [the control action for the vehicle]; the Critic network is an evaluation network, which takes [the vehicle state, the action generated by the Actor network] as input and outputs [the value of the action generated by the Actor network], which is used to evaluate the Actor network.

3. The lane keeping method for an autonomous driving vehicle based on the TD3 algorithm improved by the exploration strategy according to claim 1, characterized in that: The weighted correction process of the Ornstein-Uhlenbeck process noise based on the path tracing method in step (3) is as follows: The steering noise generated by the OU process is weighted and corrected to reduce the invalid exploration of the unmanned vehicle: steer_noise=(1-ε)*steer_noise_OU+ε*steer_action_PT, The initial value of ε is 0.5, and it gradually decreases during the training process. When the number of training steps reaches 100,000, it decreases to 0. After weighted correction of the steering noise, the steering action performed by the unmanned vehicle is: steer=original_steer+steer_noise Among them, original_steer is the original steering action output by the Actor network.

Images