An unmanned driving lane-changing decision control method based on a proximal policy optimization algorithm

Abstract

The application discloses an unmanned lane-changing decision control method based on a proximal policy optimization algorithm, and proposes an end-to-end unmanned lane-changing decision control model based on the proximal policy optimization algorithm.The model directly outputs control amounts of a vehicle accelerator, brake and steering after processing of an input state obtained according to vehicle and driving environment related information through a neural network.For the unmanned lane-changing decision control model based on the proximal policy optimization algorithm, firstly, a network structure is designed, including an Actor network, an OldActor network and a Critic network;then, a state space, an action space and a reward function are designed;finally, model training and verification are carried out on a TORCS simulation platform.The PPO algorithm is adopted in the application, which can better solve the problem that a learning rate is difficult to set in deep reinforcement learning, and meanwhile, the convergence speed of model training is improved.

Description

Technical Field

[0001] The invention relates to the fields of autonomous driving and deep reinforcement learning, and in particular to an autonomous driving lane change decision and control method based on deep reinforcement learning. Background Technology

[0002] Autonomous vehicles (AVVs) represent an advanced stage in the development of intelligent vehicles. They leverage their perception, decision-making, and control capabilities to independently perform driving tasks in specific environments, replacing human drivers. Urban road traffic safety, traffic congestion, traffic pollution, and energy consumption are significant challenges facing urban and transportation development worldwide, and the application of autonomous vehicles is a crucial approach to addressing these issues. Lane changing is a fundamental driving behavior for autonomous vehicles, involving the manipulation of the accelerator, brakes, and steering wheel to move from one lane to another.

[0003] Currently, the main methods for implementing lane-changing behavior in autonomous driving include rule-based methods, deep learning-based methods, and deep reinforcement learning-based methods. Rule-based methods can achieve lane-changing behavior in simple, closed environments; however, in complex environments, the rule base is difficult to construct and struggles to cover all scenarios. Deep learning-based methods require a large amount of labeled data as input to ensure the generalization ability of the model training, making training difficult. Deep reinforcement learning methods combine the perception capabilities of deep learning with the decision-making capabilities of reinforcement learning, making them more suitable for solving decision-making problems in high-dimensional state spaces. Currently, deep reinforcement learning methods widely used in autonomous driving research and application are prevalent. Major deep reinforcement learning methods applied in autonomous driving, such as the Deep Q Network (DQN) algorithm and the Deep Deterministic Policy Gradient (DDPG) algorithm, often face the challenge of accurately setting the learning rate during model training. Setting the learning rate too high leads to convergence difficulties; setting it too low results in slow training speed. The PPO algorithm, by utilizing the ratio of the Actor network to the Old Actor network, limits the update magnitude of the Actor network, reduces the sensitivity of the policy gradient to large step sizes, solves the problem of accurately setting the learning rate in algorithms such as DDPG, and accelerates the convergence speed. In response, this invention proposes an autonomous driving lane change decision-making and control method based on a proximal policy optimization algorithm. Summary of the Invention

[0004] In view of this, this invention proposes an autonomous driving lane-changing decision-making and control method based on a proximal policy optimization algorithm to solve the problem of difficulty in determining the learning rate when using common deep reinforcement learning methods such as deep deterministic policy gradient algorithms. Its model structure diagram is shown below. Figure 2 As shown.

[0005] The technical solution adopted in this invention is an autonomous driving lane change decision control method based on a near-end strategy optimization algorithm, comprising the following steps:

[0006] Step 1: Construct an autonomous driving lane change decision-making and control model based on a near-end policy optimization algorithm. The neural network of this model consists of an actor network and a critic network. The actor network includes an Actor network and an Old Actor network, while the critic network includes a Critic network. The Actor network is used to determine the control output when the autonomous vehicle performs a lane change, and the input is the current environmental information s obtained from the sensors. t The output is the control quantity that enables lane-changing driving behavior, i.e., the action space a. t The Old Actor network has the same structure as the Actor network; the Old Actor network consists of neural network parameters from several steps prior to the Actor network; the Critic network is used to score the control parameters for the lane-changing driving behavior decided by the Actor network, with the current environmental information s as input. t Output current environment information s t Below, the score of the control quantity output by the actor network, i.e., action a. t State value V(s) t Complete the construction of the neural network for the near-end strategy optimization algorithm. Establish a data buffer for the autonomous driving lane change decision control model to store data generated during the interaction between the autonomous vehicle and the environment, which will be used for model updates.

[0007] Step 2: Train the autonomous driving lane change decision-making and control model based on the near-end policy optimization algorithm. The autonomous vehicle interacts with its environment to obtain current environmental information. t Action space a t Action space a t The log probability Logprob in environment s t The next execution action space a t The reward value r obtained t Environmental information s at the next moment t+1 The round ends with a "is_terminal" flag. The model is trained using information obtained through the interaction between the autonomous vehicle and its environment. Current environmental information is also included. t A high-dimensional space comprising vehicle turning angle, vehicle speed, and distances from the road edge and centerline; action space a t It includes steering control, throttle control, and braking control; the reward function is designed to take into account both the speed of the autonomous vehicle and the target driving state (the final driving state of the autonomous vehicle when it completes a lane change).

[0008] Step 3: Test the autonomous driving lane change decision and control model based on the near-end policy optimization algorithm. Use the trained autonomous driving lane change decision and control model based on the near-end policy optimization algorithm to perform lane changes in a new environment to verify the reliability of the model.

[0009] Furthermore, in step 2, the process of interacting with the environment specifically involves:

[0010] Step 2.1: Initialize the network parameters of each neural network in the autonomous driving lane change decision control model based on the near-end policy optimization algorithm and start interacting with the environment where the autonomous vehicle is located.

[0011] Step 2.2: The autonomous vehicle interacts with the environment to obtain current environmental information. t s current environmental information t The OldActor network is input to determine the control inputs required for the autonomous vehicle to perform lane-changing driving actions, i.e., action a. mean .

[0012] Step 2.3, according to action a mean Create a covariance matrix with standard deviation along the diagonal, and generate a multivariate normal distribution. Sample the action 'a' from the multivariate normal distribution. t And calculate action a t Log probability Logprob.

[0013] Step 2.4, the unmanned vehicle performs action a. t Based on the reward function, environmental information s is obtained. t Next, execute action a t Reward value r t And according to the state transition rules, it transitions to the state s of the next time step. t+1 .

[0014] Step 2.5: Repeat steps 2.2 to 2.4 several times, and place the tuple information obtained after each execution of steps 2.2 to 2.4 into the data buffer. Each step will generate an array containing the current environment information s. t Action a t Action a t The log probability Logprob in environment s t Next, execute action a t The reward value r obtained t Environmental information s at the next moment t+1 The round end flag is_terminal.

[0015] Step 2.6: Based on all the data obtained from the data buffer, calculate the cumulative reward value R for each data point thereafter. tAnd calculate the cumulative reward value R for each data point thereafter. t After performing mean removal and normalization, we obtain

[0016] R t =r t +γr t+1 +γ 2 r t+2 +…+γ T-t+1 r T-1 (1)

[0017] In the total duration T, r t r represents the reward value obtained at time t. t+1 r represents the reward value obtained at time t+1. T-1 This represents the reward value obtained at time T-1. γ is the discount factor, which takes a value greater than 0 and less than 1. As can be seen from formula (1), the cumulative reward value after time t is obtained by multiplying the reward value from time t to the total duration T-1 by the discount factor raised to the power of 0 to T-t+1, respectively. That is, the further away from time t is, the smaller the impact of the time on the cumulative reward value.

[0018] Step 2.7: Set the current environment information s for each piece of data in the data buffer. t Action space a t Inputting the Actor network yields the action space a. t The estimated log probability Logprob_eval.

[0019] Step 2.8: Set the current environment information s for each piece of data in the data buffer. t Inputting the Critic network yields the state value V. θ (s t ).

[0020] Step 2.9: Calculate the probability ratio r of the output actions of the Actor network and the OldActor network based on the estimated log probability Logprob_eval calculated in Step 2.7 and the log probability Logprob stored in the data buffer. t (θ).

[0021]

[0022] The action probability is the output of the Actor network. The action probabilities output by the Old Actor network.

[0023] Step 2.10, based on the results obtained in Step 2.6 Compared with the state value V(s) in step 2.8t Calculate the dominance value

[0024]

[0025] Step 2.11: Calculate the loss value of the Actor network according to formula (4) and update the Actor network.

[0026]

[0027] Step 2.12: Calculate the loss value of the Critic network according to formula (5) and update the Critic network.

[0028]

[0029] Step 2.13: Repeat steps 2.2 to 2.12 until the Actor network converges, indicating that the training of the autonomous driving lane change decision control model based on the proximal policy optimization algorithm is complete.

[0030] Furthermore, the update frequency of the Actor network and Critic network is once for each interaction between the autonomous vehicle and the environment, and once for each training step.

[0031] Furthermore, the autonomous vehicle obtains relevant information about the vehicle and driving environment (such as vehicle turning angle, vehicle speed, vehicle distance from the road edge and center line, etc.) through various sensors as input states. After processing by the model, it directly outputs the control quantities of the vehicle's throttle, brake and steering, and completes lane changing driving behavior by controlling the autonomous vehicle's steering wheel, accelerator pedal and brake pedal.

[0032] Compared with existing technologies, the autonomous driving lane change decision control model based on the near-end policy optimization algorithm has a faster convergence speed and improves training efficiency. Attached Figure Description

[0033] Figure 1 This is a structural diagram of the model of the present invention.

[0034] Figure 2 This is a diagram illustrating the mechanism of action of PPO according to the present invention.

[0035] Figure 3 This is a schematic diagram of the lane-changing driving behavior of the present invention.

[0036] Figure 4 This is the TORCS training and testing map for this invention.

[0037] Figure 5 This is a comparison chart of the training time of the PPO model and the DDPG model of the present invention. Detailed Implementation

[0038] The invention will be further described below with reference to the accompanying drawings. This invention proposes an autonomous driving lane-change decision-making and control model based on a near-end strategy optimization algorithm, and implements it on the TORCS simulation platform, including the following specific steps:

[0039] Step 1: Set up the TORCS environment.

[0040] Step 1.1: Install the Ubuntu 20.04 operating system.

[0041] Step 1.2: Install the TORCS simulation platform version, TORCS version 1.3.1.

[0042] Step 1.3: Install Anaconda and its dependencies.

[0043] The model interacts with the TORCS virtual environment to obtain environmental information and output action information. Specifically, the environmental information includes information about the vehicle and the road within a certain distance, the vehicle's own speed, the distance and angle between the autonomous vehicle and the center line of the road, and the target driving state information. The action information includes steering control, throttle control, and braking control.

[0044] Step 2: Construct an autonomous driving lane change decision-making and control model based on a near-end policy optimization algorithm. The neural network of this model consists of an actor network and a critic network. The actor network includes an Actor network and an Old Actor network, while the critic network includes a Critic network. The Actor network is used to determine the control output when the autonomous vehicle performs a lane change, and the input is the current environmental information s obtained from the sensors. t The output is the control quantity that enables lane-changing driving behavior, i.e., the action space a. t The Old Actor network has the same structure as the Actor network; the Old Actor network consists of neural network parameters from several steps prior to the Actor network; the Critic network is used to score the control parameters for the lane-changing driving behavior decided by the Actor network, with the current environmental information s as input. t Output current environment information s t Below, the score of the control quantity output by the actor network, i.e., action a. t State value V(s) tThe neural network for the near-end policy optimization algorithm was constructed. A data buffer for the autonomous driving lane-change decision control model was established to store data generated during the interaction between the autonomous vehicle and the environment for model updates. The Actor network consists of four parts: state, hidden1, hidden2, and out; all four parts are linear fully connected layers, with state as the input layer, hidden1 and hidden2 as hidden layers, and out as the output layer, outputting braking, throttle, and steering control values; fc1 and fc2 both use the tanh function as the activation function. The OldActor network is identical to the Actor network. The Critic network consists of four parts: state, hidden1, hidden2, and out; all four parts are linear fully connected layers, with state as the input layer, hidden1 and hidden2 as hidden layers, and out as the output layer, outputting the Q-value of taking a certain action in a certain state; hidden1 and hidden2 use the ReLU function as the activation function.

[0045] Step 3: Test the autonomous driving lane change decision and control model based on the near-end policy optimization algorithm. Use the trained autonomous driving lane change decision and control model based on the near-end policy optimization algorithm to perform lane changes in a new environment to verify the reliability of the model.

[0046] Step 3.1: Initialize the network parameters of each neural network in the autonomous driving lane change decision control model based on the near-end policy optimization algorithm and start interacting with the environment where the autonomous vehicle is located.

[0047] Step 3.2: The autonomous vehicle interacts with the environment to obtain current environmental information. t s current environmental information t The OldActor network is input to determine the control inputs required for the autonomous vehicle to perform lane-changing driving actions, i.e., action a. mean .

[0048] Furthermore, current environmental information s t Current environmental information s tThe system comprises a 50-dimensional space, including information on the vehicle and road within a certain distance, the vehicle's own speed, the distance and angle between the autonomous vehicle and the road centerline, and the target driving state information. This 50-dimensional space includes 12 dimensions of the target driving state, 19 dimensions of distance information between the autonomous vehicle and the lane edge acquired by sensors, and sensor coverage of an area from -90 degrees to +90 degrees to the autonomous vehicle's axis (the sensor positions are distributed along the autonomous vehicle's axis at angles of 0, ±5, ±10, ±15, ±20, ±30, ±45, ±60, ±75, ±90). The distance and angle information between the autonomous vehicle and the road centerline, and the autonomous vehicle's speed along the lane direction each occupy 5 dimensions, the autonomous vehicle's speed perpendicular to the lane direction occupies 3 dimensions, and the autonomous vehicle's speed perpendicular to the ground occupies 1 dimension.

[0049] Step 3.3, based on action a mean Create a covariance matrix with standard deviation along the diagonal, and generate a multivariate normal distribution. Sample the action 'a' from the multivariate normal distribution. t And calculate action a t Log probability Logprob.

[0050] Step 3.4, the unmanned vehicle performs action a. t Based on the reward function, environmental information s is obtained. t Next, execute action a t Reward value r t And according to the state transition rules, it transitions to the state st of the next time step. +1 .

[0051] Furthermore, the reward function is shown in the following algorithm:

[0052]

[0053] Step 3.5: Repeat steps 3.2 to 3.4 128 times, and place the tuple information obtained after each execution of steps 3.2 to 3.4 into the data buffer. Each step will generate an array containing the current environment information s. t Action a t Action a t The log probability Logprob in environment s t Next, execute action a t The reward value r obtained t Environmental information s at the next moment t+1 The round end flag is_terminal.

[0054] Step 3.6: Based on all the data obtained from the data buffer, calculate the cumulative reward value R for each data point thereafter. t And calculate the cumulative reward value R for each data point thereafter.t After performing mean removal and normalization, we obtain Among them, R t =r t +γr t+1 +γ 2 r t+2 +…+γ T-t+1 r T-1 In the total duration T, r t r represents the reward value obtained at time t. t+1 r represents the reward value obtained at time t+1. T-1 This represents the reward value obtained at time T-1. γ is the discount factor, which takes a value greater than 0 and less than 1. As can be seen from formula (1), the cumulative reward value after time t is obtained by multiplying the reward value from time t to the total duration T-1 by the discount factor raised to the power of 0 to T-t+1, respectively. That is, the further away from time t is, the smaller the impact of the time on the cumulative reward value.

[0055] Step 3.7: Obtain the current environment information s for each piece of data in the data buffer. t Action space a t Inputting the Actor network yields the action space a. t The estimated log probability Logprob_eval.

[0056] Step 3.8: Set the current environment information s for each piece of data in the data buffer. t Inputting the Critic network yields the state value V. θ (s t ).

[0057] Step 3.9, according to Calculate the probability ratio r of the output actions of the Actor network and the Old Actor network. t (θ).

[0058] Step 3.10, based on the results obtained in step 3.6 The state value V(s) obtained in step 3.8 t Calculate the dominance value

[0059] Step 3.11, according to Calculate the loss value of the Actor network and update the Actor network.

[0060] Step 3.12, according to Calculate the loss value of the Critic network and update the Critic network.

[0061] Step 3.13: Repeat steps 3.2 to 3.12 500 times until the Actor network converges, indicating that the training of the autonomous driving lane change decision control model based on the proximal policy optimization algorithm is complete.

[0062] Step 4: Test the autonomous driving lane change decision and control model based on the near-end policy optimization algorithm. Use the trained autonomous driving lane change decision and control model based on the near-end policy optimization algorithm to perform lane changes in a new environment to verify the reliability of the model.

[0063] To verify the convergence speed advantage of this invention, it employs a controlled variable method and a deep deterministic policy gradient algorithm to implement lane change decision control for autonomous driving. The stability of the reward value, to a certain extent, represents the convergence of the model; the change in the reward value during training is as follows... Figure 5 As shown, the vertical axis represents the proportionally scaled-down cumulative reward value for each round, and the horizontal axis represents the number of rounds. The PPO Reward represents the reward value change curve of the autonomous driving lane change decision control model based on the proximal policy optimization algorithm during training, and the DDPG Reward represents the reward value change curve of the autonomous driving lane change decision control model based on the deep deterministic policy gradient algorithm during training. It can be seen that the autonomous driving lane change decision control model based on proximal policy optimization converged before 300 rounds, while the autonomous driving lane change decision control model based on the deep deterministic policy gradient algorithm converged only after nearly 400 rounds, fully demonstrating the advantage of this invention in convergence speed.

Claims

1. A lane-changing decision-making and control method for autonomous driving based on a near-end policy optimization algorithm, characterized in that, Includes the following steps: Step 1: Construct an autonomous driving lane change decision-making and control model based on a near-end policy optimization algorithm; the neural network of the autonomous driving lane change decision-making and control model consists of an actor network and a critic network. The actor network includes an Actor network and an OldActor network, and the critic network includes a Critic network; wherein, the Actor network is used to decide the control quantity output when the autonomous vehicle performs lane change driving behavior, and the input is the current environmental information of the autonomous vehicle obtained by the sensors. The output is the control quantity that enables lane-changing driving behavior, i.e., the action space. The Old Actor network has the same structure as the Actor network; the Old Actor network consists of neural network parameters from several steps prior to the Actor network; the Critic network is used to score the control parameters for the lane-changing driving behavior decided by the Actor network, with current environmental information as input. Output current environment information Below, the fraction of the control quantity output by the actor's network, i.e., the motion space. State value Complete the construction of the neural network for the near-end strategy optimization algorithm; establish a data buffer for the autonomous driving lane change decision control model to store the data generated during the interaction between the autonomous vehicle and the environment for model updates; Step 2: Train the autonomous driving lane change decision and control model based on the near-end policy optimization algorithm; the autonomous vehicle interacts with the surrounding environment to obtain current environmental information. Action space Action space Log-probability in the current environment Lower execution action space Rewards earned Environmental information at the next moment The round end flag is_terminal; the model is trained using information obtained from the interaction between the autonomous vehicle and its environment; current environmental information. A high-dimensional space comprising information on vehicle turning angle, vehicle speed, and distance from the vehicle to the road edge and centerline; action space. It includes steering control, throttle control, and braking control; when designing the reward function, the speed of the autonomous vehicle and the target driving state, i.e., the final driving state of the autonomous vehicle when it completes the lane change, are taken into account. Step 3: Test the autonomous driving lane change decision and control model based on the near-end policy optimization algorithm; use the trained autonomous driving lane change decision and control model based on the near-end policy optimization algorithm to perform lane changes in a new environment to verify the reliability of the autonomous driving lane change decision and control model based on the near-end policy optimization algorithm.

2. The autonomous driving lane change decision control method based on the near-end strategy optimization algorithm as described in claim 1, characterized in that, In step 2, the process of interacting with the environment is specifically as follows: The first step is to initialize the network parameters of each neural network in the autonomous driving lane change decision control model based on the near-end policy optimization algorithm and start interacting with the environment in which the autonomous vehicle is located. The second step involves the autonomous vehicle interacting with the environment to obtain current environmental information. Current environmental information The input Old Actor network determines the control variables, i.e., actions, required for the autonomous vehicle to perform lane-changing driving behavior. ; The third step is based on the action. Create a covariance matrix with standard deviation along the diagonal and generate a multivariate normal distribution. Sample the action space based on the multivariate normal distribution. And calculate the action space. The logarithmic probability Logprob; The fourth step is to define the action space for autonomous vehicles. Information about the current environment is obtained based on the reward function. Lower execution action space Reward value And according to the state transition rules, it transitions to the environmental information of the next time step. ; Fifth, repeat steps two through four several times, placing the tuple information obtained after each step into a data buffer. Each step will generate a data buffer containing information about the current environment. Action space Action space Log probability in environment Lower execution action space Rewards earned Environmental information at the next moment The round end flag is_terminal; Step 6: Based on all the data obtained from the data buffer, calculate the cumulative reward value for each data point thereafter. And the subsequent cumulative reward value for each calculated data point. After performing mean removal and normalization, we obtain ; Step 7: Obtain the current environment information for each piece of data in the data buffer. ,action Input the Actor network to obtain the actions. The estimated log probability Logprob_eval; Step 8: Obtain the current environment information for each piece of data in the data buffer. Input the Critic network to obtain the state value. ; Step 9: Calculate the probability ratio between the Actor network and the Old Actor network based on the estimated log probability Logprob_eval calculated in Step 7 and the log probability Logprob stored in the data buffer. ; Step 10, based on the results obtained in Step 6 Compared with the state value obtained in step 8 Calculate the advantage value ; Step 11: Calculate the loss value based on the loss function of the Actor network and update the Actor network; Step 12, based on the cumulative reward value thereafter. and state value Calculate the loss of the Critic network and update the Critic network; Step 13: Repeat steps 2 to 12 until the Actor network converges, indicating that the training of the autonomous driving lane change decision control model based on the proximal policy optimization algorithm is complete.

3. The autonomous driving lane change decision control method based on the near-end strategy optimization algorithm according to claim 1, characterized in that, The Actor network and Critic network are updated at the following frequency: once the autonomous vehicle interacts with the environment, one training step is performed; and once every several training steps, they are updated.

4. The autonomous driving lane change decision control method based on the near-end strategy optimization algorithm according to claim 1, characterized in that, Autonomous vehicles acquire vehicle and driving environment information through various sensors as input states. After processing by the model, the system directly outputs control values ​​for the vehicle's accelerator, brake, and steering. By controlling the steering wheel, accelerator pedal, and brake pedal, the autonomous vehicle can perform lane-changing driving behaviors.

Images