A hierarchical control method for unmanned vehicle following based on deep reinforcement learning

Abstract

The present invention discloses a hierarchical control method for following an unmanned vehicle based on deep reinforcement learning, and belongs to the technical field of unmanned vehicles. A kinematic model of the distance between two vehicles followed by a vehicle and the deflection angle of the target vehicle coordinates relative to the front direction of the following vehicle is established; a target recognition algorithm is trained to identify the target vehicle and the algorithm is deployed on the following vehicle; the coordinates of the target vehicle relative to the following vehicle are calculated according to the recognition result; the following vehicle problem is established as a Markov decision process, and a reward function is designed using vehicle control delay and deceleration; then a deep reinforcement learning algorithm is used to train a vehicle following agent neural network model; the acceleration and angular velocity of the following vehicle are controlled; finally, the vehicle following agent neural network model is deployed on the following vehicle, and the following vehicle is controlled using various parameters in the underlying control algorithm. The present invention does not need to know the kinematic model and dynamic model of the target vehicle, and simultaneously realizes lateral and longitudinal following, and can be directly applied to a real vehicle without migration operations.

Description

Technical Field

[0001] The present invention belongs to the technology of driverless vehicles, and in particular relates to a hierarchical vehicle following control method applicable in a real environment when the dynamics model and kinematic model of a target vehicle are unknown. Background Art

[0002] Autonomous driving technology can provide convenient and safe driving by avoiding accidents caused by driver errors, among which vehicle following control is the basis of autonomous driving. In the context of the rapid development of autonomous driving technology in recent years, the purpose of vehicle following control is to collect information about the traffic environment and target vehicles in real time through on-board sensors, help drivers judge potential dangers, predict the behavior of other vehicles, and achieve instant control of the vehicle, which has important practical significance for the safe driving of vehicles.

[0003] There are currently two methods to solve the vehicle following control problem. One is based on classical control theory and modern control theory. The control theory-based method analyzes the unmanned vehicle control system and establishes a mathematical model to design control parameters to ensure the stability of the vehicle during the following process. However, these methods require the establishment of models for the distance between the two vehicles, the speed of the two vehicles, etc., which are greatly affected by errors. In addition, some methods require a priori kinematic models of the two vehicles, and need to adjust parameters for specific problems and scenarios, which is not generalizable. The other method is a data-driven method, mainly including methods based on deep reinforcement learning. In recent years, the rapid development of deep reinforcement learning has shown its advantages in solving complex control and decision-making problems, so more and more researchers use deep reinforcement learning methods to solve vehicle following control problems. However, most methods usually assume that the target vehicle has a relatively simple motion mode, such as uniform motion or uniform acceleration motion, and only realize longitudinal following, without considering the longitudinal control of the following vehicle. In addition, since deep reinforcement learning is difficult to train on a real vehicle, most methods are experiments conducted in simulation. On the other hand, most of the proposed methods are end-to-end. Although the end-to-end approach has many advantages, such as being able to directly output control signals from input sensors without considering intermediate results, in vehicle control problems, the end-to-end approach often finds it difficult to obtain stable, reliable, and smooth outputs.

[0004] Due to the various shortcomings mentioned above, there is currently no vehicle following method based on deep reinforcement learning that is used on a real vehicle. Therefore, vehicle following control in real scenarios where the kinematic model of the target vehicle is unknown has important research value. In view of the shortcomings of existing methods, this paper proposes a hierarchical vehicle following control method based on deep reinforcement learning. Summary of the invention

[0005] The purpose of the present invention is to provide a vehicle following control method for real scenarios where the kinematic model of the target vehicle is unknown. To achieve the above purpose, the present invention proposes a layered vehicle following control method for an unmanned vehicle based on deep reinforcement learning, comprising the following steps:

[0006] S1: Establish the distance D between the vehicle following the other vehicle target The kinematic model of the deviation angle θ of the target vehicle coordinates relative to the front direction of the following vehicle;

[0007] S2: Experimental results show that the following vehicle control delay t delay and the deceleration λ during inertia deceleration;

[0008] S3: training a target recognition algorithm to identify the target vehicle and deploying the target recognition algorithm on the follower vehicle;

[0009] For example, train the YOLO algorithm to identify the target vehicle and deploy the YOLO algorithm on the follower vehicle;

[0010] S4: Calculate the coordinates of the target vehicle relative to the following vehicle based on the results of the algorithm recognition in S3, and use the kinematic model in S1 to calculate the distance D between the two vehicles. target and deflection angle θ and stored as a data set;

[0011] S5: The following problem is formulated as a Markov decision process and the vehicle control delay t in S2 is used delay and deceleration λ to design the reward function;

[0012] S6: Using the Markov decision process and reward function in S5, a deep reinforcement learning algorithm is used to train the vehicle following agent neural network model;

[0013] For example, the Markov decision process and reward function in S5 are used to train the vehicle following agent neural network model using the PPO algorithm (Schulman J, Wolski F, Dhariwal P, et al. Proximal policy optimization algorithms[J].);

[0014] S7: Adjust various parameters in the underlying control algorithm based on control theory to control the acceleration and angular velocity of the following vehicle.

[0015] For example, various parameters in the PID control algorithm are adjusted to control the acceleration and angular velocity of the following vehicle.

[0016] S8: Deploy the model trained in S6 on the follower car and use the parameters in S7 to control the follower car.

[0017] Step S1 further comprises:

[0018] S100: A right-handed coordinate system is established with the coordinates of the following vehicle as the origin and the direction of the vehicle head as the y-axis.

[0019] S101: The coordinates (x, y) of the target vehicle relative to the following vehicle, and the distance between the two vehicles The deviation angle of the target vehicle coordinates relative to the front direction of the following vehicle

[0020]

[0021] Step S2 further comprises:

[0022] S200: Control the periodic movement of the vehicle to observe the control delay t delay ;

[0023] S201: Perform multiple inertial deceleration experiments, record the speed at the beginning of each deceleration, the distance and time from deceleration to 0, and calculate the deceleration λ during inertial deceleration;

[0024] Step S4 further comprises:

[0025] S400: The following vehicle calculates the coordinates (x, y) of the target vehicle relative to the following vehicle based on the target recognition result and the high-precision positioning system;

[0026] S401: Calculate the distance D between the two vehicles using the calculation method in S201 target and deflection angle θ;

[0027] S402: Store the result in S401 as a data set.

[0028] Step S5 further comprises:

[0029] S500: The action space is a binary (V follower , O), where V follower is the speed range of the following vehicle in meters, O is the angle range of the following vehicle in degrees; the action represents the expected speed and angle of the following vehicle in the current state; the lower layer uses the underlying control algorithm based on control theory to calculate the specific acceleration and angular velocity;

[0030] For example, the lower layer uses the underlying control algorithm based on control theory, which is the PID control algorithm, to calculate the specific acceleration and angular velocity.

[0031] S501: The state space is (V follower , D target ,θ), where V follower is the speed of the following vehicle, D target is the distance between the target vehicle and the following vehicle, θ is the deflection angle of the target vehicle coordinate relative to the front direction of the following vehicle;

[0032] S502: Longitudinal control reward function R l for:

[0033]

[0034] Where D safe is the safe distance between the following vehicle and the target vehicle, D braking The braking distance of the following vehicle is calculated as follows:

[0035] D braking =V follower *t delay +0.5*V follower 2 / λ, where t delay is the control delay of the following vehicle, λ is the deceleration of the following vehicle during inertial motion;

[0036] S503: Lateral control reward function R a for:

[0037] R a =-|θ+O t |,

[0038] Among them, t is the angle that the following car should maintain at time t;

[0039] S504: The final reward function is:

[0040] R=R l +R a ;

[0041] Preferably, step S6 further comprises:

[0042] S600: Use the Markov decision process in S5 to build a deep reinforcement learning training environment;

[0043] S601: Build a deep neural network with Actor-Critic structure;

[0044] S602: Setting PPO algorithm hyperparameters and starting training;

[0045] S603: Keep the trained network model.

[0046] Effective benefits of the present invention:

[0047] 1. The present invention only uses sensor data on a real vehicle for training, does not need to know the kinematic model and dynamic model of the target vehicle, can simultaneously achieve lateral and longitudinal following, and can be directly applied to a real vehicle without any migration operation.

[0048] 2. In order to make the control of the unmanned vehicle more stable, the present invention proposes a hierarchical control method for the unmanned vehicle. The upper layer uses a deep reinforcement learning strategy to obtain the speed and angle that the following vehicle should maintain at every moment when following, and the lower layer uses PID to control the linear acceleration and angular velocity of the unmanned vehicle. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 This is a flow chart of a vehicle following control method based on deep reinforcement learning;

[0050] Figure 2 It is the vehicle following model in the present invention;

[0051] Figure 3 is a following vehicle used in a specific embodiment;

[0052] Figure 4 The reward function change curve during training of the present invention;

[0053] Figure 5 This is the longitudinal following result of the actual vehicle experiment of the present invention;

[0054] Figure 6 This is the lateral following result of the actual vehicle experiment of the present invention. DETAILED DESCRIPTION

[0055] The present invention is further described in detail below through specific examples and in conjunction with the drawings of the specification. Those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different examples, and the details in this specification can also be modified and changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention.

[0056] Figure 1 A vehicle following control method based on deep reinforcement learning according to an embodiment of the present invention is provided, comprising the following steps:

[0057] S1: First, establish the distance D between the vehicle following the other vehicle target The model of the deviation angle θ of the target vehicle coordinates relative to the front direction of the following vehicle.

[0058] Specifically, in this embodiment, the coordinates of the following vehicle are (0,0), and the coordinates of the target vehicle are (x,y). The maximum speed of the target vehicle and the following vehicle is 10m / s, and when the distance between the two vehicles exceeds 50 meters, it is determined that the following fails. The maximum angular velocity of the target vehicle and the following vehicle is 30° / s. The step S1 further includes:

[0059] S100: A right-handed coordinate system is established with the coordinates of the following vehicle as the origin and the direction of the vehicle head as the y-axis.

[0060] S101: The target vehicle coordinates are (x, y), and the distance between the two vehicles is The deviation angle of the target vehicle coordinates relative to the front direction of the following vehicle

[0061]

[0062] S2: Experimental results show that the vehicle control delay t delay And the deceleration λ during inertia deceleration.

[0063] Specifically, the vehicle following scene and the target vehicle and the following vehicle used in this embodiment are all ordinary commercial vehicles. The following vehicle is equipped with sensors such as RGB cameras and high-precision positioning systems. Figure 3 The target vehicle is not equipped with a sensor. The step S2 further comprises:

[0064] S201: Control the following vehicle to periodically accelerate and decelerate to observe the control delay t delay In this embodiment, t delay =1s

[0065] S202: The following vehicle accelerates to a certain speed and maintains it for a certain period of time, then releases the accelerator to allow the vehicle to decelerate to 0 by inertia and friction. Perform multiple experiments and record the speed at the beginning of deceleration, the distance and time of deceleration to 0 each time to calculate the deceleration rate λ during inertial deceleration. In this embodiment, λ=2m / s.

[0066] S3: Train the YOLOv5 algorithm to identify the target vehicle and deploy it on the follower vehicle.

[0067] Specifically, the following car is controlled and the RGB camera on the following car is used to take photos of the target car from multiple angles. The pictures are processed and organized into a data set, which is then trained using the YOLOv5 algorithm. Finally, the trained model is deployed on the following car, which can detect the target car in real time.

[0068] S4: Calculate the coordinates of the target vehicle relative to the following vehicle based on the recognition results in S3, and use the model in S2 to calculate the distance D between the two vehicles. target and deflection angle θ and stored as a data set.

[0069] Specifically, two vehicles are manually controlled to follow each other, and data is recorded to generate a data set. The step S4 further includes:

[0070] S400: The target vehicle performs random acceleration, deceleration, left turn, right turn and other actions, and the follower vehicle is manually controlled to track the target vehicle.

[0071] S401: The coordinates (x, y) of the target vehicle relative to the following vehicle are inferred based on the target recognition result of YOLOv5 and the high-precision positioning system, lidar and other sensors on the following vehicle.

[0072] S402: Calculate the distance D between the two vehicles according to S101 target and the deflection angle θ.

[0073] S403: Store the results in S402 in order as a data set.

[0074] S5: Model the car-following problem as a Markov decision process and use the vehicle control delay t in S2 delay and deceleration λ to design the reward function.

[0075] Specifically, this step models the vehicle following problem as a Markov decision process according to the definition, further comprising:

[0076] S500: Define the action space as a binary (V follower， O), where V follower is the speed range of the following vehicle, in meters, and O is the angle range of the following vehicle, in degrees. In this embodiment, V is defined according to the actual conditions of the following vehicle and the target vehicle. follower ∈[0, 10], O∈[-30, 30].

[0077] S501: Define the state space as (V follower , D target ,θ), where V follower is the speed of the following vehicle, D target is the distance between the target vehicle and the following vehicle, and θ is the deflection angle of the target vehicle coordinates relative to the front direction of the following vehicle.

[0078] S502: Define the safe distance D between the two vehicles based on the size and speed of the two vehicles safe =8m. According to the control delay and deceleration in S2, the maximum braking distance of the following vehicle is:

[0079] D braking =V follower *1+0.5*V follower 2 / 2.

[0080] So the vertical control reward function R l for:

[0081]

[0082] S503: Lateral control reward function R a for:

[0083] R a =-|θ+O t |,

[0084] Among them, tis the angle that the following car should maintain at time t.

[0085] S504: The final reward function is:

[0086] R=R l +R a .

[0087] S6: Using the Markov decision process and reward function in S5, the PPO algorithm is used to train the vehicle following agent.

[0088] Specifically, this step designs a deep neural network and trains a vehicle-following agent, further including:

[0089] S600: Build a deep neural network with an Actor-Critic structure to train the driverless car. Both Actor and Critic are multi-layer perceptrons, with the input being the state in S501 and the output being the value and action, which are used to calculate the loss function of the subsequent PPO algorithm.

[0090] S601: Based on experience, the hyperparameters of PPO are set as follows: the optimizer is Adam, and the learning rate is 1e -5 ; The discount factor γ = 0.9, the weight is 1e -2 ; The weight of the value function is 1e -3 ; The clipping range of the ratio between the old strategy and the new strategy is [0.8,1.2].

[0091] S602: When the cumulative rewards during training and the cumulative rewards during evaluation are both stable, save the model. Figure 4 This is the reward function change curve during training and evaluation of the present invention. It can be seen that as the training time increases, the training results gradually converge.

[0092] S7: Adjust various parameters in the PID algorithm to control the acceleration and angular velocity of the following vehicle.

[0093] S8: Deploy the model trained in S6 on the follower car and use the parameters in S7 to control the follower car.

[0094] In order to better illustrate the technical effect of the present invention, the inventor conducted an experiment in a real environment. The initial distance between the target vehicle and the following vehicle was 8m. The target vehicle periodically performed rapid acceleration and deceleration and made random turns. During the experiment, the following vehicle received the position of the target vehicle at a frequency of 5Hz to infer the speed and angle it should maintain. Figure 3 The present invention is deployed on the unmanned vehicle shown and the currently commonly used IDM method is implemented for comparison.

[0095] Figure 5 is the longitudinal follow-up result of the experiment, Figure 5(a) is the predicted braking distance of IDM and the distance to the target vehicle, Figure 5 (b) IDM’s predicted speed and target vehicle speed, Figure 5 (c) is the acceleration result predicted by IDM. Figure 5 (d) (e) (f) are the braking distance, predicted speed and acceleration results of the present invention respectively. Figure 5 (a) and (d), it can be seen that the braking distance predicted by the IDM method is greater than the target vehicle distance at certain moments, causing safety hazards. The braking distance of the present invention is always less than the target distance and remains within a stable and reasonable range. Figure 5 (e) It can be seen that the predicted speed of the present invention is consistent with the speed change of the target vehicle, indicating that the present invention can respond quickly to the sudden acceleration and deceleration of the target vehicle and maintain smooth speed changes. Figure 5 (c) and (f), the acceleration result obtained by the present invention has smaller jitter than the IDM result, which shows that the hierarchical control method brings a more stable control effect and is more suitable for vehicle chassis control.

[0096] The lateral control index is the difference between the deviation angle of the target vehicle coordinates relative to the front direction of the following vehicle and the angle of rotation of the following vehicle. Figure 6 It can be seen that its value always remains within [-3,3], which meets the lateral control requirements of the actual vehicle.

Claims

1. A hierarchical vehicle following control method for unmanned vehicles based on deep reinforcement learning, characterized in that: The following steps are involved: S1: Establish the distance D between the two vehicles following the vehicle target The kinematic model of the deviation angle θ of the target vehicle coordinates relative to the front direction of the following vehicle; S2: Experimental results show that the following vehicle control delay t delay and the deceleration λ during inertia deceleration; S3: training a target recognition algorithm to identify the target vehicle and deploying the target recognition algorithm on the follower vehicle; S4: Calculate the coordinates of the target vehicle relative to the following vehicle based on the results of the algorithm recognition in S3, and use the kinematic model in S1 to calculate the distance D between the two vehicles. target and deflection angle θ and stored as a data set; S5: The following problem is formulated as a Markov decision process and the vehicle control delay t in S2 is used delay and deceleration λ to design the reward function; S6: Using the Markov decision process and reward function in S5, a deep reinforcement learning algorithm is used to train the vehicle following agent neural network model; S7: Adjust various parameters in the underlying control algorithm based on control theory to control the acceleration and angular velocity of the following vehicle; S8: deploy the model trained in S6 on the following vehicle, and use the parameters in S7 to control the following vehicle; The step S1 further comprises: S100: Establish a right-handed coordinate system with the coordinates of the following vehicle as the origin and the direction of the vehicle head as the y-axis; S101: The coordinates of the target vehicle relative to the following vehicle are (x, y), and the distance between the two vehicles is The deviation angle θ of the target vehicle coordinates relative to the front direction of the following vehicle, The step S5 further comprises: S500: The action space is a binary (V follower ,O), where V follower is the speed range of the following vehicle in meters, O is the angle range of the following vehicle in degrees; the action represents the expected speed and angle of the following vehicle in the current state; the lower layer uses the underlying control algorithm based on control theory to calculate the specific acceleration and angular velocity; S501: The state space is (V follower ,D target ,θ), where V follower is the speed of the following vehicle, D target is the distance between the target vehicle and the following vehicle, θ is the deflection angle of the target vehicle coordinate relative to the front direction of the following vehicle; S502: Longitudinal control reward function R l for: Where D safe is the safe distance between the following vehicle and the target vehicle, D braling The braking distance of the following vehicle is calculated as follows: D braking =V follower *t delay +0.5*V follower 2 / A, where t delay is the control delay of the following vehicle, λ is the deceleration of the following vehicle during inertial motion; S503: Lateral control reward function R a for: R a =-|θ+O t | (3) Among them, t is the angle that the following car should maintain at time t; S504: The final reward function is: R=R l +R a (4)。 2. According to the method of claim 1, the layered vehicle following control method of an unmanned vehicle based on deep reinforcement learning is characterized in that: The step S2 comprises: S200: Control the periodic movement of the vehicle to observe the control delay t delay ; S201: Perform multiple inertial deceleration experiments, record the speed at the beginning of each deceleration, the distance and time from deceleration to 0, and calculate the deceleration λ during inertial deceleration.

3. The method for controlling the hierarchical vehicle following of an unmanned vehicle based on deep reinforcement learning according to claim 1 is characterized in that: The step S4 further comprises: S400: The following vehicle calculates the coordinates (x, y) of the target vehicle relative to the following vehicle based on the target recognition result and the high-precision positioning system; S401: Calculate the distance D between the two vehicles using the calculation method in S201 target and deflection angle θ; S402: Store the result in S401 as a data set.

4. The method for controlling the hierarchical vehicle following of an unmanned vehicle based on deep reinforcement learning according to claim 1 is characterized in that: The step S6 further comprises: S600: Use the Markov decision process in S5 to build a deep reinforcement learning training environment; S601: Build a deep neural network with Actor-Critic structure; S602: Setting PPO algorithm hyperparameters and starting training; S603: Keep the trained network model.

5. A method for controlling the hierarchical vehicle following of an unmanned vehicle based on deep reinforcement learning according to any one of claims 1 to 4, characterized in that: The target recognition algorithm in step S3 is the YOLO algorithm.

6. A method for controlling the hierarchical vehicle following of an unmanned vehicle based on deep reinforcement learning according to any one of claims 1 to 4, characterized in that: The deep reinforcement learning algorithm in step S6 is the PPO algorithm.

7. A method for controlling the hierarchical vehicle following of an unmanned vehicle based on deep reinforcement learning according to any one of claims 1 to 4, characterized in that: The underlying control algorithm of step S500 based on control theory is PID control algorithm.

Images