End-to-end longitudinal motion control method for autonomous driving based on deep reinforcement learning

Abstract

The present invention provides an end-to-end longitudinal motion control method for autonomous driving based on deep reinforcement learning. The reinforcement learning model established based on the Markov decision process is used to implement model training in a simulation environment. After the training is completed in the cloud, the actual vehicle is deployed. The on-board domain controller uses the perception information as input in real time and outputs instructions for the motor and brake torque, thereby realizing end-to-end longitudinal motion control of the intelligent networked electric vehicle. For the initialization of the reinforcement learning model training, the pre-training results of the hierarchical control strategy are utilized, thereby greatly accelerating the training in an imitation learning manner, and combining the optimization of multiple objectives to significantly improve the performance of the trained model. When the end-to-end control fails, the vehicle itself completes the control decision, and after all the autonomous driving strategies fail, the driver takes over. In this way, the redundancy mechanism fully guarantees the control stability and driving safety.

Description

Technical Field

[0001] The present invention belongs to the technical field of autonomous driving vehicle control, and in particular relates to an end-to-end longitudinal motion control method for autonomous driving based on deep reinforcement learning. Background Art

[0002] In the control of autonomous vehicles, model-based longitudinal motion control is currently widely used. Most of them are based on the car-following model. The optimal control law is obtained by sequentially establishing the state space, optimizing the objective function, and solving the constrained optimization problem. For more complex working conditions, the working conditions are usually divided into hierarchical multi-model and multi-mode decision-making methods. However, when faced with ever-changing driving scenarios, mode switching often leads to system instability. Although some existing technologies can improve adaptability to various scenarios including those outside the training environment by introducing deep reinforcement learning, their huge search space and sparse reward space also lead to slower learning speeds. In the commonly used upper and lower layered control strategy, since each layer is optimized only based on its own objective function, the improvement of the overall system is limited. Summary of the Invention

[0003] In view of this, and to address the technical problems existing in this field, the present invention provides an end-to-end longitudinal motion control method for autonomous driving based on deep reinforcement learning, which specifically includes the following steps:

[0004] Step 1: Establish a vehicle longitudinal following model to measure the real-time distance, speed difference, and distance error between the vehicle and the preceding vehicle, as well as the relationship between the vehicle's acceleration and speed;

[0005] Step 2: Build a reinforcement learning model based on the Markov decision process. This includes defining the vehicle actions that determine the decision output, establishing the vehicle state space using physical quantities related to vehicle motion and power, and defining the rewards in the decision-making process based on four performance aspects: following stability, safety, comfort, and energy efficiency.

[0006] Step 3: Using the vehicle state and sensor data from onboard sensors to train an existing longitudinal motion control strategy based on hierarchical control, such as an expert strategy, and using the trained strategy to initialize the training process of the reinforcement learning model to accelerate the training in an imitation learning manner;

[0007] Step 4: After completing the initialization, the reinforcement learning model is trained until convergence; the trained reinforcement learning model serves as the intelligent driving domain, and provides the vehicle-side motion control domain with hierarchical control instructions for generating desired speed and desired motor torque control instructions based on the perception data;

[0008] Step 5: After completing steps 1 to 4 above on the cloud, the trained reinforcement learning model is applied to the autonomous driving control of the actual vehicle. The motion control domain receives the instructions from the intelligent driving domain and outputs motor and brake torque instructions, achieving end-to-end longitudinal motion control of the vehicle from the cloud to the vehicle.

[0009] The vehicle also downloads the trained reinforcement learning model, which is used to perform autonomous driving control by the vehicle when end-to-end control fails. This involves first executing hierarchical control to generate the aforementioned commands, and then the motion control domain combines these commands with the driver model running on the vehicle to output motor and brake torque commands.

[0010] When both types of automatic driving control fail, the driver takes over, and the motion control domain outputs motor and brake torque commands based on the driver's pedal signal and the driver model running on the vehicle side.

[0011] Furthermore, the specific form of the car-following model established in step 1 includes:

[0012] The distance d(k) between the vehicle and the preceding vehicle at time k:

[0013] d(k)=x p (k)-x(k)

[0014] Among them, x(k), x p (k) represents the distance between the vehicle and the preceding vehicle at time k;

[0015] The speed difference Δv(k) between the vehicle and the preceding vehicle at time k:

[0016] Δv(k)=v p (k)-v(k)

[0017] Among them, v(k), v p (k) represents the speed of the vehicle and the preceding vehicle at time k respectively;

[0018] The distance error Δd(k) between the vehicle and the preceding vehicle at time k:

[0019] Δd(k)=d(k)-d * (k)

[0020] Where d*(k) represents the expected distance between the vehicle and the preceding vehicle at time k, which is determined by the selected following strategy.

[0021] The relationship between the vehicle's acceleration and velocity at time k is based on the velocity model and the time step T s Sure:

[0022]

[0023] Furthermore, the reinforcement learning model established in step 2 specifically includes:

[0024] The output of each vehicle action includes driving torque and braking torque;

[0025] The state space established takes the following form:

[0026] S={v p , v ego , a ego ,|dD safe |,v ego -v ref ,d,T mot , P b}

[0027] Where, v p is the speed of the preceding vehicle, v ego is the vehicle speed, a ego is the vehicle's acceleration, v ref is the reference speed of the vehicle, d is the distance between the vehicle and the preceding vehicle, D safe is the safe distance between the vehicle and the preceding vehicle, T mot is the motor torque, P b is the power battery output power;

[0028] Defining the rewards in the decision-making process includes:

[0029] Following stability reward: For the distance control mode that makes the following distance close to the expected following distance, and the speed control mode that makes the vehicle speed close to the speed of the preceding vehicle, a following distance reward function r is established respectively. follow1 and the following vehicle speed reward function r follow2 ;

[0030] Among them, in the vehicle distance control mode, considering that the following distance approaches the desired safety distance D safe When the reward value is larger, the following distance deviates from D safe When , the smaller the reward value, and the tanh function is used to establish the following vehicle distance reward function r follow1 , so that the function value range is [-1,0]:

[0031] r follow1 =-tanh(|DD safe |)

[0032] In the vehicle speed control mode, considering the vehicle speed v ego With reference speed v ref The difference is the smallest, and the tanh function is also used to establish the following vehicle speed reward function r follow2 :

[0033] rfollow2 =-tanh(|v ego -v ref |)

[0034] The reference speed v ref The definition includes: when there is no target vehicle in front of the vehicle, the vehicle's set speed is used as the reference speed; when the vehicle enters the cruise mode and the distance between the vehicle and the vehicle in front is less than or equal to D safe , the minimum value between the front vehicle speed and the set speed is taken:

[0035]

[0036] Take r follow1 With r follow2 The minimum value of the two is used as the following vehicle stability reward function r follow :

[0037] r follow =min{r follow1 , r follow2};

[0038] Safety reward: Considering the longitudinal motion safety of the vehicle and establishing a safety reward function r based on the distance between the vehicle and the preceding vehicle safe :

[0039]

[0040] Comfort bonus: When the vehicle is moving, consider reducing the inertia force during the vehicle's movement to ensure riding comfort, specifically using the driving acceleration a ego and jerk ego To characterize, the smaller the absolute value of acceleration and acceleration change rate, the higher the riding comfort; at the same time, considering the impact of motor torque change on comfort, combined with the action change rate Δu output by the reinforcement learning model to establish a comfort reward function:

[0041]

[0042] in, Δu=u k -u k-1 ;

[0043] Energy consumption economy bonus: Considering the vehicle's power consumption per unit mileage, including driving power consumption and braking energy recovery, d , using the tanh function to establish the following energy consumption economy reward function:

[0044] r eco = -tanh E d

[0045] Among them, P b，out is the battery output power, P b，regen is the battery's recoverable power, and the total battery power consumption at time t is E, then E d for:

[0046]

[0047] The end-to-end longitudinal motion control method for autonomous driving based on deep reinforcement learning provided by the present invention implements model training in a simulation environment based on a reinforcement learning model established based on a Markov decision process. After the training is completed in the cloud, the actual vehicle is deployed. The on-board domain controller uses the perception information as input in real time and outputs instructions for the motor and brake torque, thereby realizing end-to-end longitudinal motion control of the intelligent connected electric vehicle. For the initialization of reinforcement learning model training, the pre-training results of the hierarchical control strategy are utilized, thereby greatly accelerating the training in an imitation learning manner, and combining the optimization of multiple objectives also significantly improves the performance of the trained model. When the end-to-end control fails, the vehicle itself completes the control decision, and after all the autonomous driving strategies fail, the driver takes over. In this way, the redundancy mechanism fully guarantees the control stability and driving safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 This is a framework diagram of the reinforcement learning model provided by the present invention;

[0049] Figure 2 A diagram of a system network structure that can be selected for the method provided by the present invention;

[0050] Figure 3 This is a flowchart of the specific control and redundancy switching of the method provided by the present invention. DETAILED DESCRIPTION

[0051] The technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0052] The end-to-end longitudinal motion control method for autonomous driving based on deep reinforcement learning provided by the present invention specifically includes the following steps:

[0053] Step 1: Establish a vehicle longitudinal following model to measure the real-time distance, speed difference, and distance error between the vehicle and the preceding vehicle, as well as the relationship between the vehicle's acceleration and speed;

[0054] Step 2: Build a reinforcement learning model based on the Markov decision process. This includes defining the vehicle actions that determine the decision output, establishing the vehicle state space using physical quantities related to vehicle motion and power, and defining the rewards in the decision-making process based on four performance aspects: following stability, safety, comfort, and energy efficiency.

[0055] Step 3: Using the vehicle state and sensor data from onboard sensors to train an existing longitudinal motion control strategy based on hierarchical control, such as an expert strategy, and using the trained strategy to initialize the training process of the reinforcement learning model to accelerate the training in an imitation learning manner;

[0056] The composition framework and training process of the model are as follows Figure 1 As shown in the figure, different strategies can also be adopted for the hierarchical control of imitation learning and final real-vehicle application to further improve the applicability of the model to a variety of different scenarios.

[0057] Step 4: After completing the initialization, the reinforcement learning model is trained until convergence; the trained reinforcement learning model serves as the intelligent driving domain, and provides the vehicle-side motion control domain with hierarchical control instructions for generating the desired speed and the desired motor torque control instructions based on the perception data; Figure 2 As shown, based on the trained intelligent model, the corresponding system network architecture can be formed by the cloud platform, vehicle-side perception and execution components, the two domains, and appropriate communication cables;

[0058] Step 5: After completing steps 1 to 4 above on the cloud, the trained reinforcement learning model is applied to the autonomous driving control of the actual vehicle. The motion control domain receives the instructions from the intelligent driving domain and outputs motor and brake torque instructions, achieving end-to-end longitudinal motion control of the vehicle from the cloud to the vehicle.

[0059] Figure 3 The end-to-end control process and associated redundancy mechanisms implemented by this method are shown. The vehicle also downloads a trained reinforcement learning model, which is used to enable autonomous driving control by a single vehicle in the event of a failure in end-to-end control. This involves first executing hierarchical control to generate the aforementioned commands, and then the motion control domain combines these commands with the driver model running on the vehicle to output motor and brake torque commands.

[0060] When both types of automatic driving control fail, the driver takes over, and the motion control domain outputs motor and brake torque commands based on the driver's pedal signal and the driver model running on the vehicle side.

[0061] The above-mentioned redundancy mechanism can also execute switching between different control modes according to a pre-set safety level.

[0062] In a preferred embodiment of the present invention, the specific form of the car-following model established in step 1 includes:

[0063] The distance d(k) between the vehicle and the preceding vehicle at time k:

[0064] d(k)=x p (k)-x(k) where x(k), x p (k) represents the distance between the vehicle and the preceding vehicle at time k;

[0065] The speed difference Δv(k) between the vehicle and the preceding vehicle at time k:

[0066] Δv(k)=v p (k)-v(k)

[0067] Among them, v(k), v p (k) represents the speed of the vehicle and the preceding vehicle at time k respectively;

[0068] The distance error Δd(k) between the vehicle and the preceding vehicle at time k:

[0069] Δd(k)=d(k)-d * (k)

[0070] Where d*(k) represents the expected distance between the vehicle and the preceding vehicle at time k, which is determined by the selected following strategy.

[0071] The relationship between the vehicle's acceleration and velocity at time k is based on the velocity model and the time step T s Sure:

[0072]

[0073] In a preferred embodiment of the present invention, the reinforcement learning model established in step 2 specifically includes:

[0074] The output of each vehicle action includes driving torque and braking torque;

[0075] The state space established takes the following form:

[0076] S={v p , v ego , a egp ,|dD safe |,v ego -v ref ,d,T mot , P b}

[0077] Where, v p is the speed of the preceding vehicle, v ego is the vehicle speed, a ego is the vehicle's acceleration, v refis the reference speed of the vehicle, d is the distance between the vehicle and the preceding vehicle, D safe is the safe distance between the vehicle and the preceding vehicle, T mot is the motor torque, P b is the power battery output power;

[0078] Defining the rewards in the decision-making process includes:

[0079] Following stability reward: For the distance control mode that makes the following distance close to the expected following distance, and the speed control mode that makes the vehicle speed close to the speed of the preceding vehicle, a following distance reward function r is established respectively. follow1 and the following vehicle speed reward function r follow2 ;

[0080] Among them, in the vehicle distance control mode, considering that the following distance approaches the desired safety distance D safe When the reward value is larger, the following distance deviates from D safe When , the smaller the reward value, and the tanh function is used to establish the following vehicle distance reward function r follow1 , so that the function value range is [-1,0]:

[0081] r follow1 =-tanh(|dD safe )

[0082] In the vehicle speed control mode, considering the vehicle speed v ego With reference speed v ref The difference is the smallest, and the tanh function is also used to establish the following vehicle speed reward function r follow2 :

[0083] r follow2 =-tanh(|v ego -v ref |)

[0084] The reference speed v ref The definition includes: when there is no target vehicle in front of the vehicle, the vehicle's set speed is used as the reference speed; when the vehicle enters the cruise mode and the distance between the vehicle and the vehicle in front is less than or equal to D safe , the minimum value between the front vehicle speed and the set speed is taken:

[0085]

[0086] Take r follow1 With r follow2 The minimum value of the two is used as the following vehicle stability reward function r follow :

[0087] r follow =min{r follow1 , rfollow2};

[0088] Safety reward: Considering the longitudinal motion safety of the vehicle and establishing a safety reward function r based on the distance between the vehicle and the preceding vehicle safe :

[0089]

[0090] Comfort bonus: When the vehicle is moving, consider reducing the inertia force during the vehicle's movement to ensure riding comfort, specifically using the driving acceleration a ego and jerk ego To characterize, the smaller the absolute value of acceleration and acceleration change rate, the higher the riding comfort; at the same time, considering the impact of motor torque change on comfort, combined with the action change rate Δu output by the reinforcement learning model to establish a comfort reward function:

[0091]

[0092] in, Δu=u k -u k-1 ;

[0093] Energy consumption economy bonus: Considering the vehicle's power consumption per unit mileage, including driving power consumption and braking energy recovery, d , using the tanh function to establish the following energy consumption economy reward function:

[0094] r eco = -tanh E d

[0095] Among them, P b，out is the battery output power, P b，regen is the battery's recoverable power, and the total battery power consumption at time t is E, then E d for:

[0096]

[0097] It should be understood that the size of the serial numbers of the steps in the embodiment of the present invention does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0098] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A deep reinforcement learning-based end-to-end longitudinal motion control method for autonomous driving, characterized by: The following steps are involved: Step 1: Establish a longitudinal car-following model to describe the real-time distance, speed difference, and distance error between the vehicle and the preceding vehicle, as well as the relationship between the vehicle's acceleration and speed. Step 2: Establish a reinforcement learning model based on the Markov decision process, including: Output of various vehicle actions, including driving torque and braking torque; Establish the state space S: S={v p ,v ego ,a ego ,|d-D safe |,v ego -v ref ,d,T mot ,P b } v p is the speed of the preceding vehicle, v ego is the vehicle speed, a ego is the vehicle's acceleration, v ref is the reference speed of the vehicle, d is the distance between the vehicle and the preceding vehicle, D safe is the safe distance between the vehicle and the preceding vehicle, T mot is the motor torque, P b is the power battery output power; Define the rewards in the decision-making process, including: Following vehicle stability reward r follow : r follow =min{r follow1 ,r follow2 } r follow1 =-tanh(|dD safe |) r follow2 =-tanh(|v ego -v ref |) v set Set the speed for your vehicle; Security Reward safe : Comfort Bonus confort : a ego is the vehicle acceleration, Jerk ego is the acceleration change rate, Δu is the action change rate; Energy economy reward eco : r eco =-tanh E d P b，out is the battery output power, P b，regen is the battery's recoverable power, and the total battery power consumption at time t is E; Step 3: Using the vehicle state and sensor data from the onboard sensors to train a longitudinal motion control strategy based on hierarchical control, and using the trained strategy to initialize the training process of the reinforcement learning model to accelerate the training in an imitation learning manner; Step 4: After completing the initialization, the reinforcement learning model is trained until convergence; the trained reinforcement learning model serves as the intelligent driving domain, and provides the vehicle-side motion control domain with hierarchical control instructions for generating desired speed and desired motor torque control instructions based on the perception data; Step 5. After completing steps 1 to 4 above on the cloud, the trained reinforcement learning model is applied to the autonomous driving control of the actual vehicle. The motion control domain receives the various instructions from the intelligent driving domain and outputs motor and brake torque instructions, realizing end-to-end longitudinal motion control of the entire vehicle from the cloud to the vehicle side.

2. The method according to claim 1, wherein: The vehicle longitudinal following model established in step 1 includes: The distance d(k) between the vehicle and the preceding vehicle at time k: d(k)=x p (k)-x(k) x(k), x p (k) represents the distance between the vehicle and the preceding vehicle at time k; The speed difference Δv(k) between the vehicle and the preceding vehicle at time k: Δv(k)=v p (k)-v(k) v(k), v p (k) represents the speed of the vehicle and the preceding vehicle at time k respectively; The distance error Δd(k) between the vehicle and the preceding vehicle at time k: Δd(k)=d(k)-d * (k) d*(k) represents the expected distance between the vehicle and the preceding vehicle at time k, which is determined by the selected following strategy; The relationship between the vehicle's acceleration and velocity at time k is based on the velocity model and the time step T s Sure:

3. The method according to claim 1, wherein: In step 5, the vehicle also downloads the trained reinforcement learning model, which is used to perform autonomous driving control by the single vehicle when end-to-end control fails. This includes first performing hierarchical control to generate the hierarchical control instructions, and then the motion control domain combines the instructions with the driver model running on the vehicle to output motor and brake torque instructions; When both types of automatic driving control fail, the driver takes over, and the motion control domain outputs motor and brake torque commands based on the driver's pedal signal and the driver model running on the vehicle side.

Images