Real-time longitudinal car following control method based on A-C reinforcement learning technology

Abstract

The invention discloses a real-time longitudinal vehicle following control method based on an A-C reinforcement learning technology, and belongs to the technical field of intelligent vehicle control. The invention aims to provide a real-time longitudinal vehicle following control method based on an A-C reinforcement learning technology, which can simultaneously estimate a value function and a strategy gradient based on an A-C neural network framework, is generally used for solving the problems in a continuous action space and a high-latitude state space, and is more suitable for vehicle longitudinal driving control in a dynamic scene. According to the method, firstly, reinforcement learning is carried out on an A-C framework, then a longitudinal car following control learning system is designed, and finally, A-C reinforcement learning neural network parameter updating is carried out. The method is suitable for an A-C reinforcement learning vehicle following framework of a future vehicle intelligent control technology, a simple reinforcement learning neural network structure and a parameter updating method based on a backstepping method, the control strategy is simple, the universality is high, and industrial practice is easy.

Description

Technical Field

[0001] The invention belongs to the technical field of intelligent vehicle control. Background Art

[0002] In addition to ensuring driving safety, the vehicle longitudinal driving assistance system can also appropriately free the driver's feet, reduce driving fatigue, and improve the convenience and comfort of driving. Traditional longitudinal driving control methods mainly come from engineering calibration, mechanism modeling or model predictive control (MPC) methods. Among them, the most widely used is PID control, which has the advantages of convenient parameter adjustment and easy implementation. Some scholars determined the fuzzy rules of control variables based on engineering experience and proposed a fuzzy-PID composite control method. Some scholars designed a real-time dynamic programming (DP) speed trajectory tracking controller under the framework of MPC to achieve longitudinal control of autonomous driving trucks. However, these control methods are developed based on experimental calibration and scenario verification environments, so it is necessary to establish accurate models, calibrate controller parameters through a large number of scenarios, and verify the control effect. The development cycle of these cumbersome control methods will be very long, and there is a serious mismatch problem compared with the current vehicle replacement cycle. In addition, most of these control methods will be frozen after the calibration development is completed and will not be iteratively updated. As the vehicle age increases, the control effect will gradually decline. As vehicles become more intelligent, the longitudinal driving system of vehicles in dynamic scenarios needs to have adaptive learning capabilities. Summary of the invention

[0003] The purpose of the present invention is to simultaneously estimate the value function and the policy gradient based on the AC neural network framework, which is usually used to solve problems in continuous action space and high-dimensional state space. It is also more suitable for real-time longitudinal following control method based on AC reinforcement learning technology for longitudinal driving control of vehicles in dynamic scenarios.

[0004] The steps of the present invention are: S1. Reinforcement Learning AC Framework: Reinforcement learning is divided into two parts: strategy evaluation and action exploration: S11. Gradient descent learning for updating the referee value network parameters: The estimated expression of the value network at the current moment is: y t =r t +γ·V(s t+1 ,ω t ), (1) Among them, r t Indicates that in state, s t Next, perform action a t The reward obtained, γ represents the discount rate, V(s t+1 ,ωt ) indicates that in the successor state s t+1 The estimated value of the downvalue function, ω t is the network parameter; The update formula of the error signal is expressed as: δ t =r t +γV(s t+1 ,ω t )-V(s t ,ω t ). (2) ω' t The update iteration formula is expressed as: Among them, β represents the learning rate, Represents the parameter ω t Gradient of change; S12. Gradient ascent learning for updating the parameters of the athlete strategy network: The player strategy network is represented as: Among them, a t Indicates action, s t represents the state, σ(s t ) indicates that t The associated normal distribution deviation, μ(s t θ t ) represents the probability distribution of all actions output by the policy network, θ t is the network parameter, π(a t |s t θ t ) is the strategy network; θ' t The update iteration formula is expressed as: Among them, α represents the learning rate, It means that with the parameter θ t Gradient of change; S2, Longitudinal Following Control Learning System: S21. Longitudinal car following Markov process design: S211, State Design: Where TH = Δx / v h , ΔTH=TH tar -TH; S212, Action Design: Torque Requirement T of Wheels w is a control variable; S213, Reward Design: Among them, Δx is the relative distance at the previous moment; d 0 is the safe distance; l =(v l (k-1)+v l (k)) / 2·ΔT is the displacement of the preceding vehicle within ΔT time, k is the time step; ΔT = Δt·n, Δt is the control period of the system, n is the control parameter; TH tar The expected following distance at the current moment; The reward function is expressed as: Where Δu represents the change in acceleration, Δx cra Indicates the collision distance, TH max Indicates the maximum headway; S3. AC reinforcement learning neural network parameter update method: S31. Design a four-layer neural network structure: The network parameters are divided into three parts, namely ω t =[ω t1 ω t2 ω t3 ], its activation function is expressed as: f(x)=sigmoid(x). (9) Add the scalar 1 to the matrix s t Connect to get the matrix s' t , Let x 1 =ω t1 ·s' t +b 1 ,ω t1 is the weight connecting the input layer and the first hidden layer, b i (i=1,2,3,4,5,6) is the bias, H is the output of the first hidden layer, It is expressed as: H=f(x 1 ). (10) Concatenate scalar 1 with matrix H to get matrix H', Let x 2 =ω t2 ·H'+b 2 ,ω t2 is the weight connecting the first hidden layer and the second hidden layer, I is the output of the first hidden layer, It is expressed as: I=f(x 2 ). (11) Let x 3 =ω t3 Τ I+b 3 ,ω t3 is the weight connecting the second hidden layer and the output layer, V(s t ,ω t3 ) is the output of the output layer, expressed as: V(s t ,ω t3 )=f(x 3 ), (12) Among them, V(s t ,ω t3 ) is abbreviated as V t ; The loss function is expressed as: S32. Value network parameter updating method: For the weights between the output layer and the second hidden layer, the partial derivative is expressed as: Let η t =(V t -y t )V t (1-V t ), then the above formula can be rewritten as: The partial derivative of the weights between the first hidden layer and the second hidden layer is as follows: The partial derivative of the weights between the input layer and the first hidden layer is as follows: The parameter update formula of the value network is: S33, action network parameter update method: The network parameters between each layer of the action network are described as θ t =[θ t1 θ t2 θ t3 ]; let z 1 =θ t1 ·s' t +b 4 ,θ t1 is the weight connecting the input layer and the first hidden layer, E is the output of the first hidden layer, It is expressed as: E=f(z 1). (19) Concatenate scalar 1 with matrix E to get matrix E', Let z 2 =θ t2 ·E'+b 5 ,θ t2 is the weight connecting the first hidden layer and the second hidden layer, F is the output of the first hidden layer, It is expressed as: F=f(z 2 ). (20) make θ t3 is the weight connecting the second hidden layer and the output layer, It is expressed as: G=f(z 3 ). (twenty one) The partial derivative of the weight vector connecting the output layer and the second hidden layer is expressed as: Order t =(1-G), then the above formula can be rewritten as: The partial derivative of the weight vector connecting the second hidden layer and the first hidden layer is expressed as: The partial derivative of the weight vector connecting the first hidden layer and the input layer is expressed as: The parameter update formula of the action network is:

[0005] The present invention is suitable for the AC reinforcement learning following framework of future vehicle intelligent control technology, a simple reinforcement learning neural network structure and a parameter updating method based on backstepping method, with simple control strategy, strong versatility and easy industrial practice. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 It is a longitudinal following control reinforcement learning system; Figure 2 This is a curve chart under NEDC working conditions, where the working conditions from top to bottom are vehicle speed, acceleration, headway, engine torque, and wheel torque; Figure 3 This is a curve chart under WLTC working conditions, where the working conditions from top to bottom are vehicle speed, acceleration, headway, engine torque, and wheel torque. DETAILED DESCRIPTION

[0007] In order to meet the self-learning needs of higher-level intelligent driving systems, vehicle longitudinal driving technology integrated with intelligent learning has become a research hotspot. These methods make adaptive strategy adjustments for changing driving environments (such as changes in road slope, changes in driving characteristics of the preceding vehicle, etc.), which can well compensate for problems such as model inaccuracy, environmental changes, and adaptability to working conditions. As a typical intelligent learning method, reinforcement learning (RL) technology has received extensive attention in the field of vehicle longitudinal driving control research. In response to different decision-making and control problems, RL methods can be divided into tabular, Monte Carlo, temporal difference, and (Actor-Critic, AC) methods. In order to overcome the challenges of modeling difficulties and reduced computational efficiency faced by the current ACC method based on optimization control, some scholars have proposed a RL adaptive cruise control (SFRL-ACC) method under safe conditions by utilizing the model-free characteristics and high real-time reasoning efficiency of RL. Based on the TORCS simulation software, some scholars have designed a joint training method based on RL technology, including using neural networks to fit actions and state spaces, setting training strategies and reward functions, and other mechanisms. Compared with the above RL technology, the AC neural network framework can estimate the value function and policy gradient at the same time. It is usually used to solve problems in continuous action space and high-dimensional state space, and is more suitable for longitudinal driving control of vehicles in dynamic scenes. Some scholars have proposed an AC method with asynchronous generalized advantages, conducted decision simulation on the CARLA autonomous driving simulation platform, and verified good autonomous driving control effects. However, the generalization ability of the vehicle control method based on RL technology for dynamic scenes needs to be further improved, and it is also a challenge to achieve multiple goals such as energy saving, following the vehicle, and comfort at the same time.

[0008] The present invention proposes a real-time longitudinal following control method based on AC reinforcement learning technology, including the following steps: 1. Reinforcement learning algorithm under AC framework, the specific design steps are as follows: Firstly, a reinforcement learning AC framework is proposed. Next, a gradient descent learning algorithm for updating the parameters of the referee's value network is designed. Finally, a gradient ascent learning algorithm for updating the parameters of the player's strategy network is proposed.

[0009] 1) Reinforcement Learning AC Framework like Figure 1 As shown in the figure, it is a schematic diagram of the AC framework. The proposed reinforcement learning (RL) algorithm is divided into two parts: strategy evaluation and action exploration.

[0010] 2) Gradient descent learning algorithm for updating the referee value network parameters; Value Network V(s t ,ω t ) is used to evaluate the value of the review network, where ωt is the network parameter. The estimated expression of the value network at the current moment is: y t =r t +γ·V(s t+1 ,ω t ), (1) Among them, r t Indicates that in state, s t Next, perform action a t The reward obtained, γ represents the discount rate, V(s t+1 ,ω t ) indicates that in the successor state s t+1 An estimate of the downvalue function.

[0011] In summary, the Bellman equation is used to update the policy parameters, and its derivation process requires an error signal. This error signal is calculated using the time difference algorithm (TD), which includes the prediction error. In the TD method, the update formula of the error signal is expressed as: δ t =r t +γV(s t+1 ,ω t )-V(s t ,ω t ). (2)

[0012] Calculate ω' using the gradient descent algorithm t The update iteration formula can be expressed as Among them, β represents the learning rate, Represents the parameter ω t Gradient of change.

[0013] 3) Gradient ascent learning algorithm for updating the parameters of the player strategy network In the athlete strategy network, the strategy network π(a t |s t θ t ) is used to approximate the policy function, where θ t is the network parameter. The action selection in the player strategy network follows a Gaussian distribution. This method can effectively integrate strategy search and strategy utilization. Its formula can be expressed as Among them, a t Indicates action, s t represents the state, σ(s t ) indicates that t The associated normal distribution deviation, μ(s t θ t ) represents the probability distribution of all actions output by the policy network.

[0014] Calculate θ' using the gradient ascent algorithm t The update iteration formula can be expressed as Among them, α represents the learning rate, It means that with the parameter θ t Gradient of change.

[0015] 2. The longitudinal following control learning system of the present invention is implemented by a Markov longitudinal following process, which is as follows: 1) Longitudinal vehicle following reinforcement learning control system in a joint simulation environment Carsim software is widely used in automotive control research as a high-fidelity automotive equipment model. Matlab / Simulink software is suitable for designing control systems. Therefore, a joint simulation platform for learning systems was established using Carsim and Matlab / Simulink. A car following environment was simulated in Carsim, and an RL car following control system was established in Simulink.

[0016] 2) Design of longitudinal car-following Markov process; The Markov decision process is used to describe the car following activity. The following scenario is a multi-car environment, S is the state of the leading car and the main car, A is the action choice of the main car, and r is the instantaneous reward. The following content will introduce the Markov decision process of following control from three aspects: state design, action design, and reward design.

[0017] I. State Design: The characteristics of the current environment are described by the following state vectors, namely, the main vehicle speed, relative distance, relative speed, front vehicle acceleration, headway and time headway difference Where TH = Δx / v h , ΔTH=TH tar -TH.

[0018] II. Action Design: Wheel Torque Requirement T w is the control variable.

[0019] III. Reward design: According to the state and evaluation index of the car following system, the equivalent expected speed of this paper is described as follows Among them, Δx is the relative distance at the previous moment; d 0 is a safe distance; l =(v l (k-1)+v l(k)) / 2·ΔT is the displacement of the preceding vehicle within ΔT time, k is the time step; ΔT = Δt·n, Δt is the control period of the system, n is the control parameter; TH tar is the expected following distance at the current moment.

[0020] At the same time, fuel consumption and acceleration terms are introduced into the reward function to optimize the fuel consumption and comfort of the main vehicle during the following process. Therefore, the reward function is expressed as Where Δu represents the change in acceleration, Δx cra Indicates the collision distance, TH max Indicates the maximum headway.

[0021] 3. The AC reinforcement learning neural network parameter updating method of the present invention has the following specific design steps: First, a four-layer neural network structure was designed, and then the neural network parameters were updated using the reverse gradient algorithm under the Bellman theory framework.

[0022] 1) Four-layer neural network structure Taking the value network as an example, a neural network consisting of an input layer, two hidden layers and an output layer is designed. According to the four-layer network structure, the network parameters are divided into three parts, namely ω t =[ω t1 ω t2 ω t3 ].

[0023] Its activation function can be expressed as: f(x)=sigmoid(x). (9)

[0024] Input state matrix s t As shown in formula (6), the scalar 1 and the matrix s t Connect to get the matrix s' t , Let x 1 =ω t1 ·s' t +b 1 ,ω t1 is the weight connecting the input layer and the first hidden layer, b i (i=1,2,3,4,5,6) is the bias. H is the output of the first hidden layer, It is expressed as: H=f(x 1 ). (10)

[0025] Concatenate scalar 1 with matrix H to get matrix H', Let x 2 =ω t2·H'+b 2 ,ω t2 is the weight connecting the first hidden layer and the second hidden layer, I is the output of the first hidden layer, It is expressed as: I=f(x 2 ). (11)

[0026] Let x 3 =ω t3 Τ I+b 3 ,ω t3 is the weight connecting the second hidden layer and the output layer, V(s t ,ω t3 ) is the output of the output layer, expressed as: V(s t ,ω t3 )=f(x 3 ), (12) Among them, V(s t ,ω t3 ) can be abbreviated as V t .

[0027] The loss function can be expressed as:

[0028] 2) Value network parameter update method The back-propagation method is used to update the weights of the neural network parameters. The chain rule is used to calculate the partial derivatives of the errors of each weight. For the weights between the output layer and the second hidden layer, the partial derivative can be expressed as:

[0029] Let η t =(V t -y t )V t (1-V t ), the above formula can be rewritten as:

[0030] The partial derivative of the weights between the first hidden layer and the second hidden layer is as follows:

[0031] The partial derivative of the weights between the input layer and the first hidden layer is as follows:

[0032] The parameter update formula of the value network is:

[0033] 3) Action network parameter update method The action network consists of four layers: an input layer, two hidden layers, and an output layer. Therefore, the network parameters between each layer can be described as θ t =[θ t1 θ t2 θ t3 ].

[0034] Let z 1 =θ t1 ·s' t +b 4 ,θ t1 is the weight connecting the input layer and the first hidden layer, E is the output of the first hidden layer, It is expressed as: E=f(z 1 ). (19)

[0035] Concatenate scalar 1 with matrix E to get matrix E', Let z 2 =θ t2 ·E'+b 5 ,θ t2 is the weight connecting the first hidden layer and the second hidden layer, F is the output of the first hidden layer, It is expressed as: F=f(z 2 ). (20)

[0036] make θ t3 is the weight connecting the second hidden layer and the output layer, It is expressed as: G=f(z 3 ). (twenty one)

[0037] The partial derivative of the weight vector connecting the output layer and the second hidden layer can be expressed as:

[0038] Order t =(1-G). Then the above formula can be rewritten as:

[0039] The partial derivative of the weight vector connecting the second hidden layer and the first hidden layer can be expressed as:

[0040] The partial derivative of the weight vector connecting the first hidden layer and the input layer can be expressed as:

[0041] The parameter update formula of the action network is:

[0042] Simulation experiment In order to verify the robustness and anti-interference ability of the longitudinal vehicle following reinforcement learning controller proposed in this invention, two sets of common urban transient conditions were used for testing. Figure 2 As shown, under the working condition nedc, the traditional PID method is compared with the method proposed in the present invention. The average speed tracking error value of the traditional PID method is 0.41m / s, and the average speed tracking error value of the reinforcement learning method proposed in the present invention is 0.51m / s; the average fuel volume of the traditional PID control method is 15.64Nm, and the average fuel volume of the method proposed in the present invention is 15.66Nm; the average acceleration increment of the traditional PID control method is 0.26m / s 2 The average acceleration increment of the method proposed in this invention is 0.22m / s 2 .like Figure 3 As shown in the figure, under the working condition wltc, the average speed tracking error value of the traditional PID method is 0.48m / s, and the average speed tracking error value of the reinforcement learning method proposed in the present invention is 0.61m / s; the average fuel volume of the traditional PID control method is 34.27Nm, and the average fuel volume of the method proposed in the present invention is 34.28Nm; the average acceleration increment of the traditional PID control method is 0.34m / s 2 The average acceleration increment of the method proposed in this invention is 0.31m / s 2 .

[0043] The test results of vehicle speed target transient operating condition tracking show that the traditional PID controller is only applicable to specific parameter adjustment conditions, and has poor robustness and anti-interference ability for changing and complex urban conditions. The method proposed in this invention can not only achieve the same vehicle following performance as the PID method, but also meet multiple goals such as energy saving and comfort at the same time, and has a strong generalization ability for changes in operating conditions.

Claims

1. A real-time longitudinal following control method based on AC reinforcement learning technology, characterized in that: The steps are: S1. Reinforcement Learning AC Framework: Reinforcement learning is divided into two parts: strategy evaluation and action exploration: S11. Gradient descent learning for updating the referee value network parameters: The estimated expression of the value network at the current moment is: y t =r t +γ·V(s t+1 ,oh t ), (1) Among them, r t Indicates that in state, s t Next, perform action a t The reward obtained, γ represents the discount rate, V(s t+1 ,ω t ) indicates that in the successor state s t+1 The estimated value of the downvalue function, ω t is the network parameter; The update formula of the error signal is expressed as: δ t =r t +γV(s t+1 ,ω t )-V(s t ,ω t ). (2) ω′ t The update iteration formula is expressed as: oh' t =ω t -b·d t t▽ ω V(s t ,oh t ), (3) Among them, β represents the learning rate, ▽ ω Represents the parameter ω t Gradient of change; S12. Gradient ascent learning for updating the parameters of the athlete strategy network: The player strategy network is represented as: Among them, a t Indicates action, s t represents the state, σ(s t ) indicates that t The associated normal distribution deviation, μ(s t θ t ) represents the probability distribution of all actions output by the policy network, θ t is the network parameter, π(a t |s t θ t ) is the strategy network; θ′ t The update iteration formula is expressed as: θ′ t =θ t +a·d t ·▽ θ logπ(a t |s t ;θ t ), (5) Among them, α represents the learning rate, ▽ θ It means that with the parameter θ t Gradient of change; S2, Longitudinal Following Control Learning System: S21. Longitudinal car following Markov process design: S211, State Design: where TH = Δx / v h , ΔTH = TH tar - TH; S212, Action Design: Wheel Torque Requirement T w is a control variable; S213, Reward Design: Among them, Δx is the relative distance at the previous moment; d0 is the safety distance; s l =(v l (k-1)+v l (k)) / 2·ΔT is the displacement of the preceding vehicle within ΔT time, k is the time step; ΔT = Δt·n, Δt is the control period of the system, n is the control parameter; TH tar The expected following distance at the current moment; The reward function is expressed as: Where Δu represents the change in acceleration, Δx cra Indicates the collision distance, TH max Indicates the maximum headway; S3. AC reinforcement learning neural network parameter update method: S31. Design a four-layer neural network structure: The network parameters are divided into three parts, namely ω t =[ω t1 ω t2 ω t3 ], its activation function is expressed as: f(x)=sigmoid(x). (9) Add the scalar 1 to the matrix s t Connect to get the matrix s′ t , Let x1 = ω t1 ·s′ t +b1,ω t1 is the weight connecting the input layer and the first hidden layer, b i (i=1,2,3,4,5,6) is the bias, H is the output of the first hidden layer, It is expressed as: H=f(x1). (10) Concatenate scalar 1 with matrix H to get matrix H', Let x2 = ω t2 ·H'+b2,ω t2 is the weight connecting the first hidden layer and the second hidden layer, I is the output of the first hidden layer, It is expressed as: I=f(x2). (11) Let x3 = ω t3 Τ I+b3,ω t3 is the weight connecting the second hidden layer and the output layer, V(s t ,ω t3 ) is the output of the output layer, expressed as: V(s t ,ω t3 )=f(x3), (12) Among them, V(s t ,ω t3 ) is abbreviated as V t ; The loss function is expressed as: S32. Value network parameter updating method: For the weights between the output layer and the second hidden layer, the partial derivative is expressed as: Let η t =(V t -y t )V t (1-V t ), then the above formula can be rewritten as: The partial derivative of the weights between the first hidden layer and the second hidden layer is as follows: The partial derivative of the weights between the input layer and the first hidden layer is as follows: The parameter update formula of the value network is: S33, action network parameter update method: The network parameters between each layer of the action network are described as θ t =[θ t1 θ t2 θ t3 ]; let z1 = θ t1 ·s′ t +b4,θ t1 is the weight connecting the input layer and the first hidden layer, E is the output of the first hidden layer, It is expressed as: E=f(z1). (19) Concatenate scalar 1 with matrix E to get matrix E', Let z2 = θ t2 ·E'+b5,θ t2 is the weight connecting the first hidden layer and the second hidden layer, F is the output of the first hidden layer, It is expressed as: F=f(z2). (20) make θ t3 is the weight connecting the second hidden layer and the output layer, It is expressed as: G=f(z3). (21) The partial derivative of the weight vector connecting the output layer and the second hidden layer is expressed as: Order t =(1-G), then the above formula can be rewritten as: The partial derivative of the weight vector connecting the second hidden layer and the first hidden layer is expressed as: The partial derivative of the weight vector connecting the first hidden layer and the input layer is expressed as: The parameter update formula of the action network is:

Images