Vehicle path tracking control method for deep reinforcement learning actor-evaluator network structure based on double-flow information bottleneck

Abstract

The invention discloses a deep reinforcement learning actor-evaluator network structure based on double-flow information bottleneck and a path tracking control method for a four-wheel independent steering and independent driving vehicle, and the method comprises the steps: adding a known information bottleneck module in a Critic network before the estimation of a Q value, extracting known information from the known information bottleneck module, and carrying out the estimation of a Q value; designing a training target to minimize a time difference error and minimize mutual information, and constraining a parameter norm; in the Actor network, an unknown information bottleneck module is added in front of a strategy generation module, unknown class information is extracted from the unknown information bottleneck module, and a training target is designed to maximize expected accumulated rewards, maximize mutual information and minimize conditional entropy; in the training process, states and actions in samples are input into a Critic network and a mutual information estimator network, Q value estimation and mutual information estimation are obtained, and Critic network parameters are updated through a minimization objective function; actor network parameters are updated through a strategy gradient theorem and importance sampling so as to maximize an objective function; meanwhile, a mutual information estimator network is independently trained, and maximum estimation of mutual information is achieved through sampling and optimization; in order to realize dynamic adjustment, a weight scheduler is set, coefficients in an Actor objective function and a Critic objective function are dynamically adjusted according to the current training round number and the average return, and a linear attenuation strategy is used for adjustment.

Description

Technical Field

[0001] The present invention relates to the actor-evaluator network structure in deep reinforcement learning and the path tracking control of the vehicle. The specific design uses the dual-stream information bottleneck technology to remove redundant information in the sample, extract high-dimensional features in the data, and reduce generalization errors. It alleviates the overfitting of DRL to known data caused by IB and improves the control accuracy and generalization ability of the DRL controller for vehicle path tracking. Background Art

[0002] Deep reinforcement learning (DRL) is a technique that combines deep learning and reinforcement learning to solve complex decision-making and control problems. In DRL, the actor-critic (actor-critic) network structure is a widely used algorithm architecture with important theoretical and practical significance. The structure consists of two main components: the actor network and the critic network, which work together to optimize the behavior strategy of the agent in the environment. The actor network is responsible for generating policies, that is, selecting the optimal action under a given state, and updating the parameters through the policy gradient method to maximize the expected reward accumulated in the long term; the critic network is responsible for evaluating the value of the action selected by the actor network, that is, calculating the state-action value function (Q value), and updating the parameters by minimizing the temporal difference error (TD error), thereby providing a more accurate value assessment. Traditional reinforcement learning methods face challenges in dealing with high-dimensional state spaces and continuous action spaces. With the development of deep learning, the introduction of deep neural networks into reinforcement learning has produced deep reinforcement learning methods. Among them, the actor-critic network structure has received widespread attention and application due to its advantages in dealing with continuous action spaces and improving algorithm stability. This structure improves the stability of training because the evaluator network helps to make policy updates more stable by providing value estimates. In terms of processing continuous action space, compared with the optimization of discrete action space, the actor network can generate continuous actions, and through collaboration with the evaluator network, it can achieve effective exploration and optimization of the continuous action space. In addition, the actor-evaluator network structure can more efficiently utilize the samples in the experience replay pool, and by separating value estimation and policy updating, the algorithm can converge more quickly.

[0003] However, when controlling complex high-dimensional intelligent agents such as vehicles, there will be diverse control behaviors and complex external environments. DRL needs to have a strong ability to extract high-dimensional features in interactive data, avoid overfitting of known samples, and have good generalization capabilities for control environments that have not been encountered. Summary of the invention

[0004] The present invention aims to overcome the above-mentioned shortcomings of the prior art and designs a vehicle path tracking control method based on a new actor-critic architecture with dual-stream information bottleneck (TIB). The information bottleneck (IB) is introduced into the critic network to remove redundancy and enhance the discriminative power of features by minimizing the mutual information between state features and Q function estimation, thereby improving the feature extraction and generalization error of the critic. The reverse information bottleneck (RIB) is introduced into the actor network to maximize the mutual information between state features and actions, strengthen the discriminative feature representation of unknown environmental features, and improve the generalization ability of DRL.

[0005] The vehicle path tracking control method based on the deep reinforcement learning actor-evaluator network structure with dual-stream information bottleneck has the following specific design steps:

[0006] Step 1, establishing a four-wheel steering vehicle dynamics model;

[0007] Step 2: The vehicle motion control process directly uses the MDP form:

[0008] Step 3, select TD3 algorithm to implement the DRL part;

[0009] Step 4: DRL passes the action mapped by the vehicle state to the vehicle, updates the vehicle state and passes it to DRL again.

[0010] In step 1, the vehicle dynamics model includes a vehicle body dynamics model and a tire model. v X v Y v Z v The four tires are indexed as i = fl, fr, rl, rr. In the tire coordinate system, the lateral force and steering angle of each tire are respectively represented by F and , respectively. li ,F Li ,δ i The vehicle body dynamics can be written as:

[0011]

[0012] In the formula, M and I γ represents the vehicle mass and moment of inertia, (L bi ,l bi ) represents the distance between each tire and the center of gravity of the vehicle in the coordinate system, They represent the vehicle's longitudinal acceleration, lateral acceleration and yaw angular acceleration respectively.

[0013] The vehicle acceleration in the body coordinate system is calculated by the forces applied by the tires. In the design of the path tracking controller, the longitudinal and lateral forces at each tire are determined by controlling the steering angle and wheel torque. In the tire coordinate system, the magic formula is used to represent the nonlinear characteristics of the tire model.

[0014] Then, the lateral dynamics model of the tire is established:

[0015] δ i =arctan((v y ±ω i I bi ) / (v x ±ω i L bi ))-α i (3)

[0016] Based on the above analysis, a three-degree-of-freedom vehicle dynamics model is established.

[0017] In step 2, the state of DRL is designed as where y and represent the lateral displacement and heading angle of the vehicle respectively, Indicates the current vehicle state error.

[0018] The action of the DRL controller a = [δ dfl δ dfr δ drl δ drr ], representing the control input of the steering angle of each wheel.

[0019] The reward function is designed as a positive gain parameter K r With error e ref The product of .

[0020] Among them, step 3 specifically includes: first initialize the strategy network π φ , two Q networks and and their corresponding target networks π φ‘ , two Q networks and

[0021] Then according to the current policy network π φ and status t Select action a t After executing the action, record the immediate reward r t and the new state s t+1 , and the new transfer sample (s t ,a t ,r t ,s t+1) is stored in the experience replay pool Ω.

[0022] Randomly sample a batch of samples (s) from the experience replay pool Ω t ,a t ,r t ,s t+1 ). Then use the target policy network π φ‘ Generate the next state s t+1 The target action a' is used to prevent the policy network from overfitting the action.

[0023] In the critic network, an IB module is introduced mainly to improve the generalization ability and sample efficiency of the network. By compressing the redundant information in the high-dimensional features while retaining the basic information of the Q-value estimation, the IB module enables the network to learn the state-action value function more effectively. It is hoped that the IB module can reduce the complexity of high-dimensional inputs, mitigate the risk of overfitting, extract the most relevant features, improve the accuracy of Q-value estimation, and enhance the adaptability of the model in different environments. Based on these design goals, the training loss function of the critic network is formulated as follows:

[0024]

[0025] where θ represents the parameters of the critic network, y t is the target value of TD, p(z t |s t ,a t ) is the conditional distribution defined by the IB module, r(z t ) is the prior distribution. is the standard TD error term, ensuring accurate Q-value estimation; KL(p(z t |s t ,a t ),r(z t ))The information bottleneck is achieved by minimizing the divergence between the conditional distribution and its prior distribution, thereby constraining the information flow from high-dimensional features to features and achieving feature compression and selection.

[0026] By optimizing this loss function, TD3 gradually adjusts the parameters θ1 and θ2 of the Q network to improve the accuracy of Q value estimation.

[0027] TD3 introduces a delayed policy update mechanism, that is, the policy network π is updated once every two updates of the Q network. φ The goal of the policy network is to maximize the Q value and delay the update to avoid the policy network relying on inaccurate Q values ​​too early.

[0028] Among them, the RIB module is introduced into the design of the objective function J(φ) of the policy network, mainly to improve the quality and expressiveness of the policy. Through the RIB module, the goal is to maximize the mutual information between state representation and action, save the key information required for policy generation, enhance the distinguishing ability of state representation to better distinguish the value of various actions, and improve the exploration ability of the policy to generate more diverse and effective actions. Based on these design goals, the training loss function of the actor network is formulated as follows:

[0029]

[0030] Among them, φ represents the parameters of the actor network, p(u t |s t ) is the conditional distribution defined by the RIB module, and r(u t ) is the prior distribution. is a standard policy gradient term that aims to maximize the expected cumulative return to ensure that the generated policy can bring high returns; KL(p(u t |s t ),r(u t )) realizes the reverse information bottleneck by maximizing the conditional distribution p(u t |s t ) and its prior distribution r(u t ), thereby increasing the state representation u t The mutual information between the actions retains more policy-related information.

[0031] Finally, the parameters of the target Q network and the target policy network are updated using a soft update method.

[0032] After the training is completed, the optimal action a is directly calculated through the policy network to guide the actual decision-making of the agent in the environment.

[0033] The present invention is used to improve the generalization ability and feature extraction performance of the DRL controller in a high-dimensional environment. By introducing the information bottleneck (IB) and reverse information bottleneck (RIB) modules, the redundant information in the sample is removed, the high-dimensional features are extracted, and the overfitting of DRL to known data is alleviated. The specific implementation scheme includes: in the Critic network, a known information bottleneck module (KIB) is added before the Q value estimation, from which the known class information is extracted, and the training objectives are designed to minimize the time difference (TD) error, minimize the mutual information, and constrain the parameter norm; in the Actor network, an unknown information bottleneck module (UIB) is added before the policy generation module, from which the unknown class information is extracted, and the training objectives are designed to maximize the expected cumulative reward, maximize the mutual information, and minimize the conditional entropy. During the training process, the state and action in the sample are input into the Critic network and the mutual information estimator network to obtain the Q value estimation and the mutual information estimation, and the Critic network parameters are updated by minimizing the objective function; the Actor network parameters are updated by the policy gradient theorem and importance sampling to maximize the objective function. At the same time, the mutual information estimator network is trained independently, and the maximum estimation of the mutual information is achieved through sampling and optimization. In order to achieve dynamic adjustment, a weight scheduler is set to dynamically adjust the coefficients in the Actor and Critic objective functions according to the current number of training rounds and the average return, and a linear attenuation strategy is used for adjustment. The present invention improves the Actor-Critic network structure so that the target representation contains sufficient task-related information and minimal task-irrelevant information, thereby obtaining a more accurate nonlinear feature representation, thereby reducing generalization errors and improving the control accuracy and generalization ability of the DRL controller for unknown environments.

[0034] Compared with the prior art, the present invention has the following significant advantages:

[0035] The designed actor-critic architecture can make the target representation contain enough task-related information and minimal task-irrelevant information to obtain more accurate nonlinear features and reduce generalization errors. The target representation of unknown samples that is more relevant to high-quality samples is obtained by inverting the optimization goal of standard IB. This alleviates the impact of lack of unknown data and improves the generalization ability and accuracy of the DRL controller. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a control flow diagram of the present invention.

[0037] Figure 2 Schematic diagram of the actor-evaluator network structure based on the dual-stream information bottleneck of the present invention. DETAILED DESCRIPTION

[0038] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0039] A deep reinforcement learning vehicle path tracking control method based on a dual-stream information bottleneck (TIB) actor-critic architecture uses information bottleneck (IB) to remove redundant information in samples, extract high-dimensional features in data, and reduce generalization errors. In order to alleviate the overfitting of deep reinforcement learning (DRL) to known data caused by IB, the reverse information bottleneck (RIB) reverses the optimization goal of IB, generates discriminative unknown environment feature information, and improves the generalization ability of the DRL controller. The specific implementation steps include:

[0040] Step 1: Establish a four-wheel steering vehicle dynamics model.

[0041] The vehicle dynamics model includes the body dynamics model and the tire model, such as Figure 1 OXYZ and O v X v Y v Z v The four tires are indexed as i = fl, fr, rl, rr. In the tire coordinate system, the lateral force and steering angle of each tire are respectively represented by F and , respectively. li ,F Li ,δ i The vehicle body dynamics can be written as:

[0042]

[0043] In the formula, M and I γ represents the vehicle mass and moment of inertia, (L bi ,l bi ) represents the distance between each tire and the center of gravity of the vehicle in the coordinate system, They represent the vehicle's longitudinal acceleration, lateral acceleration and yaw angular acceleration respectively.

[0044] The vehicle acceleration in the body coordinate system is calculated from the forces applied by the tires. In the design of the path tracking controller, the longitudinal and lateral forces at each tire are determined by controlling the steering angle and wheel torque, as shown in the tire dynamic model. In the tire coordinate system, the magic formula is used to represent the nonlinear characteristics of the tire model.

[0045] Then, the lateral dynamics model of the tire is established:

[0046] δ i =arctan((v y ±ω i I bi ) / (v x ±ω i Lbi ))-α i (3)

[0047] Based on the above analysis, a three-degree-of-freedom vehicle dynamics model is established.

[0048] Step 2: Establish an MDP model for motion control of the DRL vehicle.

[0049] The vehicle motion control process directly uses the MDP form:

[0050] The state of DRL is designed as where y and represent the lateral displacement and heading angle of the vehicle respectively, Indicates the current vehicle state error.

[0051] The action of the DRL controller a = [δ dfl δ dfr δ drl δ drr ], representing the control input of the steering angle of each wheel.

[0052] The reward function is designed as a positive gain parameter K r With error e ref The product of .

[0053] Step 3, establishing a DRL controller according to step 1;

[0054] Select the TD3 algorithm to implement the DRL part. First, initialize the policy network π φ , two Q networks and and their corresponding target networks π φ‘ , two Q networks and

[0055] Then according to the current strategy network π φ and status t Select action a t After executing the action, record the immediate reward r t and the new state s t+1 , and the new transfer sample (s t ,a t ,r t ,s t+1 ) is stored in the experience replay pool Ω.

[0056] Randomly sample a batch of samples (s) from the experience replay pool Ω t ,a t ,r t ,s t+1). Then use the target policy network π φ‘ Generate the next state s t+1 The target action a' is used to prevent the policy network from overfitting the action.

[0057] In the critic network, an IB module is introduced mainly to improve the generalization ability and sample efficiency of the network. By compressing the redundant information in the high-dimensional features while retaining the basic information of the Q-value estimation, the IB module enables the network to learn the state-action value function more effectively. It is hoped that the IB module can reduce the complexity of high-dimensional inputs, mitigate the risk of overfitting, extract the most relevant features, improve the accuracy of Q-value estimation, and enhance the adaptability of the model in different environments. Based on these design goals, the training loss function of the critic network is formulated as follows:

[0058]

[0059] where θ represents the parameters of the critic network, y t is the target value of TD, p(z t |s t ,a t ) is the conditional distribution defined by the IB module, r(z t ) is the prior distribution. is the standard TD error term, ensuring accurate Q-value estimation; KL(p(z t |s t ,a t ),r(z t ))The information bottleneck is achieved by minimizing the divergence between the conditional distribution and its prior distribution, thereby constraining the information flow from high-dimensional features to features and achieving feature compression and selection.

[0060] By optimizing this loss function, TD3 gradually adjusts the parameters θ1 and θ2 of the Q network to improve the accuracy of Q value estimation.

[0061] TD3 introduces a delayed policy update mechanism, that is, the policy network π is updated once every two updates of the Q network. φ The goal of the policy network is to maximize the Q value and delay the update to avoid the policy network relying on inaccurate Q values ​​too early.

[0062] Among them, the RIB module is introduced into the design of the objective function J(φ) of the policy network, mainly to improve the quality and expressiveness of the policy. Through the RIB module, the goal is to maximize the mutual information between state representation and action, save the key information required for policy generation, enhance the distinguishing ability of state representation to better distinguish the value of various actions, and improve the exploration ability of the policy to generate more diverse and effective actions. Based on these design goals, the training loss function of the actor network is formulated as follows:

[0063]

[0064] Among them, φ represents the parameters of the actor network, p(u t |s t ) is the conditional distribution defined by the RIB module, and r(u t ) is the prior distribution. is a standard policy gradient term that aims to maximize the expected cumulative return to ensure that the generated policy can bring high returns; KL(p(u t |s t ),r(u t )) realizes the reverse information bottleneck by maximizing the conditional distribution p(u t |s t ) and its prior distribution r(u t ), thereby increasing the state representation u t The mutual information between the actions retains more policy-related information.

[0065] Finally, the parameters of the target Q network and the target policy network are updated using a soft update method.

[0066] After the training is completed, the optimal action a is directly calculated through the policy network to guide the actual decision-making of the agent in the environment.

[0067] Step 4: Combine the vehicle dynamics model of step 1 and the DRL controller of step 3 through the MDP model of step 2

[0068] The action mapped by the DRL via the vehicle state is passed to the vehicle, the vehicle state is updated and passed to the DRL again.

[0069] The contents described in the implementation cases of this specification are merely descriptions of the dual-stream information bottleneck improved actor-critic network structure. The protection scope of the present invention should not be deemed to be limited to the specific forms and parameters stated in the implementation cases. The protection scope of the present invention also involves the improvement of the actor-critic network by different information bottleneck technologies.

Claims

1. A vehicle path tracking control method based on a deep reinforcement learning actor-critic architecture with dual-stream information bottleneck uses information bottleneck IB to remove redundant information from samples, extract high-dimensional features from data, and reduce generalization errors. In order to alleviate the overfitting of deep reinforcement learning DRL to known data caused by IB, the reverse information bottleneck RIB inverts the optimization goal of IB, generates discriminative unknown environment feature information, and improves the generalization ability of the DRL controller. The specific implementation steps include: Step 1, establishing a four-wheel steering vehicle dynamics model; Step 2: The vehicle motion control process directly uses the MDP form: Step 3, select TD3 algorithm to implement the DRL part; Step 4: DRL passes the action mapped by the vehicle state to the vehicle, updates the vehicle state and passes it to DRL again.

2. The vehicle path tracking control method based on the deep reinforcement learning actor-critic architecture with dual-stream information bottleneck as claimed in claim 1 is characterized in that: In step 1, the vehicle dynamics model includes the body dynamics model and the tire model; OXYZ and O v X v Y v Z v are represented as the earth inertial reference system and the vehicle body inertial reference system respectively; it is assumed that the path tracking occurs on a flat road; the yaw motion of the vehicle body is mainly controlled by the longitudinal force and lateral force on each tire; for ease of notation, the four tires are indexed as i = fl, fr, rl, rr; in the tire coordinate system, the lateral force and steering angle of each tire are respectively represented by F li ,F Li ,δ i The vehicle body dynamics can be written as: In the formula, M and I γ represents the vehicle mass and moment of inertia, (L bi ,l bi ) represents the distance between each tire and the center of gravity of the vehicle in the coordinate system, They represent the longitudinal acceleration, lateral acceleration and yaw angular acceleration of the vehicle respectively; The vehicle acceleration in the body coordinate system is calculated by the forces applied by the tires. In the design of the path tracking controller, the longitudinal and lateral forces at each tire are determined by controlling the steering angle and wheel torque. In the tire coordinate system, the magic formula is used to represent the nonlinear characteristics of the tire model. Then, the lateral dynamics model of the tire is established: δ i =arctan((v y ±ω i I bi ) / (v x ±ω i L bi ))-α i (3) Based on the above analysis, a three-degree-of-freedom vehicle dynamics model is established.

3. The vehicle path tracking control method based on the deep reinforcement learning actor-critic architecture with dual-stream information bottleneck as claimed in claim 1, characterized in that: In step 2, the state of DRL is designed as where y and represent the lateral displacement and heading angle of the vehicle respectively, Indicates the current vehicle state error; The action of the DRL controller a = [δ dfl δ dfr δ drl δ drr ], represents the control input of the steering angle of each wheel; The reward function is designed as a positive gain parameter K r With error e ref The product of .

4. The vehicle path tracking control method based on the deep reinforcement learning actor-critic architecture with dual-stream information bottleneck as claimed in claim 1, characterized in that: In step 3, first initialize the policy network π φ , two Q networks and and their corresponding target networks π φ‘ , two Q networks and Then according to the current strategy network π φ and status t Select action a t ; After executing the action, record the immediate reward r t and the new state s t+1 , and the new transfer sample (s t ,a t ,r t ,s t+1 ) is stored in the experience replay pool Ω; Randomly sample a batch of samples (s) from the experience replay pool Ω t ,a t ,r t ,s t+1 ), and then use the target policy network π φ ‘ Generate the next state s t+1 The target action a' is used to prevent the policy network from overfitting the action; In the critic network, an IB module is introduced to improve the generalization ability and sample efficiency of the network; by compressing the redundant information in the high-dimensional features while retaining the basic information of the Q-value estimation, the IB module enables the network to learn the state-action value function more effectively; it is hoped that the IB module can reduce the complexity of high-dimensional inputs, alleviate the risk of overfitting, extract the most relevant features, improve the accuracy of Q-value estimation, and enhance the adaptability of the model in different environments; based on these design goals, the training loss function of the critic network is formulated as follows: where θ represents the parameters of the critic network, y t is the target value of TD, p(z t |s t ,a t ) is the conditional distribution defined by the IB module, r(z t ) is the prior distribution; is the standard TD error term, ensuring accurate Q-value estimation; KL(p(z t |s t ,a t ),r(z t )) The information bottleneck is achieved by minimizing the divergence between the conditional distribution and its prior distribution, thereby constraining the information flow from high-dimensional features to features, achieving feature compression and selection; Through the optimization of this loss function, TD3 gradually adjusts the parameters θ1 and θ2 of the Q network to improve the accuracy of Q value estimation; TD3 introduces a delayed policy update mechanism, that is, the policy network π is updated once every two updates of the Q network. φ ; The goal of the policy network is to maximize the Q value and avoid premature reliance on inaccurate Q values ​​by delaying updates; Among them, the RIB module is introduced into the design of the objective function J(φ) of the policy network to improve the quality and expressiveness of the policy; through the RIB module, the goal is to maximize the mutual information between state representation and action, save the key information required for policy generation, enhance the distinguishing ability of state representation to better distinguish the value of various actions, and improve the exploration ability of the policy to generate more diverse and effective actions; based on these design goals, the training loss function of the actor network is formulated as follows: J(φ)=E s～Ω [Q θ1 (s,π φ (s))]+KL(p(u t |s t ),r(u t ))(7) Among them, φ represents the parameters of the actor network, p(u t |s t ) is the conditional distribution defined by the RIB module, and r(u t ) is the prior distribution; E s～Ω [Q θ1 (s,π φ (s))] is the standard policy gradient term, which aims to maximize the expected cumulative return to ensure that the generated policy can bring high returns; KL(p(u t |s t ),r(u t )) realizes the reverse information bottleneck by maximizing the conditional distribution p(u t |s t ) and its prior distribution r(u t ), thereby increasing the state representation u t The mutual information between actions retains more information related to the strategy; Finally, the parameters of the target Q network and the target policy network are updated using the soft update method; After the training is completed, the optimal action a is directly calculated through the policy network to guide the actual decision-making of the agent in the environment.

Images