Vehicle path tracking control method based on double-flow information bottleneck deep reinforcement learning actor-critic network structure

Abstract

The application discloses a deep reinforcement learning actor-critic network structure based on a double-flow information bottleneck and a path tracking control method for a four-wheel independent steering and independent driving vehicle, and the method comprises the following steps: in the Critic network, a known information bottleneck module is added before Q value estimation, known class information is extracted from the known information bottleneck module, and a training target is designed to minimize a time difference error, minimize mutual information and constrain a parameter norm; in the Actor network, an unknown information bottleneck module is added before a strategy generation module, unknown class information is extracted from the unknown information bottleneck module, and a training target is designed to maximize an expected cumulative reward, maximize mutual information and minimize a conditional entropy; in the training process, a state and an action in a sample are input into the Critic network and a mutual information estimator network to obtain Q value estimation and mutual information estimation, and the Critic network parameters are updated by minimizing a target function; the Actor network parameters are updated by using a policy gradient theorem and importance sampling to maximize a target function; meanwhile, the mutual information estimator network is independently trained, and sampling and optimization are used to realize the maximum estimation of mutual information; in order to realize dynamic adjustment, a weight scheduler is arranged, coefficients in the Actor and Critic target functions are dynamically adjusted according to a current training round number and an average return, and a linear decay strategy is used for adjustment.

Description

TECHNICAL FIELD

[0001] The present application relates to an actor-critic network structure in deep reinforcement learning and path tracking control of a vehicle, specifically designed to utilize a dual-flow information bottleneck technique to remove redundant information in samples, extract high-dimensional features in data, and reduce generalization errors. This alleviates the overfitting of DRL to known data caused by IB, and improves the control accuracy and generalization ability of the DRL controller for vehicle path tracking. BACKGROUND

[0002] Deep reinforcement learning (DRL) is a technique that combines deep learning and reinforcement learning to solve complex decision-making and control problems. In DRL, the actor-critic network structure is a widely used algorithm architecture with important theoretical and practical significance. This structure consists of two main components: the actor network and the critic network, which work together to optimize the agent's behavior strategy in the environment. The actor network is responsible for generating a policy, i.e., selecting the optimal action in a given state, and updating the parameters through a policy gradient method to maximize the long-term cumulative expected reward. The critic network is responsible for evaluating the value of the action selected by the actor network, i.e., calculating the state-action value function (Q-value), and updating the parameters by minimizing the temporal difference error (TD error) to provide a more accurate value estimate. Traditional reinforcement learning methods face challenges in handling high-dimensional state spaces and continuous action spaces. With the development of deep learning, the introduction of deep neural networks into reinforcement learning has given rise to deep reinforcement learning methods, among which the actor-critic network structure has received widespread attention and application due to its advantages in handling continuous action spaces and improving algorithm stability. This structure improves the stability of training, as the critic network provides value estimates to help the policy update more stably. In terms of handling continuous action spaces, the actor network can generate continuous actions, which, through collaboration with the critic network, enables effective exploration and optimization of continuous action spaces. In addition, the actor-critic network structure can more efficiently utilize samples in the experience replay pool, allowing the algorithm to converge more quickly through the separation of value estimation and policy update.

[0003] However, when controlling complex high-dimensional agents similar to vehicles, there are diverse control behaviors and complex external environments. DRL needs to have strong feature extraction capabilities for high-dimensional features in interaction data and avoid overfitting to known samples, and have good generalization ability for unencountered control environments. SUMMARY

[0004] The present application overcomes the above-mentioned shortcomings of the prior art and designs a vehicle path tracking control method based on a new actor-critic architecture of a double-flow information bottleneck (TIB). An information bottleneck (IB) is introduced into the critic network to remove redundancy and enhance the discriminability of features by minimizing the mutual information between state features and Q function estimates, improving feature extraction and generalization errors of the critic. A reverse information bottleneck (RIB) is introduced into the actor network to maximize the mutual information between state features and actions, strengthen the discriminative feature representation of unknown environmental features, and improve the generalization ability of DRL.

[0005] The vehicle path tracking control method based on the deep reinforcement learning actor-critic network structure of the double-flow information bottleneck includes the following specific design steps:

[0006] Step 1, establish a four-wheel steering vehicle dynamics model;

[0007] Step 2, the motion control process of the vehicle directly uses the MDP form:

[0008] Step 3, select the TD3 algorithm to implement the DRL part;

[0009] Step 4, the DRL transmits the action mapped by the vehicle state to the vehicle, updates the vehicle state and transmits it to the DRL again.

[0010] In step 1, the vehicle dynamics model includes a vehicle body dynamics model and a tire model. OXYZ and O v X v Y v Z v respectively represent the earth inertial reference frame and the body inertial reference frame. It is assumed that path tracking occurs on a flat road. The yaw motion of the vehicle body is mainly controlled by the longitudinal and lateral forces on each tire. For ease of labeling, the four tires are indexed as i = fl, fr, rl, rr. In the tire coordinate system, the lateral force and steering angle of each tire are represented by F li ,F Li ,δ i . The vehicle body dynamics can be written as:

[0011]

[0012] where M and I γ represent the vehicle mass and moment of inertia, (L bi ,l bi ) represents the distance of each tire from the center of gravity of the vehicle in the coordinate system, represents the longitudinal acceleration, lateral acceleration and yaw angular acceleration of the vehicle, respectively.

[0013] The vehicle acceleration in the body coordinate system is calculated by the forces applied by the tires. In the design of the path following controller, the longitudinal and lateral forces at each tire are determined by controlling the steering angle and the wheel torque. In the tire coordinate system, the magic formula is used to represent the nonlinear characteristics of the tire model.

[0014] Then, the lateral dynamics model of the tire is established:

[0015] δ i = arctan((v y ± ω i I bi ) / (v x ± ω i L bi ))- α i (3)

[0016] Based on the above analysis, a three-degree-of-freedom vehicle dynamics model is established.

[0017] wherein in step 2, the state of the DRL is designed as where y and represent the lateral displacement and the heading angle of the vehicle, respectively, represents the current vehicle state error.

[0018] The action a = [δ dfl δ dfr δ drl δ drr ] of the DRL controller represents the control input of the steering angle of each wheel.

[0019] The reward function is designed as the product of a positive gain parameter K r and the error e ref .

[0020] wherein step 3 specifically comprises: first initializing the policy network π φ , two Q networks and and their corresponding target networks π φ‘ , two Q networks and

[0021] Then, according to the current policy network π φ and the state s t , the action a t is selected. After the action is executed, the immediate reward r t and the new state s t+1 are recorded, and the new transition sample (s t , a t , r t , s t+1) into the experience replay pool Ω.

[0022] A batch of samples (s t ,a t ,r t ,s t+1 ) is randomly sampled from the experience replay pool Ω. φ‘ Then the target policy network π t+1 is used to generate the target action a' for the next state s t , preventing the policy network from overfitting to the action.

[0023] In the critic network, an IB module is introduced, mainly to improve the generalization ability and sample efficiency of the network. By compressing the redundant information in the high-dimensional features while retaining the basic information of the Q value estimate, the IB module enables the network to learn the state-action value function more effectively. It is hoped that the IB module can reduce the complexity of high-dimensional input, mitigate the risk of overfitting, extract the most relevant features, improve the accuracy of Q value estimation, and enhance the adaptability of the model in different environments. Based on these design goals, the training loss function of the critic network is formulated as follows:

[0024]

[0025] Where θ represents the parameters of the critic network, y t is the target value of TD, p(z t |s t ,a t ) is the conditional distribution defined by the IB module, and r(z t ) is the prior distribution. is the standard TD error term, which ensures accurate Q value estimation; KL(p(z t |s t ,a t ), r(z t )) achieves information bottlenecking by minimizing the divergence between the conditional distribution and its prior distribution, thereby constraining the information flow from high-dimensional features to features, achieving feature compression and selection.

[0026] Through the optimization of this loss function, TD3 gradually adjusts the parameters θ1 and θ2 of the Q network to improve the accuracy of Q value estimation.

[0027] TD3 introduces a delayed policy update mechanism, that is, the policy network π φ is updated only after updating the Q network twice. The goal of the policy network is to maximize the Q value, and by delaying the update, the policy network avoids relying on inaccurate Q values too early.

[0028] Wherein, the design of the objective function J(φ) of the policy network introduces the RIB module, mainly to improve the quality and expressiveness of the policy. Through the RIB module, the objective is to maximize the mutual information between the state representation and the action, preserve the key information needed for policy generation, enhance the distinguishing ability of the state representation to better distinguish the value of various actions, and improve the exploration ability of the policy to generate more diverse and effective actions. Based on these design objectives, the training loss function of the actor network is formulated as follows:

[0029]

[0030] Wherein, φ represents the parameters of the actor network, p(u t |s t ) is the conditional distribution defined by the RIB module, and r(u t ) is the prior distribution. is the standard policy gradient term, aiming to maximize the expected cumulative return to ensure that the generated policy can bring high returns; KL(p(u t |s t ), r(u t )) realizes the reverse information bottleneck, by maximizing the KL divergence between the conditional distribution p(u t |s t ) and its prior distribution r(u t ), thereby increasing the mutual information between the state representation u t and the action, and preserving more information related to the policy.

[0031] Finally, the soft update method is used to update the parameters of the target Q network and the target policy network.

[0032] After training is completed, the optimal action a is directly calculated by the policy network, which is used to guide the actual decision-making of the agent in the environment.

[0033] The application is used to improve the generalization ability and feature extraction performance of the DRL controller in a high-dimensional environment. By introducing the information bottleneck (IB) and reverse information bottleneck (RIB) modules, the redundant information in the sample is removed, the high-dimensional features are extracted, and the overfitting of DRL to known data is alleviated. The specific embodiments include: in the Critic network, a known information bottleneck module (KIB) is added before the Q value estimation, the known class information is extracted therefrom, and the training target is designed to minimize the time difference (TD) error, minimize the mutual information, and constrain the parameter norm; in the Actor network, an unknown information bottleneck module (UIB) is added before the policy generation module, the unknown class information is extracted therefrom, and the training target is designed to maximize the expected cumulative reward, maximize the mutual information, and minimize the conditional entropy. In the training process, the state and action in the sample are input into the Critic network and the mutual information estimator network to obtain the Q value estimation and mutual information estimation, and the Critic network parameters are updated by minimizing the objective function; the Actor network parameters are updated by the policy gradient theorem and importance sampling to maximize the objective function. At the same time, the mutual information estimator network is independently trained, and the maximization estimation of the mutual information is realized by sampling and optimization. In order to realize dynamic adjustment, a weight scheduler is set, the coefficients in the Actor and Critic objective functions are dynamically adjusted according to the current training round and the average return, and a linear decay strategy is used for adjustment. The application improves the Actor-Critic network structure, so that the target representation contains sufficient task-related information and minimal task-independent information, more accurate nonlinear feature representation is obtained, generalization errors are reduced, and the control accuracy and generalization ability of the DRL controller for unknown environments are improved.

[0034] Compared with the prior art, the significant advantages of the application are:

[0035] The designed actor-critic architecture can make the target representation contain sufficient task-related information and minimal task-independent information to obtain more accurate nonlinear features and reduce generalization errors. The optimization target of the standard IB is reversed to obtain the target representation of the unknown sample more related to the high-quality sample. The influence of the lack of unknown data is alleviated, and the generalization ability and accuracy of the DRL controller are improved. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 is a control flow diagram of the application.

[0037] Figure 2 is a schematic diagram of the actor-evaluator network structure based on the double-flow information bottleneck of the application. DETAILED DESCRIPTION

[0038] The technical solutions of the application will be further described below with reference to the drawings.

[0039] A deep reinforcement learning vehicle path tracking control method based on a two-stream information bottleneck (TIB) actor-critic architecture uses an information bottleneck (IB) to remove redundant information in samples, extract high-dimensional features in data, and reduce generalization errors. To alleviate the overfitting of deep reinforcement learning (DRL) to known data caused by IB, a reverse information bottleneck (RIB) reverses the optimization objective of IB to generate discriminative unknown environment feature information and improve the generalization ability of the DRL controller. The specific implementation steps include:

[0040] Step 1, a four-wheel steering vehicle dynamics model is established.

[0041] The vehicle dynamics model includes a vehicle body dynamics model and a tire model, as shown in Figure 1 OXYZ and O v X v Y v Z v respectively represent the earth inertial reference frame and the vehicle body inertial reference frame. It is assumed that the path tracking occurs on a flat road. The yaw motion of the vehicle body is mainly controlled by the longitudinal force and the lateral force on each tire. For ease of labeling, the four tires are indexed as i = fl, fr, rl, rr. In the tire coordinate system, the lateral force and the steering angle of each tire are represented by F li ,F Li ,δ i . The vehicle body dynamics can be written as:

[0042]

[0043] In the formula, M and I γ represent the vehicle mass and the moment of inertia, (L bi ,l bi ) represents the distance of each tire from the center of gravity of the vehicle in the coordinate system, respectively represent the longitudinal acceleration, the lateral acceleration, and the yaw angle acceleration of the vehicle.

[0044] The vehicle acceleration in the body coordinate system is calculated by the forces applied by the tires. In the design of the path tracking controller, the longitudinal force and the lateral force at each tire are determined by controlling the steering angle and the wheel torque, as shown in the tire dynamic model. In the tire coordinate system, the magic formula is used to represent the nonlinear characteristics of the tire model.

[0045] Then, the lateral dynamics model of the tire is established:

[0046] δ i = arctan ((v y ± ω i I bi ) / (v x ± ω i Lbi ))-a i (3)

[0047] Based on the above analysis, a three-degree-of-freedom vehicle dynamics model is established.

[0048] Step 2, establish the MDP model of vehicle motion control based on DRL.

[0049] The vehicle motion control process directly uses the MDP form:

[0050] The state of DRL is designed as where y and represent the lateral displacement and heading angle of the vehicle, respectively, and e represents the current vehicle state error.

[0051] dfl δ dfr δ drl δ drr , represents the control input of the steering angle of each wheel.

[0052] The reward function is designed as the product of the positive gain parameter K r and the error e ref .

[0053] Step 3, establish the DRL controller according to step 1;

[0054] Select the TD3 algorithm to implement the DRL part, first initialize the policy network π φ , two Q networks and and their corresponding target networks π φ‘ , two Q networks and

[0055] Then select the action a φ according to the current policy network π t and the state s t . After performing the action, record the immediate reward r t and the new state s t+1 , and store the new transition sample (s t , a t , r t , s t+1 ) into the experience replay pool Ω.

[0056] Randomly sample a batch of samples (s t , a t , r t , s t+1). Then the target policy network π φ‘ generates the next state s t+1 and the target action a', preventing the overfitting of the policy network to the actions.

[0057] In the critic network, an IB module is introduced, mainly to improve the generalization ability and sample efficiency of the network. By compressing the redundant information in the high-dimensional features while preserving the basic information of the Q value estimation, the IB module enables the network to learn the state-action value function more effectively. It is hoped that the IB module can reduce the complexity of high-dimensional input, reduce the risk of overfitting, extract the most relevant features, improve the accuracy of Q value estimation, and enhance the adaptability of the model in different environments. Based on these design goals, the training loss function of the critic network is formulated as follows:

[0058]

[0059] where θ represents the parameters of the critic network, y t is the target value of TD, p(z t |s t ,a t ) is the conditional distribution defined by the IB module, and r(z t ) is the prior distribution. is the standard TD error term to ensure accurate Q value estimation; KL(p(z t |s t ,a t ), r(z t )) achieves information bottlenecking by minimizing the divergence between the conditional distribution and its prior distribution, thereby constraining the information flow from high-dimensional features to features, achieving feature compression and selection.

[0060] Through the optimization of this loss function, TD3 gradually adjusts the parameters θ1 and θ2 of the Q network to improve the accuracy of Q value estimation.

[0061] TD3 introduces a delayed policy update mechanism, that is, the policy network π φ is updated only after updating the Q network twice. The goal of the policy network is to maximize the Q value, and by delaying the update, the policy network avoids relying on inaccurate Q values too early.

[0062] where the design of the objective function J(φ) for the policy network introduces an RIB module, mainly to improve the quality and expressiveness of the policy. Through the RIB module, the goal is to maximize the mutual information between the state representation and the action, preserve the key information needed for policy generation, enhance the discriminative ability of the state representation to better distinguish the values of various actions, and improve the exploration ability of the policy to generate more diverse and effective actions. Based on these design goals, the training loss function of the actor network is formulated as follows:

[0063]

[0064] where φ represents the parameters of the actor network, p(u t |s t ) is the conditional distribution defined by the RIB module, and r(u t ) is the prior distribution. is the standard policy gradient term, aiming to maximize the expected cumulative return to ensure that the generated policy can bring high return; KL(p(u t |s t ), r(u t )) realizes the reverse information bottleneck, which increases the mutual information between the state representation u t and the action by maximizing the KL divergence between the conditional distribution p(u t |s t ) and its prior distribution r(u t ), and retains more information related to the policy.

[0065] Finally, the parameters of the target Q network and the target policy network are updated using the soft update method.

[0066] After training is completed, the optimal action a is directly calculated by the policy network, which is used to guide the actual decision of the agent in the environment.

[0067] Step 4, combine the vehicle dynamics model of step 1 and the DRL controller of step 3 through the MDP model of step 2

[0068] The DRL transmits the action mapped by the vehicle state to the vehicle, updates the vehicle state and transmits it to the DRL again.

[0069] The content described in the implementation cases of the specification is only a description of the actor-critic network structure improved by the double-flow information bottleneck, and the protection scope of the present application should not be regarded as limited to the specific forms and parameters stated in the implementation cases. The protection scope of the present application also relates to the improvement of the actor-critic network by different information bottleneck technologies.

Claims

1.A vehicle path tracking control method based on a double-flow information bottleneck deep reinforcement learning actor-critic architecture, using an information bottleneck (IB) to remove redundant information in samples, extract high-dimensional features in data, and reduce generalization errors; in order to alleviate the overfitting of a deep reinforcement learning (DRL) to known data caused by the IB, a reverse information bottleneck (RIB) reverses the optimization objective of the IB to generate discriminative unknown environment feature information and improve the generalization ability of the DRL controller; The specific implementation steps include: Step 1, establish a four-wheel steering vehicle dynamics model; Step 2, the motion control process of the vehicle directly uses the MDP form: Step 3, select TD3 algorithm to realize DRL part; Step 4, DRL passes the action mapped by the vehicle state to the vehicle, updates the vehicle state and passes it to DRL again. 2.The vehicle path following control method based on the double-flow information bottleneck deep reinforcement learning actor-critic architecture according to claim 1, wherein, In Step 1, the vehicle dynamics model includes a body dynamics model and a tire model; OXYZ and O v X v Y v Z v represent the earth and body inertial reference frames, respectively; it is assumed that path following occurs on a flat road; the body's yaw motion is primarily controlled by the longitudinal and lateral forces on each tire; for ease of notation, the four tires are indexed as i = fl, fr, rl, rr; in the tire coordinate system, the lateral force and steering angle of each tire are denoted by F li ,F Li ,δ i , respectively; the body dynamics can be written as: where M and I γ represent the vehicle mass and moment of inertia, (L bi , bi represent the distance of each tire from the center of gravity of the vehicle in the coordinate system, represent the longitudinal, lateral and yaw angular acceleration of the vehicle, respectively; The vehicle acceleration in the body coordinate system is calculated by the force applied by the tire; in the design of the path tracking controller, the longitudinal force and lateral force at each tire are determined by controlling the steering angle and wheel torque; in the tire coordinate system, the magic formula is used to represent the nonlinear characteristics of the tire model; Then, the lateral dynamics model of the tire is established: delta i = arctan((v y + omega i I bi ) / (v x + omega i L bi ))- alpha i (3) Based on the above analysis, a three-degree-of-freedom vehicle dynamics model is established. 3.The vehicle path following control method based on the double-flow information bottleneck deep reinforcement learning actor-critic architecture according to claim 1, wherein, In step 2, the state of DRL is designed as where y and denote the lateral displacement and the heading angle of the vehicle, respectively, denote the current vehicle state error; Actions a = [δ dfl δ dfr δ drl δ drr ] of the DRL controller, representing the control input of the steering angle of each wheel; The reward function is designed as a positive gain parameter K r a product of the error e ref ​ 4.The vehicle path following control method based on the double-flow information bottleneck deep reinforcement learning actor-critic architecture according to claim 1, wherein, In step 3, the policy network π is first initialized φ , two Q networks and and their corresponding target networks π φ‘ , two Q networks and Then according to the current policy network π φ and state s t select action a t ; after performing the action, record the immediate reward r t and new state s t+1 , and store the new transition sample (s t , a t , r t , s t+1 ) into the experience replay pool Ω; Randomly sample a batch of samples (s t ,a t ,r t ,s t+1 ); then use the target policy network π φ ‘ Generate the next state s t+1 The target action a' is used to prevent the policy network from overfitting the action; In the critic network, an IB module is introduced to improve the generalization ability and sample efficiency of the network; by compressing the redundant information in high-dimensional features while preserving the basic information of Q value estimation, the IB module enables the network to learn the state-action value function more effectively; it is hoped that the IB module can reduce the complexity of high-dimensional input, reduce the risk of overfitting, extract the most relevant features, improve the accuracy of Q value estimation, and enhance the adaptability of the model in different environments; based on these design goals, the training loss function of the critic network is formulated as follows: where θ denotes the parameters of the critic network, y t is the target value of TD, p(z t | s t , a t ) is the conditional distribution defined by the IB module, r(z t ) is the prior distribution; is the standard TD error term that ensures accurate Q-value estimates; KL(p(z t | s t , a t ), r(z t )) achieves information bottleneck by minimizing the divergence between the conditional distribution and its prior, thus constraining the flow of information from high-dimensional features to features, achieving feature compression and selection; Through the optimization of this loss function, TD3 gradually adjusts the parameters θ1 and θ2 of the Q network to improve the accuracy of Q value estimation; TD3 introduces a delayed policy update mechanism, that is, the policy network π is updated only once after every two Q network updates φ ; the goal of the policy network is to maximize the Q value, and by delaying the update, the policy network avoids relying on inaccurate Q values too early; Among them, the design of the target function J(φ) of the policy network introduces the RIB module to improve the quality and expressiveness of the policy; through the RIB module, the goal is to maximize the mutual information between the state representation and the action, save the key information needed for policy generation, enhance the discrimination ability of the state representation to better distinguish the value of various actions, and improve the exploration ability of the policy to generate more diverse and effective actions; based on these design goals, the training loss function of the actor network is formulated as follows: J(φ) = E s～Ω [Q θ1 (s,π φ (s))]+KL(p(u t |s t ),r(u t ))(7) where φ represents the parameters of the actor network, p(u t |s t ) is the conditional distribution defined by the RIB module, and r(u t ) is the prior distribution; E s～Ω [Q θ1 (s, π φ (s))] is the standard policy gradient term, which aims to maximize the expected cumulative return to ensure that the generated policy can bring high returns; and KL(p(u t |s t ), r(u t )) realizes the reverse information bottleneck, which increases the mutual information between the state representation u t and the action by maximizing the KL divergence between the conditional distribution p(u t |s t ) and its prior distribution r(u t ), thereby retaining more information related to the policy. Finally, the parameters of the target Q network and the target policy network are updated using the soft update method; After training is completed, the optimal action a is directly calculated by the policy network, which is used to guide the actual decision-making of the agent in the environment.

Images