Crawler vehicle trajectory tracking precision optimization method based on deep reinforcement learning

Abstract

The present application belongs to the technical field of track tracking control of unmanned tracked vehicles, and particularly relates to a tracked vehicle track tracking precision optimization method based on deep reinforcement learning, comprising the following steps: step one: establishing a kinematics model based on instantaneous steering center according to the tracked vehicle; step two: setting a prediction time domain control time domain according to experience, and predicting a future track; step three: designing deep reinforcement learning state quantity and action quantity; step four: designing the structure of deep reinforcement learning Critic network and Actor network; step five: training an agent; performing multiple rounds of iterative optimization until an ideal training effect is achieved; step six: exporting the trained agent for online operation, and real-time solving the required compensation output value, which is added to the control quantity output in step two as the input of the bottom layer execution system; thus, the compensation of the control quantity is completed, and real-time optimization of track tracking control is achieved. The method improves the precision of tracked vehicle track tracking.

Description

Technical Field

[0001] This invention belongs to the field of unmanned tracked vehicle trajectory tracking and control technology. It addresses the problem of low trajectory tracking accuracy of tracked vehicles under complex working conditions and specifically involves a tracked vehicle trajectory tracking accuracy optimization method based on deep reinforcement learning. Background Technology

[0002] Tracked vehicle control is shifting from manned to unmanned control. Unmanned tracked vehicles are a type of highly nonlinear and coupled control system, especially under off-road conditions, where the nonlinear characteristics of the vehicle are significantly enhanced.

[0003] Model predictive controllers (MPCs) can be applied to trajectory tracking control of unmanned tracked vehicles. This method predicts the future trajectory of the tracked vehicle using a model, designs a loss function, and solves for the optimal control sequence using quadratic programming. Therefore, the control performance of MPCs largely depends on the accuracy of the model.

[0004] Tracked vehicles can be modeled using either kinematic or dynamic models. However, if a dynamic model is used, parameters such as the drag on both tracks and the air resistance experienced by the vehicle cannot be accurately obtained in practical applications, leading to inaccurate modeling and poor control performance. Therefore, kinematic models that consider track slippage are more commonly used. Considering slippage requires obtaining the vehicle's longitudinal and lateral velocities in the vehicle's coordinate system, or its coordinates in the global coordinate system, and using Kalman filtering to estimate the slip parameters.

[0005] In actual vehicles, due to actuator delays, inaccurate estimation of sliding parameters, and other reasons, the kinematic model cannot describe the vehicle's motion state without bias, and the control effect of the MPC controller will inevitably have deviations. Summary of the Invention

[0006] (a) Technical problems to be solved

[0007] The technical problem to be solved by this invention is: how to provide a method for optimizing the trajectory tracking accuracy of tracked vehicles based on deep reinforcement learning.

[0008] (II) Technical Solution

[0009] To address the aforementioned technical problems, this invention provides a method for optimizing the trajectory tracking accuracy of tracked vehicles based on deep reinforcement learning, the method comprising:

[0010] Step 1: Establish a kinematic model based on the instantaneous steering center of the tracked vehicle, expand the kinematic model with a first-order Taylor at the reference point, and use the Jacobian matrix to find the coefficient matrices A and B of the state-space equations to linearize the nonlinear system.

[0011] Step 2: Based on experience, set the prediction time domain control time domain, predict the future trajectory, determine the loss function J, use the parameters in the prediction process to represent the loss function, set constraints to consider the response characteristics of the motor and the lateral stability of the vehicle, use a quadratic programming solver to calculate the optimal control sequence ΔU, use the first set of control quantities Δu(t), sum the past control quantities with the solved Δu(t) and output the result.

[0012] Step 3: Design the state and action variables for deep reinforcement learning; select vehicle state parameters as observations, determine compensation parameters as output variables, and design the reward function r;

[0013] Step 4: Design the structure of the deep reinforcement learning Critic network and Actor network, build an agent based on the deep reinforcement learning algorithm, and set the training parameters;

[0014] Step 5: Train the agent; adjust the reward function based on the training results, and perform multiple rounds of iterative optimization until the ideal training effect is achieved;

[0015] Step Six: Export the trained agent for online computation, calculate the required compensation output value in real time, add it to the control quantity output in Step Two, and use it as the input of the underlying execution system; at this point, the compensation of the control quantity is completed, and real-time optimization of trajectory tracking control is achieved.

[0016] In step one, a kinematic model based on the instantaneous steering center is established for the tracked vehicle.

[0017]

[0018] The vehicle's position and orientation in the global coordinate system are (X,Y,θ), where θ is the heading angle of the tracked vehicle; in the local vehicle coordinate system, v x ,v y ,ω z These are the vehicle's longitudinal velocity, lateral velocity, and yaw rate, respectively; (x c ,y c (x) is the steering center of the tracked vehicle. l ,y l (x) is the instantaneous turning center of the left track of the tracked vehicle. r ,y r () is the instantaneous turning center of the right track of the tracked vehicle. These are the winding speeds on the left and right sides of the track, respectively.

[0019] The kinematic model is expanded using a first-order Taylor series at the reference point. The coefficient matrix of the state-space equations is obtained using the Jacobian matrix. The nonlinear system is then linearized and discretized to obtain the state-space equations:

[0020]

[0021] Where X = [X, Y, θ] T , represents a state variable. To control the quantity;

[0022] It needs to be linearized and discretized; the kinematic equations are expanded using Taylor at the reference point and then discretized using forward Euler.

[0023]

[0024] in,

[0025]

[0026]

[0027] Among them, T s Sampling time, θ serves as a reference value for the wheel speeds of the left and right drive wheels. d This is a reference value for the heading angle.

[0028] Step two involves building the MPC controller. The loss function used in designing the trajectory tracking controller is as follows:

[0029]

[0030] Where, N p For prediction in the time domain; N c To control the time domain; ρ is the weighting coefficient; ε is the relaxation factor;

[0031] Then let:

[0032]

[0033] The resulting new state-space expression is:

[0034]

[0035]

[0036] in, n represents the state variable dimension; m represents the control variable dimension.

[0037] Through iterative derivation, the predicted output expression of the system is obtained:

[0038] Y(k)=ψξ(k)+ΘΔU(k)

[0039]

[0040] in,

[0041]

[0042] The loss function is then expressed as:

[0043]

[0044] Where E = Ψξ(k);

[0045] Considering the motor's response capability and to prevent the vehicle from overturning, it is necessary to limit the incremental speed of the drive wheel, the maximum value, and the speed difference between the left and right sides.

[0046] First, the maximum torque that the motor can provide at the current speed is obtained by using the external characteristic curve of the motor. Combined with the resistance torque under the current state, the maximum acceleration of the motor speed is obtained.

[0047] Δu min (t+k)≤Δu(t+k)≤Δu max (t+k),

[0048] k = 0, 1, ..., N c -1

[0049] Where Δu is the change in control quantity at each time step;

[0050] Secondly, it is necessary to limit the maximum speed and speed difference of the active wheel speed; that is, it is necessary to ensure that the control output is within the constraints during the model prediction process.

[0051] Therefore, the constraints are:

[0052]

[0053] in

[0054]

[0055]

[0056]

[0057]

[0058] This constraint simultaneously constrains the rotational speed value and the speed difference between the left and right sides; where m is the control variable dimension. U represents the Kronecker product; u(k-1) is the control quantity at the previous moment, and u1(k-1) and u2(k-1) represent the first and second elements of the control quantity, i.e., the left and right wheel speeds; U max U wmaxThese are the maximum speed of the driving wheel and the maximum difference in speed between the two driving wheels, respectively.

[0059] Step three involves: determining the observed quantities as the deviation of the vehicle's position and attitude, the vehicle's yaw rate, and the position and attitude at the previous moment, using state quantities including the error between the current position and attitude predicted by the kinematic model and the actual position and attitude at the current moment; determining the output quantity as the difference in active wheel speeds on both sides; and designing a reward function.

[0060] In step four, the structure of the Critic network and the Actor network is designed. The Critic network merges the state space and action space and outputs the Q value after passing through several fully connected hidden layers.

[0061] During training, the Critic network is updated more frequently than the Actor network, and a soft update method is used to update the target network.

[0062] w′←τw+(1-τ)w′

[0063] φ′←τφ+(1-τ)φ′

[0064] During training, the agent takes a set of four-tuples from the experience replay pool. This four-tuple contains state information 's', including the current position and attitude deviation of the tracked vehicle. Based on the current state information, the Actor network outputs an action 'a', which compensates for the speed difference. The reward 'r' given by the environment is then provided, along with the state information 's' for the next moment. The agent calculates two estimated Q values, Q1 and Q2, for the current moment using two Critic networks based on (s, a). Then, based on the state 's' for the next moment and the noisy action value μ'(s'; φ) + ε, it calculates two more Q values, Q1' and Q2', for the next moment using a Target Critic network. The smaller Q value is added to the reward value to obtain the actual Q value for the current moment. The TD error of the two Critic networks is calculated separately, and the Critic network is updated using gradient descent of the loss function.

[0065]

[0066]

[0067] Where μ′(…;φ) represents the TargetActor network, Q′(…;w Q′ ) represents the Target Critic network, and μ′(s′;φ) represents the action output at the next time step. y represents the Q-value estimated by the Target Critic network for the next time step; r is the reward of the environment; γ is the discount factor that determines the priority of short-term rewards; α is the learning rate; Q(s,a;w) represents the Q-value estimated by the Critic network for the current time step; s′,a′ represent the state space and action space for the next time step; Q(s′,a′;w) represents the Q-value estimated by the Critic network for the next time step.

[0068] After the Critic network is updated several times, the Actor network is updated using gradient ascent. During the online process, only the Actor network participates in the calculation, and the compensation speed difference is given in real time according to the current tracked vehicle error, so that the computational load will not be too large due to the dual Critic network.

[0069] Step six involves exporting the trained agent for online computation to calculate the required compensation output value in real time.

[0070] The method described herein establishes an MPC controller and designs an output compensation module based on a deep reinforcement learning algorithm for the MPC controller.

[0071] The method is used for the control of model-free, nonlinear control systems.

[0072] The output compensation module uses environmental information, including vehicle position and attitude, to directly compensate the output of the MPC controller.

[0073] In offline mode, the output compensation module learns the parameters of the compensation module through the environment;

[0074] In online mode, the output compensation module performs output compensation on the MPC controller in real time through observation.

[0075] (III) Beneficial Effects

[0076] To address the problems of existing technologies, this invention employs deep reinforcement learning as the output compensation for the MPC controller, providing a method for optimizing the trajectory tracking accuracy of tracked vehicles based on deep reinforcement learning. Specifically, it involves a method for building an MPC controller, a method for designing a deep reinforcement learning algorithm, and an error compensation method. This method improves the accuracy of tracked vehicle trajectory tracking. Attached Figure Description

[0077] Figure 1 This is a schematic diagram of an MPC controller based on deep reinforcement learning.

[0078] Figure 2 This is a schematic diagram of an MPC controller embodiment based on TD3 compensation.

[0079] Figure 3This is a schematic diagram of the deviation in the X and Y axes of a pure MPC controller.

[0080] Figure 4 This is a schematic diagram of the deviation in the X and Y axes of the MPC controller based on TD3 compensation. Detailed Implementation

[0081] To make the objectives, contents, and advantages of the present invention clearer, the specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples.

[0082] To address the aforementioned technical problems, this invention provides a method for optimizing the trajectory tracking accuracy of tracked vehicles based on deep reinforcement learning, the method comprising:

[0083] Step 1: Establish a kinematic model based on the instantaneous steering center of the tracked vehicle, expand the kinematic model with a first-order Taylor at the reference point, and use the Jacobian matrix to find the coefficient matrices A and B of the state-space equations to linearize the nonlinear system.

[0084] Step 2: Based on experience, set the prediction time domain control time domain, predict the future trajectory, determine the loss function J, use the parameters in the prediction process to represent the loss function, set constraints to consider the response characteristics of the motor and the lateral stability of the vehicle, use a quadratic programming solver to calculate the optimal control sequence ΔU, use the first set of control quantities Δu(t), sum the past control quantities with the solved Δu(t) and output the result.

[0085] Step 3: Design the state and action variables for deep reinforcement learning; select vehicle state parameters as observations, determine compensation parameters as output variables, and design the reward function r;

[0086] Step 4: Design the structure of the deep reinforcement learning Critic network and Actor network, build an agent based on the deep reinforcement learning algorithm, and set the training parameters;

[0087] Step 5: Train the agent; adjust the reward function based on the training results, and perform multiple rounds of iterative optimization until the ideal training effect is achieved;

[0088] Step Six: Export the trained agent for online computation, calculate the required compensation output value in real time, add it to the control quantity output in Step Two, and use it as the input of the underlying execution system; at this point, the compensation of the control quantity is completed, and real-time optimization of trajectory tracking control is achieved.

[0089] In step one, a kinematic model based on the instantaneous steering center is established for the tracked vehicle.

[0090]

[0091] The vehicle's position and orientation in the global coordinate system are (X,Y,θ), where θ is the heading angle of the tracked vehicle; in the local vehicle coordinate system, v x ,v y ,ω z These are the vehicle's longitudinal velocity, lateral velocity, and yaw rate, respectively; (x c ,y c (x) is the steering center of the tracked vehicle. l ,y l (x) is the instantaneous turning center of the left track of the tracked vehicle. r ,y r () is the instantaneous turning center of the right track of the tracked vehicle. These are the winding speeds on the left and right sides of the track, respectively.

[0092] The kinematic model is expanded using a first-order Taylor series at the reference point. The coefficient matrix of the state-space equations is obtained using the Jacobian matrix. The nonlinear system is then linearized and discretized to obtain the state-space equations:

[0093]

[0094] Where X = [X, Y, θ] T , represents a state variable. To control the quantity;

[0095] It needs to be linearized and discretized; the kinematic equations are expanded using Taylor at the reference point and then discretized using forward Euler.

[0096]

[0097] in,

[0098]

[0099]

[0100] Among them, T s Sampling time, θ serves as a reference value for the wheel speeds of the left and right drive wheels. d This is a reference value for the heading angle.

[0101] Step two involves building the MPC controller. The loss function used in designing the trajectory tracking controller is as follows:

[0102]

[0103] Where, N p For prediction in the time domain; N c To control the time domain; ρ is the weighting coefficient; ε is the relaxation factor;

[0104] Then let:

[0105]

[0106] The resulting new state-space expression is:

[0107]

[0108]

[0109] in, n represents the state variable dimension; m represents the control variable dimension; through iterative derivation, the predicted output expression of the system is obtained:

[0110] Y(k)=ψξ(k)+ΘΔU(k)

[0111]

[0112] in,

[0113]

[0114] The loss function is then expressed as:

[0115]

[0116] Where E = Ψξ(k);

[0117] Considering the motor's response capability and to prevent the vehicle from overturning, it is necessary to limit the incremental speed of the drive wheel, the maximum value, and the speed difference between the left and right sides.

[0118] First, the maximum torque that the motor can provide at the current speed is obtained by using the external characteristic curve of the motor. Combined with the resistance torque under the current state, the maximum acceleration of the motor speed is obtained.

[0119] Δu min (t+k)≤Δu(t+k)≤Δu max (t+k),

[0120] k = 0, 1, ..., N c -1

[0121] Where Δu is the change in control quantity at each time step;

[0122] Secondly, it is necessary to limit the maximum speed and speed difference of the active wheel speed; that is, it is necessary to ensure that the control output is within the constraints during the model prediction process.

[0123] Therefore, the constraints are:

[0124]

[0125] in

[0126]

[0127]

[0128]

[0129]

[0130] This constraint simultaneously constrains the rotational speed value and the speed difference between the left and right sides; where m is the control variable dimension. U represents the Kronecker product; u(k-1) is the control quantity at the previous moment, and u1(k-1) and u2(k-1) represent the first and second elements of the control quantity, i.e., the left and right wheel speeds; U max U wmax These are the maximum speed of the driving wheel and the maximum difference in speed between the two driving wheels, respectively.

[0131] Step three involves: determining the observed quantities as the deviation of the vehicle's position and attitude, the vehicle's yaw rate, and the position and attitude at the previous moment, using state quantities including the error between the current position and attitude predicted by the kinematic model and the actual position and attitude at the current moment; determining the output quantity as the difference in active wheel speeds on both sides; and designing a reward function.

[0132] In step four, the structure of the Critic network and the Actor network is designed. The Critic network merges the state space and action space and outputs the Q value after passing through several fully connected hidden layers.

[0133] During training, the Critic network is updated more frequently than the Actor network, and a soft update method is used to update the target network.

[0134] w′←τw+(1-τ)w′

[0135] φ′←τφ+(1-τ)φ′

[0136] During training, the agent takes a set of four-tuples from the experience replay pool. This four-tuple contains state information 's', including the current position and attitude deviation of the tracked vehicle. Based on the current state information, the Actor network outputs an action 'a', which compensates for the speed difference. The reward 'r' given by the environment is then provided, along with the state information 's' for the next moment. The agent calculates two estimated Q values, Q1 and Q2, for the current moment using two Critic networks based on (s, a). Then, based on the state 's' for the next moment and the noisy action value μ'(s'; φ) + ε, it calculates two more Q values, Q1' and Q2', for the next moment using a Target Critic network. The smaller Q value is added to the reward value to obtain the actual Q value for the current moment. The TD error of the two Critic networks is calculated separately, and the Critic network is updated using gradient descent of the loss function.

[0137]

[0138]

[0139] Where μ′(…;φ) represents the TargetActor network, Q′(…;w Q′ ) represents the Target Critic network, and μ′(s′;φ) represents the action output at the next time step. y represents the Q-value estimated by the Target Critic network for the next time step; r is the reward of the environment; γ is the discount factor that determines the priority of short-term rewards; α is the learning rate; Q(s,a;w) represents the Q-value estimated by the Critic network for the current time step; s′,a′ represent the state space and action space for the next time step; Q(s′,a′;w) represents the Q-value estimated by the Critic network for the next time step.

[0140] After the Critic network is updated several times, the Actor network is updated using gradient ascent. During the online process, only the Actor network participates in the calculation, and the compensation speed difference is given in real time according to the current tracked vehicle error, so that the computational load will not be too large due to the dual Critic network.

[0141] Step six involves exporting the trained agent for online computation to calculate the required compensation output value in real time.

[0142] The method described herein establishes an MPC controller and designs an output compensation module based on a deep reinforcement learning algorithm for the MPC controller.

[0143] The method is used for the control of model-free, nonlinear control systems.

[0144] The output compensation module uses environmental information, including vehicle position and attitude, to directly compensate the output of the MPC controller.

[0145] In offline mode, the output compensation module learns the parameters of the compensation module through the environment;

[0146] In online mode, the output compensation module performs output compensation on the MPC controller in real time through observation.

[0147] Example 1

[0148] According to the present invention, a specific embodiment for optimizing the trajectory tracking control effect of a certain tracked vehicle is as follows:

[0149] See the control block diagram for a specific embodiment. Figure 2 .

[0150] In step one, a kinematic model based on the instantaneous steering center is established for the tracked vehicle.

[0151]

[0152] The vehicle's position and orientation in the global coordinate system are (X,Y,θ), where θ is the heading angle of the tracked vehicle; in the local vehicle coordinate system, v x ,v y ,ω z These are the vehicle's longitudinal velocity, lateral velocity, and yaw rate, respectively; (x c ,y c (x) is the steering center of the tracked vehicle. l ,y l (x) is the instantaneous turning center of the left track of the tracked vehicle. r ,y r () is the instantaneous turning center of the right track of the tracked vehicle. These are the winding speeds on the left and right sides of the track, respectively.

[0153] The kinematic model is expanded using a first-order Taylor series at the reference point. The coefficient matrix of the state-space equations is obtained using the Jacobian matrix. The nonlinear system is then linearized and discretized to obtain the state-space equations:

[0154]

[0155] Where X = [X, Y, θ] T , represents a state variable. To control the quantity;

[0156] It needs to be linearized and discretized; the kinematic equations are expanded using Taylor at the reference point and then discretized using forward Euler.

[0157]

[0158] in,

[0159]

[0160]

[0161] Among them, T s Sampling time, θ serves as a reference value for the wheel speeds of the left and right drive wheels. d This is a reference value for the heading angle.

[0162] Step 2: Build the MPC controller; the loss function used in designing the trajectory tracking controller is as follows:

[0163]

[0164] Where, N p For prediction in the time domain; N c To control the time domain; ρ is the weighting coefficient; ε is the relaxation factor;

[0165] Then let:

[0166]

[0167] The resulting new state-space expression is:

[0168]

[0169]

[0170] in, n represents the state variable dimension; m represents the control variable dimension.

[0171] Through iterative derivation, the predicted output expression of the system is obtained:

[0172] Y(k)=ψξ(k)+ΘΔU(k)

[0173]

[0174] in,

[0175]

[0176] The loss function can then be expressed as:

[0177]

[0178] Where E = Ψξ(k);

[0179] Considering the motor's response capability and to prevent the vehicle from overturning, it is necessary to limit the incremental speed of the drive wheel, the maximum value, and the speed difference between the left and right sides.

[0180] First, the maximum torque that the motor can provide at the current speed is obtained by using the external characteristic curve of the motor. Combined with the resistance torque under the current state, the maximum acceleration of the motor speed is obtained.

[0181] Δu min (t+k)≤Δu(t+k)≤Δu max (t+k),

[0182] k = 0, 1, ..., N c -1

[0183] Where Δu is the change in control quantity at each time step;

[0184] Secondly, it is necessary to limit the maximum speed and speed difference of the active wheel speed; that is, it is necessary to ensure that the control output is within the constraints during the model prediction process.

[0185] Therefore, the constraints are:

[0186]

[0187] in

[0188]

[0189]

[0190]

[0191]

[0192] This constraint can simultaneously constrain the rotational speed value and the difference in rotational speed between the left and right sides; where m is the dimension of the control variable. U represents the Kronecker product; u(k-1) is the control quantity at the previous moment, and u1(k-1) and u2(k-1) represent the first and second elements of the control quantity, i.e., the left and right wheel speeds; U max U wmax These are the maximum speed of the driving wheel and the maximum difference in speed between the two driving wheels, respectively.

[0193] Step 3: Determine the observed quantities as the deviation of vehicle position and attitude, vehicle yaw rate, and position and attitude at the previous moment. Use state quantities including the error between the current position and attitude predicted by the kinematic model and the actual position and attitude at the current moment; determine the output quantity as the difference in active wheel speeds on both sides; and design a reward function.

[0194] Step 4: Design the structure of the Critic network and the Actor network. The Critic network merges the state space and action space, and outputs the Q value after passing through several fully connected hidden layers.

[0195] During training, the Critic network is updated more frequently than the Actor network, and a soft update method is used to update the target network.

[0196] w′←τw+(1-τ)w′

[0197] φ′←τφ+(1-τ)φ′

[0198] During training, the agent takes a set of four-tuples from the experience replay pool. This set contains state information 's', including the current position and attitude deviation of the tracked vehicle. Based on the current state information, the Actor network outputs an action 'a', which compensates for the speed difference. The reward 'r' given by the environment is then provided, along with the state information 's' for the next moment. The agent calculates two estimated Q values, Q1 and Q2, for the current moment using two Critic networks based on (s, a). Then, based on the state 's' for the next moment and the noisy action value μ'(s'; φ) + ε, it calculates two more Q values, Q1' and Q2', for the next moment using a Target Critic network. The smaller Q value is added to the reward value to obtain the actual Q value for the current moment. The TD error of the two Critic networks is calculated separately, and the Critic network is updated using gradient descent of the loss function.

[0199]

[0200]

[0201] Where μ′(…;φ) represents the TargetActor network, Q′(…;w Q′ ) represents the Target Critic network, and μ′(s′;φ) represents the action output at the next time step. y represents the Q-value estimated by the Target Critic network for the next time step; r is the reward of the environment; γ is the discount factor that determines the priority of short-term rewards; α is the learning rate; Q(s,a;w) represents the Q-value estimated by the Critic network for the current time step; s′,a′ represent the state space and action space for the next time step; Q(s′,a′;w) represents the Q-value estimated by the Critic network for the next time step.

[0202] After the Critic network is updated several times, the Actor network is updated using gradient ascent. During the online process, only the Actor network participates in the calculation, and the compensation speed difference is given in real time according to the current tracked vehicle error, so that the computational load will not be too large due to the dual Critic network.

[0203] Step 5: Train the agent; adjust the reward function based on the training results, and perform multiple rounds of iterative optimization until the ideal training effect is achieved;

[0204] Step Six: Export the trained agent for online computation, calculating the required compensation output value in real time. Add this value to the control output of the MPC controller from Step Two, and use it as the input to the underlying execution system; thus, the compensation of the control quantity is completed, achieving real-time optimization of trajectory tracking control.

[0205] The designed compensation module based on deep reinforcement learning improves the accuracy of tracked vehicle trajectory tracking. The tracking performance on circular trajectories at 30 km / h is shown in the comparison. Figure 3 and Figure 4 As shown.

[0206] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for track tracking precision optimization of a tracked vehicle based on deep reinforcement learning, characterized in that, The method comprises: Step one: according to the tracked vehicle, a kinematics model based on instantaneous steering center is established, the kinematics model is first-order Taylor expanded at the reference point, the coefficient matrix A and B of the state space equation are solved by using the Jacobian matrix, and the nonlinear system is linearized; Step two: according to experience, the prediction time domain and the control time domain are set, the future trajectory is predicted, the loss function J is determined, the loss function is expressed by using the parameters in the prediction process, the constraint conditions are set to consider the response characteristics of the motor and the lateral stability of the vehicle, the optimal control sequence ΔU is calculated by using the quadratic programming solver, the first group of control quantities Δu(t) is adopted, the past control quantity and the calculated Δu(t) are summed and output; Step three: the state quantity and the action quantity of the deep reinforcement learning are designed; the vehicle state parameter is selected as the observation quantity, the compensation parameter is determined as the output quantity, and the reward function r is designed; Step four: the structure of the Critic network and the Actor network of the deep reinforcement learning is designed, the intelligent agent based on the deep reinforcement learning algorithm is built, and the training parameters are set; Step five: the intelligent agent is trained; according to the training effect, the reward function is adjusted, and multiple rounds of iterative optimization are carried out until the ideal training effect is achieved; Step six: the trained intelligent agent is exported for online operation, the required compensation output value is calculated in real time, and the compensation output value is added to the control quantity output in step two as the input of the bottom execution system; thus, the compensation of the control quantity is completed, and the real-time optimization of the trajectory tracking control is realized; In step one, a kinematics model based on instantaneous steering center is established according to the tracked vehicle, Wherein the position and attitude of the vehicle in the global coordinate system is (X, Y, θ), wherein θ is the heading angle of the tracked vehicle; in the local vehicle body coordinate system, v x , v y , ω z are the longitudinal velocity, lateral velocity and yaw rate of the vehicle respectively; (x c ,y c ) is the steering center of the tracked vehicle, (x l ,y l ) is the instantaneous steering center of the left track of the tracked vehicle, (x r ,y r ) is the instantaneous steering center of the right track of the tracked vehicle, are the left and right winding speeds of the track respectively; The kinematics model is first-order Taylor expanded at the reference point, the coefficient matrix of the state space equation is solved by using the Jacobian matrix, the nonlinear system is linearized, and then discretized to obtain the state space equation: where X = [X, Y, θ] T , is the state variable, is the control variable; It needs to be linearized and discretized; the kinematics equation is Taylor expanded at the reference point, and discretized by forward Euler to obtain Wherein, where T s is the sampling time, is the reference value of the left and right wheel speed, θ d is the reference value of the heading angle.

2. The deep reinforcement learning based tracked vehicle trajectory tracking precision optimization method of claim 1, wherein, In step two, the MPC controller is built; when designing the trajectory tracking controller, the loss function form is as follows where N p is the prediction horizon; N c is the control horizon; p is a weighting factor; and e is a relaxation factor. Then let: The new state space expression is obtained as follows: wherein, n is the state dimension; m is the control dimension; After iterative derivation, the prediction output expression of the system is obtained as follows: Y(k)=ψξ(k)+ΘΔU(k) wherein Then the loss function is expressed as: Wherein, E=Ψξ(k); Considering the response ability of the motor and preventing the vehicle from rolling over, the increment, the maximum value and the speed difference of the left and right sides of the driving wheel speed need to be limited; Firstly, the maximum torque that can be provided by the motor at the current speed is obtained through the external characteristic curve of the motor, and the maximum acceleration of the motor speed is obtained by combining the current state resistance torque; Δu min (t+k)≤Δu(t+k)≤Δu max (t+k), k = 0, 1,..., N c -1 Wherein, Δu is the change of the control quantity at each time step; Secondly, the maximum speed and the speed difference of the driving wheel speed need to be limited; that is, it is necessary to ensure that the control quantity output in the model prediction process is within the constraint condition; Then the constraint condition is: Wherein This constraint condition simultaneously constrains the speed value and the speed difference value on the left and right sides; wherein, m is the dimension of the control quantity, represents the Kronecker product; u(k-1) is the control quantity at the last moment, and u1(k-1), u2(k-1) represent the first element and the second element in the control quantity, i.e. the left and right wheel speeds; max ,U wmax are the maximum value of the driving wheel speed and the maximum value of the speed difference of the driving wheels on the left and right sides, respectively.

3. The deep reinforcement learning based tracked vehicle trajectory tracking precision optimization method of claim 2, wherein, The third step is to determine the observation as the deviation of the vehicle position and attitude, the vehicle yaw rate and the state quantity including the error between the current position and attitude predicted by the kinematic model and the actual position and attitude at the current time; determine the output quantity as the speed difference of the two driving wheels; and design the reward function.

4. The deep reinforcement learning based tracked vehicle trajectory tracking precision optimization method of claim 3, wherein, The fourth step is to design the structure of the Critic network and the Actor network, the Critic network combines the state space and the action space, and outputs the Q value after passing through several layers of fully connected hidden layers; During the training process, the Critic network has a higher update frequency than the Actor network, and the target network is updated in a soft update manner: w′←τw+(1-τ)w′ φ′←τφ+(1-τ)φ′ During the training process, the agent takes a set of four tuples from the experience replay pool, which contains state quantity information s including the position and attitude deviation of the tracked vehicle at the current time, the output action a given by the Actor network through the current state quantity information, the reward r given by the environment, and the state quantity information s' at the next time; the agent calculates the two Q values Q1 and Q2 at the current time according to (s, a) through the two Critic networks, then calculates the two Q values Q1' and Q2' at the next time according to the state s' at the next time and the action value μ'(s'; φ) + ε with noise, takes the smaller Q value and adds the reward value to obtain the actual Q value at the current time, calculates the TD error of the two Critic networks respectively, and updates the Critic network through the gradient descent of the loss function: wherein μ'(·; φ) represents the TargetActor network, Q'(·; w Q′ ) represents the TargetCritic network, μ'(s'; φ) represents the action output at the next time point, represents the Q value at the next time point estimated by the Target Critic network; y represents the target Q value, r is the reward of the environment, γ is a discount factor that determines the priority of short-term rewards, α is the learning rate, Q(s, a; w) represents the Q value at the current time point estimated by the Critic network, s', a' represents the state space and action space at the next time point, and Q(s', a'; w) represents the Q value at the next time point estimated by the Critic network. After updating the Critic network for several times, the Actor network is updated using gradient ascent; during the online process, only the Actor network is involved in the calculation, and the compensation speed difference is given in real time according to the current tracked vehicle error, which will not cause excessive calculation due to the double Critic network.

5. The deep reinforcement learning based tracked vehicle trajectory tracking precision optimization method of claim 4, wherein, The method builds an MPC controller and designs an output compensation module based on a deep reinforcement learning algorithm for the MPC controller.

6. The deep reinforcement learning-based tracked vehicle trajectory tracking precision optimization method of claim 5, wherein, The method is used for the control of a model-free and nonlinear control system.

7. The deep reinforcement learning-based tracked vehicle trajectory tracking precision optimization method of claim 6, wherein, The output compensation module uses environmental information including vehicle position and attitude to directly compensate the output of the MPC controller.

8. The deep reinforcement learning based tracked vehicle trajectory tracking precision optimization method of claim 7, wherein, In the offline state, the output compensation module learns the parameters of the compensation module through the environment; In the online state, the output compensation module compensates the output of the MPC controller in real time through the observation.

Images