Tracked vehicle trajectory tracking precision optimization method based on deep reinforcement learning

Abstract

The invention belongs to the technical field of unmanned tracked vehicle trajectory tracking control, and particularly relates to a tracked vehicle trajectory tracking precision optimization method based on deep reinforcement learning, and the method comprises the steps: 1, building a kinematic model based on an instantaneous steering center according to a tracked vehicle; 2, setting a prediction time domain and a control time domain according to experience, and predicting a future trajectory; 3, designing a deep reinforcement learning state quantity and an action quantity; step 4, designing structures of a deep reinforcement learning Critic network and an Actor network; 5, training the intelligent agent; performing multi-round iterative optimization until an ideal training effect is achieved; step 6, exporting the trained intelligent agent to carry out online operation, resolving a required compensation output value in real time, adding the compensation output value with the control quantity output in the step 2, and taking the added value as the input of a bottom execution system; therefore, compensation of the control quantity is completed, and real-time optimization of trajectory tracking control is achieved. According to the method, the track tracking precision of the tracked vehicle is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of unmanned tracked vehicle trajectory tracking control, and specifically relates to a tracked vehicle trajectory tracking accuracy optimization method based on deep reinforcement learning, in order to solve the problem of low track tracking accuracy of tracked vehicles under complex working conditions. Background Art

[0002] Tracked vehicle control is shifting from manned control to unmanned control. Unmanned tracked vehicles are a type of control system with strong nonlinearity and high coupling. Especially in off-road conditions, the nonlinear characteristics of the vehicle will be significantly enhanced.

[0003] Model Predictive Controller (MPC) can be applied to trajectory tracking control of unmanned tracked vehicles. This method predicts the future vehicle trajectory through the model of the tracked vehicle, designs the loss function, and uses quadratic programming to solve the optimal control sequence. Therefore, the control effect of MPC depends largely on the accuracy of the model.

[0004] The model of the tracked vehicle can be a kinematic model or a dynamic model. If the dynamic model is used, in actual use, the resistance of the tracks on both sides and the air resistance of the vehicle cannot be accurately obtained, and the model is inaccurate and the control effect is poor. Therefore, the kinematic model that considers the slip of the track is often used. Considering the slip requires obtaining the longitudinal speed and lateral speed of the vehicle in the body coordinate system, or the coordinates of the vehicle in the global coordinate system, and using Kalman filtering to estimate the sliding parameters.

[0005] In actual vehicles, due to actuator delays, inaccurate sliding parameter estimation and other reasons, the kinematic model cannot describe the vehicle's motion state without deviation, and the control effect of the MPC controller is bound to have deviations. Summary of the invention

[0006] 1. Technical issues to be resolved

[0007] The technical problem to be solved by the present invention is: how to provide a method for optimizing the trajectory tracking accuracy of a tracked vehicle based on deep reinforcement learning.

[0008] (II) Technical solution

[0009] In order to solve the above technical problems, the present invention provides a method for optimizing track tracking accuracy of a tracked vehicle based on deep reinforcement learning, the method comprising:

[0010] Step 1: A kinematic model based on the instantaneous steering center is established according to the tracked vehicle, the kinematic model is expanded in the first order Taylor at the reference point, the coefficient matrices A and B of the state space equation are calculated using the Jacobian matrix, and the nonlinear system is linearized;

[0011] Step 2: Set the prediction time domain and control time domain based on experience, predict the future trajectory, determine the loss function J, use the parameters in the prediction process to represent the loss function, set constraints to consider the response characteristics of the motor and the lateral stability of the vehicle, use the quadratic programming solver to calculate the optimal control sequence ΔU, use the first set of control quantities Δu(t), and sum the past control quantities with the solved Δu(t) and output them;

[0012] Step 3: Design deep reinforcement learning state and action quantities; select vehicle state parameters as observation quantities, determine compensation parameters as output quantities, and design reward function r;

[0013] Step 4: Design the structure of the deep reinforcement learning Critic network and Actor network, build an intelligent agent based on the deep reinforcement learning algorithm, and set training parameters;

[0014] Step 5: Train the agent; adjust the reward function according to the training results, and perform multiple rounds of iterative optimization until the ideal training effect is achieved;

[0015] Step 6: Export the trained intelligent agent for online calculation, calculate the required compensation output value in real time, add it to the control amount output in step 2, and use it as the input of the underlying execution system; at this point, the compensation of the control amount is completed, and real-time optimization of trajectory tracking control is achieved.

[0016] Wherein, in the step 1, a kinematic model based on the instantaneous steering center is established according to the tracked vehicle.

[0017]

[0018] The position and attitude of the vehicle in the global coordinate system are (X, Y, θ), where θ is the heading angle of the tracked vehicle; in the local body coordinate system, v x ,v y ,ω z are the longitudinal velocity, lateral velocity and yaw rate of the vehicle respectively; (x c ,y c ) is the turning center of the crawler vehicle, (x l ,y l ) is the instantaneous turning center of the left track of the tracked vehicle, (x r ,y r ) is the instantaneous turning center of the right track of the tracked vehicle, They are the winding speed on the left side and the winding speed on the right side of the track respectively;

[0019] Expand the kinematic model in the first order Taylor at the reference point, use the Jacobian matrix to find the coefficient matrix of the state space equation, linearize the nonlinear system, and then discretize it to obtain the state space equation:

[0020]

[0021] Where X = [X, Y, θ] T , is the state quantity, To control the amount;

[0022] It needs to be linearized and discretized; the kinematic equation is Taylor expanded at the reference point and discretized by forward Euler to obtain

[0023]

[0024] in,

[0025]

[0026]

[0027] Among them, T s is the sampling time, is the reference value of the left and right driving wheel speed, θ d is the reference value of the heading angle.

[0028] Among them, the step 2: build an MPC controller; when designing the trajectory tracking controller, the loss function used is as follows

[0029]

[0030] Among them, N p is the prediction time domain; N c is the control time domain; ρ is the weight coefficient; ε is the relaxation factor;

[0031] Then let:

[0032]

[0033] The new state space expression is obtained as follows:

[0034]

[0035]

[0036] in, n is the state quantity dimension; m is the control quantity dimension;

[0037] After iterative derivation, the predicted output expression of the system is obtained:

[0038] Y(k)=ψξ(k)+ΘΔU(k)

[0039]

[0040] in,

[0041]

[0042] Then the loss function is expressed as:

[0043]

[0044] Where, E = Ψξ(k);

[0045] Considering the responsiveness of the motor and preventing the vehicle from rolling over, it is necessary to limit the increment of the driving wheel speed, the maximum value and the speed difference between the left and right sides;

[0046] First, the maximum torque that the motor can provide at the current speed is obtained through the external characteristic curve of the motor, and the maximum acceleration of the motor speed is obtained by combining the resistance torque in the current state;

[0047] Δu min (t+k)≤Δu(t+k)≤Δu max (t+k),

[0048] k=0,1,…,N c -1

[0049] Among them, Δu is the change of the control quantity in each time step;

[0050] Secondly, it is necessary to limit the maximum speed and speed difference of the active wheel speed; that is, it is necessary to ensure that the control amount output each time is within the constraints during the model prediction process;

[0051] Then the constraints are:

[0052]

[0053] in

[0054]

[0055]

[0056]

[0057]

[0058] This constraint condition constrains both the speed value and the speed difference between the left and right sides; where m is the control dimension, represents the Kronecker product; u(k-1) is the control quantity at the previous moment, u1(k-1), u2(k-1) represent the first and second elements in the control quantity, i.e., the left and right wheel speeds; U max ,U wmaxThey are respectively the maximum value of the driving wheel speed and the maximum value of the speed difference of the driving wheels on both sides.

[0059] Among them, the step three: determine the observation quantity as the deviation of the vehicle position and attitude, the vehicle yaw angular velocity and the position and attitude of the previous moment, using state quantities including the error between the current position and attitude predicted by the kinematic model and the actual position and attitude at the current moment; determine the output quantity as the difference in active wheel speed on both sides; and design a reward function.

[0060] Among them, the step 4: designing the structure of the Critic network and the Actor network, the Critic network merges the state space and the action space, and outputs the Q value through several layers of fully connected hidden layers;

[0061] During training, the Critic network is updated more frequently than the Actor network, and the target network is updated using soft updates:

[0062] w′←τw+(1-τ)w′

[0063] φ′←τφ+(1-τ)φ′

[0064] During the training process, the agent takes out a set of four-tuples from the experience replay pool. The four-tuple contains the state information s including the position and posture deviation of the tracked vehicle at the current moment. The Actor network will give an output action a based on the current state information, that is, to compensate for the speed difference, the reward r given by the environment, and the state information s′ at the next moment. The agent will calculate the two estimated Q values ​​Q1 and Q2 at the current moment through the two Critic networks based on (s, a), and then calculate the two Q values ​​Q1′ and Q2′ at the next moment through the Target Critic network based on the state s′ at the next moment and the action value μ′(s′; φ)+ε after adding noise. The smaller Q value is added to the reward value to get the actual Q value at the current moment, and the TD error of the two Critic networks is calculated respectively. The Critic network is updated by the gradient descent of the loss function:

[0065]

[0066]

[0067] Among them, μ′(…;φ) represents the TargetActor network, Q′(…;w Q′ ) represents the Target Critic network, μ′(s′; φ) represents the action to be output at the next moment, represents the Q value at the next moment estimated by the Target Critic network; y represents the target Q value, r is the reward of the environment, γ is the discount factor that determines the priority of short-term rewards, α is the learning rate, Q(s,a;w) represents the Q value at the current moment estimated by the Critic network, s′,a′ represents the state space and action space at the next moment, and Q(s′,a′;w) represents the Q value at the next moment estimated by the Critic network;

[0068] After the Critic network is updated several times, the Actor network is updated using gradient ascent. During the online process, only the Actor network participates in the calculation, and the compensation speed difference is given in real time according to the current tracked vehicle error, which avoids excessive calculation due to the dual Critic networks.

[0069] Among them, the step six is: exporting the trained intelligent agent for online calculation, and solving the required compensation output value in real time.

[0070] Among them, the method builds an MPC controller and designs an output compensation module based on a deep reinforcement learning algorithm for the MPC controller.

[0071] The described method is used for the control of model-free, nonlinear control systems.

[0072] The output compensation module uses environmental information, including vehicle position and attitude, to directly compensate the output of the MPC controller.

[0073] In an offline state, the output compensation module learns the parameters of the compensation module through the environment;

[0074] In the online state, the output compensation module performs output compensation on the MPC controller through observations in real time.

[0075] (III) Beneficial effects

[0076] In order to solve the problems of the prior art, the present invention adopts deep reinforcement learning as the output compensation of the MPC controller and provides a track tracking accuracy optimization method for tracked vehicles based on deep reinforcement learning. The method specifically involves a method for building an MPC controller, a method for designing a deep reinforcement learning algorithm, and an error compensation method. The method improves the accuracy of track tracking of tracked vehicles. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] Figure 1 Schematic diagram of the MPC controller based on deep reinforcement learning.

[0078] Figure 2 A schematic diagram of an MPC controller implementation based on TD3 compensation.

[0079] Figure 3Schematic diagram of the deviation of the pure MPC controller in the X and Y axis directions.

[0080] Figure 4 Schematic diagram of the deviation in the X and Y axis directions of the MPC controller based on TD3 compensation. DETAILED DESCRIPTION

[0081] In order to make the purpose, content, and advantages of the present invention more clear, the specific implementation methods of the present invention are further described in detail below in conjunction with the accompanying drawings and examples.

[0082] In order to solve the above technical problems, the present invention provides a method for optimizing track tracking accuracy of a tracked vehicle based on deep reinforcement learning, the method comprising:

[0083] Step 1: A kinematic model based on the instantaneous steering center is established according to the tracked vehicle, the kinematic model is expanded in the first order Taylor at the reference point, the coefficient matrices A and B of the state space equation are calculated using the Jacobian matrix, and the nonlinear system is linearized;

[0084] Step 2: Set the prediction time domain and control time domain based on experience, predict the future trajectory, determine the loss function J, use the parameters in the prediction process to represent the loss function, set constraints to consider the response characteristics of the motor and the lateral stability of the vehicle, use the quadratic programming solver to calculate the optimal control sequence ΔU, use the first set of control quantities Δu(t), and sum the past control quantities with the solved Δu(t) and output them;

[0085] Step 3: Design deep reinforcement learning state and action quantities; select vehicle state parameters as observation quantities, determine compensation parameters as output quantities, and design reward function r;

[0086] Step 4: Design the structure of the deep reinforcement learning Critic network and Actor network, build an intelligent agent based on the deep reinforcement learning algorithm, and set training parameters;

[0087] Step 5: Train the agent; adjust the reward function according to the training results, and perform multiple rounds of iterative optimization until the ideal training effect is achieved;

[0088] Step 6: Export the trained intelligent agent for online calculation, calculate the required compensation output value in real time, add it to the control amount output in step 2, and use it as the input of the underlying execution system; at this point, the compensation of the control amount is completed, and real-time optimization of trajectory tracking control is achieved.

[0089] Wherein, in the step 1, a kinematic model based on the instantaneous steering center is established according to the tracked vehicle.

[0090]

[0091] The position and attitude of the vehicle in the global coordinate system are (X, Y, θ), where θ is the heading angle of the tracked vehicle; in the local body coordinate system, v x ,v y ,ω z are the longitudinal velocity, lateral velocity and yaw rate of the vehicle respectively; (x c ,y c ) is the turning center of the crawler vehicle, (x l ,y l ) is the instantaneous turning center of the left track of the tracked vehicle, (x r ,y r ) is the instantaneous turning center of the right track of the tracked vehicle, They are the winding speed on the left side and the winding speed on the right side of the track respectively;

[0092] Expand the kinematic model in the first order Taylor at the reference point, use the Jacobian matrix to find the coefficient matrix of the state space equation, linearize the nonlinear system, and then discretize it to obtain the state space equation:

[0093]

[0094] Where X = [X, Y, θ] T , is the state quantity, To control the amount;

[0095] It needs to be linearized and discretized; the kinematic equation is Taylor expanded at the reference point and discretized by forward Euler to obtain

[0096]

[0097] in,

[0098]

[0099]

[0100] Among them, T s is the sampling time, is the reference value of the left and right driving wheel speed, θ d is the reference value of the heading angle.

[0101] Among them, the step 2: build an MPC controller; when designing the trajectory tracking controller, the loss function used is as follows

[0102]

[0103] Among them, N p is the prediction time domain; N c is the control time domain; ρ is the weight coefficient; ε is the relaxation factor;

[0104] Then let:

[0105]

[0106] The new state space expression is obtained as follows:

[0107]

[0108]

[0109] in, n is the state dimension; m is the control dimension; after iterative derivation, the predicted output expression of the system is obtained:

[0110] Y(k)=ψξ(k)+ΘΔU(k)

[0111]

[0112] in,

[0113]

[0114] Then the loss function is expressed as:

[0115]

[0116] Where, E = Ψξ(k);

[0117] Considering the responsiveness of the motor and preventing the vehicle from rolling over, it is necessary to limit the increment of the driving wheel speed, the maximum value and the speed difference between the left and right sides;

[0118] First, the maximum torque that the motor can provide at the current speed is obtained through the external characteristic curve of the motor, and the maximum acceleration of the motor speed is obtained by combining the resistance torque in the current state;

[0119] Δu min (t+k)≤Δu(t+k)≤Δu max (t+k),

[0120] k=0,1,…,N c -1

[0121] Among them, Δu is the change of the control quantity in each time step;

[0122] Secondly, it is necessary to limit the maximum speed and speed difference of the active wheel speed; that is, it is necessary to ensure that the control amount output each time is within the constraints during the model prediction process;

[0123] Then the constraints are:

[0124]

[0125] in

[0126]

[0127]

[0128]

[0129]

[0130] This constraint condition constrains both the speed value and the speed difference between the left and right sides; where m is the control dimension, represents the Kronecker product; u(k-1) is the control quantity at the previous moment, u1(k-1), u2(k-1) represent the first and second elements in the control quantity, i.e., the left and right wheel speeds; U max ,U wmax They are respectively the maximum value of the driving wheel speed and the maximum value of the speed difference of the driving wheels on both sides.

[0131] Among them, the step three: determine the observation quantity as the deviation of the vehicle position and attitude, the vehicle yaw angular velocity and the position and attitude of the previous moment, using state quantities including the error between the current position and attitude predicted by the kinematic model and the actual position and attitude at the current moment; determine the output quantity as the difference in active wheel speed on both sides; and design a reward function.

[0132] Among them, the step 4: designing the structure of the Critic network and the Actor network, the Critic network merges the state space and the action space, and outputs the Q value through several layers of fully connected hidden layers;

[0133] During training, the Critic network is updated more frequently than the Actor network, and the target network is updated using soft updates:

[0134] w′←τw+(1-τ)w′

[0135] φ′←τφ+(1-τ)φ′

[0136] During the training process, the agent takes out a set of four-tuples from the experience replay pool. The four-tuple contains the state information s including the position and posture deviation of the tracked vehicle at the current moment. The Actor network will give an output action a based on the current state information, that is, to compensate for the speed difference, the reward r given by the environment, and the state information s′ at the next moment. The agent will calculate the two estimated Q values ​​Q1 and Q2 at the current moment through the two Critic networks based on (s, a), and then calculate the two Q values ​​Q1′ and Q2′ at the next moment through the Target Critic network based on the state s′ at the next moment and the action value μ′(s′; φ)+ε after adding noise. The smaller Q value is added to the reward value to get the actual Q value at the current moment, and the TD error of the two Critic networks is calculated respectively. The Critic network is updated by the gradient descent of the loss function:

[0137]

[0138]

[0139] Among them, μ′(…;φ) represents the TargetActor network, Q′(…;w Q′ ) represents the Target Critic network, μ′(s′; φ) represents the action to be output at the next moment, represents the Q value at the next moment estimated by the Target Critic network; y represents the target Q value, r is the reward of the environment, γ is the discount factor that determines the priority of short-term rewards, α is the learning rate, Q(s,a;w) represents the Q value at the current moment estimated by the Critic network, s′,a′ represents the state space and action space at the next moment, and Q(s′,a′;w) represents the Q value at the next moment estimated by the Critic network;

[0140] After the Critic network is updated several times, the Actor network is updated using gradient ascent. During the online process, only the Actor network participates in the calculation, and the compensation speed difference is given in real time according to the current tracked vehicle error, which avoids excessive calculation due to the dual Critic networks.

[0141] Among them, the step six is: exporting the trained intelligent agent for online calculation, and solving the required compensation output value in real time.

[0142] Among them, the method builds an MPC controller and designs an output compensation module based on a deep reinforcement learning algorithm for the MPC controller.

[0143] The described method is used for the control of model-free, nonlinear control systems.

[0144] The output compensation module uses environmental information, including vehicle position and attitude, to directly compensate the output of the MPC controller.

[0145] In an offline state, the output compensation module learns the parameters of the compensation module through the environment;

[0146] In the online state, the output compensation module performs output compensation on the MPC controller through observations in real time.

[0147] Example 1

[0148] According to the present invention, a specific embodiment is as follows for optimizing the track tracking control effect of a tracked vehicle:

[0149] The control block diagram of the specific embodiment is shown in Figure 2 .

[0150] In step 1, a kinematic model based on the instantaneous steering center is established according to the tracked vehicle.

[0151]

[0152] The position and attitude of the vehicle in the global coordinate system are (X, Y, θ), where θ is the heading angle of the tracked vehicle; in the local body coordinate system, v x ,v y ,ω z are the longitudinal velocity, lateral velocity and yaw rate of the vehicle respectively; (x c ,y c ) is the turning center of the crawler vehicle, (x l ,y l ) is the instantaneous turning center of the left track of the tracked vehicle, (x r ,y r ) is the instantaneous turning center of the right track of the tracked vehicle, They are the winding speed on the left side and the winding speed on the right side of the track respectively;

[0153] Expand the kinematic model in the first order Taylor at the reference point, use the Jacobian matrix to find the coefficient matrix of the state space equation, linearize the nonlinear system, and then discretize it to obtain the state space equation:

[0154]

[0155] Where X = [X, Y, θ] T , is the state quantity, To control the amount;

[0156] It needs to be linearized and discretized; the kinematic equation is Taylor expanded at the reference point and discretized by forward Euler to obtain

[0157]

[0158] in,

[0159]

[0160]

[0161] Among them, T s is the sampling time, is the reference value of the left and right driving wheel speed, θ d is the reference value of the heading angle.

[0162] Step 2: Build an MPC controller; when designing the trajectory tracking controller, the loss function used is as follows

[0163]

[0164] Among them, N p is the prediction time domain; N c is the control time domain; ρ is the weight coefficient; ε is the relaxation factor;

[0165] Then let:

[0166]

[0167] The new state space expression is obtained as follows:

[0168]

[0169]

[0170] in, n is the state quantity dimension; m is the control quantity dimension;

[0171] After iterative derivation, the predicted output expression of the system is obtained:

[0172] Y(k)=ψξ(k)+ΘΔU(k)

[0173]

[0174] in,

[0175]

[0176] Then the loss function can be expressed as:

[0177]

[0178] Where, E = Ψξ(k);

[0179] Considering the responsiveness of the motor and preventing the vehicle from rolling over, it is necessary to limit the increment of the driving wheel speed, the maximum value and the speed difference between the left and right sides;

[0180] First, the maximum torque that the motor can provide at the current speed is obtained through the external characteristic curve of the motor, and the maximum acceleration of the motor speed is obtained by combining the resistance torque in the current state;

[0181] Δu min (t+k)≤Δu(t+k)≤Δu max (t+k),

[0182] k=0,1,…,N c -1

[0183] Among them, Δu is the change of the control quantity in each time step;

[0184] Secondly, it is necessary to limit the maximum speed and speed difference of the active wheel speed; that is, it is necessary to ensure that the control amount output each time is within the constraints during the model prediction process;

[0185] Then the constraints are:

[0186]

[0187] in

[0188]

[0189]

[0190]

[0191]

[0192] This constraint can constrain the speed value and the speed difference between the left and right sides at the same time; where m is the control quantity dimension, represents the Kronecker product; u(k-1) is the control quantity at the previous moment, u1(k-1), u2(k-1) represent the first and second elements in the control quantity, i.e., the left and right wheel speeds; U max ,U wmax They are respectively the maximum value of the driving wheel speed and the maximum value of the speed difference of the driving wheels on both sides.

[0193] Step 3: Determine the observed quantity as the deviation of the vehicle's position and attitude, the vehicle's yaw rate and the position and attitude of the previous moment, using state quantities including the error between the current position and attitude predicted by the kinematic model and the actual position and attitude at the current moment; determine the output quantity as the difference in active wheel speed on both sides; and design a reward function.

[0194] Step 4: Design the structure of the Critic network and the Actor network. The Critic network merges the state space and the action space and outputs the Q value through several layers of fully connected hidden layers.

[0195] During training, the Critic network is updated more frequently than the Actor network, and the target network is updated using soft updates:

[0196] w′←τw+(1-τ)w′

[0197] φ′←τφ+(1-τ)φ′

[0198] During the training process, the agent takes out a set of four-tuples from the experience replay pool. The four-tuple contains the state information s including the position and posture deviation of the tracked vehicle at the current moment. The Actor network will give an output action a based on the current state information, that is, to compensate for the speed difference, the reward r given by the environment, and the state information s′ at the next moment. The agent will calculate the two estimated Q values ​​Q1 and Q2 at the current moment based on (s, a) through the two Critic networks, and then calculate the two Q values ​​Q1′ and Q2′ at the next moment based on the state s′ at the next moment and the action value μ′(s′; φ)+ε after adding noise through the Target Critic network. The smaller Q value among them is added to the reward value to get the actual Q value at the current moment, and the TD error of the two Critic networks is calculated respectively. The Critic network is updated by the gradient descent of the loss function:

[0199]

[0200]

[0201] Among them, μ′(…;φ) represents the TargetActor network, Q′(…;w Q′ ) represents the Target Critic network, μ′(s′; φ) represents the action to be output at the next moment, represents the Q value at the next moment estimated by the Target Critic network; y represents the target Q value, r is the reward of the environment, γ is the discount factor that determines the priority of short-term rewards, α is the learning rate, Q(s,a;w) represents the Q value at the current moment estimated by the Critic network, s′,a′ represents the state space and action space at the next moment, and Q(s′,a′;w) represents the Q value at the next moment estimated by the Critic network;

[0202] After the Critic network is updated several times, the Actor network is updated using gradient ascent. During the online process, only the Actor network participates in the calculation, and the compensation speed difference is given in real time according to the current tracked vehicle error, which avoids excessive calculation due to the dual Critic networks.

[0203] Step 5: Train the agent; adjust the reward function according to the training results, and perform multiple rounds of iterative optimization until the ideal training effect is achieved;

[0204] Step 6: Export the trained agent for online calculation and solve the required compensation output value in real time. Add it to the control quantity output by the MPC controller in step 2 as the input of the underlying execution system; at this point, the compensation of the control quantity is completed and the real-time optimization of trajectory tracking control is achieved.

[0205] The designed compensation module based on deep reinforcement learning improves the accuracy of track tracking of tracked vehicles. Figure 3 and Figure 4 shown.

[0206] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A track tracking accuracy optimization method for tracked vehicles based on deep reinforcement learning, characterized in that: The method comprises: Step 1: A kinematic model based on the instantaneous steering center is established according to the tracked vehicle, the kinematic model is expanded in the first order Taylor at the reference point, the coefficient matrices A and B of the state space equation are calculated using the Jacobian matrix, and the nonlinear system is linearized; Step 2: Set the prediction time domain and control time domain based on experience, predict the future trajectory, determine the loss function J, use the parameters in the prediction process to represent the loss function, set constraints to consider the response characteristics of the motor and the lateral stability of the vehicle, use the quadratic programming solver to calculate the optimal control sequence ΔU, use the first set of control quantities Δu(t), and sum the past control quantities with the solved Δu(t) and output them; Step 3: Design deep reinforcement learning state and action quantities; select vehicle state parameters as observation quantities, determine compensation parameters as output quantities, and design reward function r; Step 4: Design the structure of the deep reinforcement learning Critic network and Actor network, build an intelligent agent based on the deep reinforcement learning algorithm, and set training parameters; Step 5: Train the agent; adjust the reward function according to the training results, and perform multiple rounds of iterative optimization until the ideal training effect is achieved; Step 6: Export the trained intelligent agent for online calculation, calculate the required compensation output value in real time, add it to the control amount output in step 2, and use it as the input of the underlying execution system; at this point, the compensation of the control amount is completed, and real-time optimization of trajectory tracking control is achieved.

2. The track tracking accuracy optimization method for tracked vehicles based on deep reinforcement learning according to claim 1, characterized in that: In the step 1, a kinematic model based on the instantaneous steering center is established according to the tracked vehicle. The position and attitude of the vehicle in the global coordinate system are (X, Y, θ), where θ is the heading angle of the tracked vehicle; in the local body coordinate system, v x ,v y ,ω z are the longitudinal velocity, lateral velocity and yaw rate of the vehicle respectively; (x c ,y c ) is the turning center of the crawler vehicle, (x l ,y l ) is the instantaneous turning center of the left track of the tracked vehicle, (x r ,y r ) is the instantaneous turning center of the right track of the tracked vehicle, They are the winding speed on the left side and the winding speed on the right side of the track respectively; Expand the kinematic model in the first order Taylor at the reference point, use the Jacobian matrix to find the coefficient matrix of the state space equation, linearize the nonlinear system, and then discretize it to obtain the state space equation: Where X = [X, Y, θ] T , is the state quantity, To control the amount; It needs to be linearized and discretized; the kinematic equation is Taylor expanded at the reference point and discretized by forward Euler to obtain in, Among them, T s is the sampling time, is the reference value of the left and right driving wheel speed, θ d is the reference value of the heading angle.

3. The track tracking accuracy optimization method for tracked vehicles based on deep reinforcement learning according to claim 2, characterized in that: Step 2: Building an MPC controller; When designing a trajectory tracking controller, the loss function used is as follows Among them, N p is the prediction time domain; N c is the control time domain; ρ is the weight coefficient; ε is the relaxation factor; Then let: The new state space expression is obtained as follows: in, n is the state quantity dimension; m is the control quantity dimension; After iterative derivation, the predicted output expression of the system is obtained: Y(k)=ψξ(k)+ΘΔU(k) in, Then the loss function is expressed as: Where, E = Ψξ(k); Considering the responsiveness of the motor and preventing the vehicle from rolling over, it is necessary to limit the increment of the driving wheel speed, the maximum value and the speed difference between the left and right sides; First, the maximum torque that the motor can provide at the current speed is obtained through the external characteristic curve of the motor, and the maximum acceleration of the motor speed is obtained by combining the resistance torque in the current state; Δu min (t+k)≤Δu(t+k)≤Δu max (t+k), k=0,1,…,N c -1 Among them, Δu is the change of the control quantity in each time step; Secondly, it is necessary to limit the maximum speed and speed difference of the active wheel speed; that is, it is necessary to ensure that the control amount output each time during the model prediction process is within the constraints; Then the constraints are: in This constraint condition constrains both the speed value and the speed difference between the left and right sides; where m is the control dimension, represents the Kronecker product; u(k-1) is the control quantity at the previous moment, u1(k-1), u2(k-1) represent the first and second elements in the control quantity, i.e., the left and right wheel speeds; U max ,U wmax They are respectively the maximum value of the driving wheel speed and the maximum value of the speed difference of the driving wheels on both sides.

4. The track tracking accuracy optimization method for tracked vehicles based on deep reinforcement learning according to claim 3, characterized in that: The step three is as follows: determining the observed quantity as the deviation of the vehicle position and attitude, the vehicle yaw rate and the position and attitude at the previous moment, using state quantities including the error between the current position and attitude predicted by the kinematic model and the actual position and attitude at the current moment; determining the output quantity as the difference in active wheel speed on both sides; and designing a reward function.

5. The track tracking accuracy optimization method for tracked vehicles based on deep reinforcement learning according to claim 4, characterized in that: Step 4: Design the structures of the Critic network and the Actor network. The Critic network merges the state space and the action space and outputs the Q value through several layers of fully connected hidden layers. During training, the Critic network is updated more frequently than the Actor network, and the target network is updated using soft updates: w′←τw+(1-τ)w′ φ′←τφ+(1-τ)φ′ During the training process, the agent takes out a set of four-tuples from the experience replay pool. The four-tuple contains the state information s including the position and posture deviation of the tracked vehicle at the current moment. The Actor network will give an output action a based on the current state information, that is, to compensate for the speed difference, the reward r given by the environment, and the state information s′ at the next moment. The agent will calculate the two estimated Q values ​​Q1 and Q2 at the current moment through the two Critic networks based on (s, a), and then calculate the two Q values ​​Q′1 and Q′2 at the next moment through the Target Critic network based on the state s′ at the next moment and the action value μ′(s′; φ)+ε after adding noise. The smaller Q value is added to the reward value to get the actual Q value at the current moment, and the TD error of the two Critic networks is calculated respectively. The Critic network is updated by the gradient descent of the loss function: Among them, μ′(…;φ) represents the TargetActor network, Q′(…;w Q′ ) represents the Target Critic network, μ′(s′; φ) represents the action to be output at the next moment, represents the Q value at the next moment estimated by the Target Critic network; y represents the target Q value, r is the reward of the environment, γ is the discount factor that determines the priority of short-term rewards, α is the learning rate, Q(s,a;w) represents the Q value at the current moment estimated by the Critic network, s′,a′ represents the state space and action space at the next moment, and Q(s′,a′;w) represents the Q value at the next moment estimated by the Critic network; After the Critic network is updated several times, the Actor network is updated using gradient ascent. During the online process, only the Actor network participates in the calculation, and the compensation speed difference is given in real time according to the current tracked vehicle error, which avoids excessive calculation due to the dual Critic networks.

6. The track tracking accuracy optimization method for tracked vehicles based on deep reinforcement learning according to claim 5, characterized in that: The step six is ​​to export the trained intelligent agent for online calculation and calculate the required compensation output value in real time.

7. The track tracking accuracy optimization method for tracked vehicles based on deep reinforcement learning according to claim 6, characterized in that: The method builds an MPC controller and designs an output compensation module based on a deep reinforcement learning algorithm for the MPC controller.

8. The track tracking accuracy optimization method for tracked vehicles based on deep reinforcement learning according to claim 7, characterized in that: The described method is used for the control of model-free, nonlinear control systems.

9. The track tracking accuracy optimization method for tracked vehicles based on deep reinforcement learning according to claim 8, characterized in that: The output compensation module uses environmental information, including vehicle position and attitude, to directly compensate the output of the MPC controller.

10. The track tracking accuracy optimization method for tracked vehicles based on deep reinforcement learning according to claim 9, characterized in that: In an offline state, the output compensation module learns the parameters of the compensation module through the environment; In the online state, the output compensation module performs output compensation on the MPC controller through observations in real time.

Images