Method, device, equipment and medium for adaptive terrain trajectory tracking of tracked vehicles based on deep reinforcement learning

Abstract

A method, device, equipment and medium for adaptive terrain trajectory tracking of a tracked vehicle based on deep reinforcement learning, relating to the field of trajectory tracking technology. This trajectory tracking method first constructs an action space and a state space, and designs a reward function that comprehensively considers position error, heading deviation and speed deviation. Then, a training and target Actor-Critic network is established to generate action strategies and evaluate the value of state-action pairs. Experience samples are obtained by allowing the vehicle to interact with the environment, and the training network and the target network are updated accordingly to continuously optimize the control strategy. This iterative process continues until the convergence criterion is met or the predetermined number of training times is reached, and the optimal Actor network is finally obtained. Finally, the optimal Actor network is used to control the vehicle to travel along the predetermined trajectory, achieving efficient and stable trajectory tracking. This method significantly enhances the adaptability and robustness of tracked vehicles in variable terrains, and improves the trajectory tracking performance of vehicles under complex terrain conditions.

Description

Technical Field

[0001] The present invention relates to the field of trajectory tracking technology, and in particular to a method, device, equipment and medium for adaptive terrain trajectory tracking of a tracked vehicle based on deep reinforcement learning. Background Art

[0002] In the field of modern mechanical engineering, tracked vehicles are widely used in military, agriculture, construction, rescue and other fields due to their excellent passability and load-bearing capacity in complex terrain. However, the motion performance of tracked vehicles under changing terrain and diverse surface materials, especially trajectory tracking technology, has always been the focus and difficulty of research. The goal of trajectory tracking technology is to ensure that the vehicle can travel along the predetermined path, effectively avoid obstacles, and accurately reach the destination.

[0003] Traditional trajectory tracking methods mainly rely on predefined rules, models or prior knowledge. These methods often have difficulty adjusting parameters or strategies flexibly when faced with complex and changing environments, resulting in poor adaptability and robustness of trajectory tracking. In addition, the performance of existing methods is highly dependent on initial parameter settings and engineering experience, lacks self-learning capabilities, cannot be adaptively adjusted according to environmental changes or changes in device status, and is difficult to achieve optimal performance in a variety of scenarios.

[0004] In view of this, the applicant filed this application after studying the existing technology. Summary of the invention

[0005] The present invention provides a method, device, equipment and medium for adaptive terrain trajectory tracking of a tracked vehicle based on deep reinforcement learning to improve at least one of the above-mentioned technical problems.

[0006] In a first aspect, the present invention provides a method for adaptive terrain trajectory tracking of a tracked vehicle based on deep reinforcement learning, which comprises steps S01 to S11.

[0007] S01. Construct an action space for a tracked vehicle.

[0008] S02. Define the state space of the tracked vehicle.

[0009] S03. Design a reward function for tracked vehicles.

[0010] S04. Establish a training Actor network and a training Critic network, as well as the corresponding target Actor network and target Critic network.

[0011] S05. According to the state space, the action space, the reward function and the trained Actor network, the tracked vehicle is controlled to interact with the environment to obtain experience samples.

[0012] S06. Update and train the Critic network based on the experience samples.

[0013] S07. Update the target critic network according to the updated training critic network.

[0014] S08. Update the training Actor network according to the updated training Critic network and the experience samples.

[0015] S09. Update the target Actor network according to the updated training Actor network.

[0016] S10, repeating steps S05 to S09 to obtain experience samples, update the training Critic network, update the target Critic network, update the training Actor network, and update the target Actor network until a preset convergence standard is reached, or the total number of training rounds reaches a preset value, and the optimal Actor network is obtained.

[0017] S11. Obtain a target trajectory of the tracked vehicle, and control the tracked vehicle to move along the target trajectory through the optimal Actor network.

[0018] In an optional embodiment, the action space includes the left track speed and right track speed The action variables constituted by the speed adjustment range of the left and right tracks, and the action range constituted by the speed adjustment range of the left and right tracks [ , ].

[0019] In an optional embodiment, the state space It includes position state variables, heading state variables and speed state variables. The position state variables are determined by the deviation of the vehicle from the target trajectory in the lateral direction. , and the vehicle's current deviation from the target trajectory in the longitudinal direction The heading state variable is the deviation between the current vehicle heading angle and the expected heading angle of the target trajectory. The speed state variable is the difference between the current left track speed of the vehicle and the target speed , and the difference between the vehicle's current right track speed and the target speed constitute.

[0020] In an optional embodiment, the reward function includes a trajectory error reward function, a control cost reward function and an optimization reward function.

[0021] Trajectory Error Reward Function :

[0022] ;

[0023] In the formula, is the weight of the Euclidean distance error, is the Euclidean distance error, is the weight of heading deviation, is the heading deviation, is the weight of speed deviation, is the speed deviation.

[0024] Control Cost Reward Function :

[0025] ;

[0026] In the formula, is the action at the current moment, The action at the previous moment.

[0027] Optimizing the Reward Function :

[0028] ;

[0029] ;

[0030] ;

[0031] In the formula, is a proportionality factor, is the base of natural logarithms, is the area of ​​the coupling region, is the straight-line distance between the current position and the target trajectory point, The deviation between the current vehicle heading angle and the desired heading angle of the target trajectory, is the current deviation of the vehicle from the target trajectory in the lateral direction, is the current deviation of the vehicle from the target trajectory in the longitudinal direction.

[0032] In an optional implementation, step S04 specifically includes steps S041 to S046.

[0033] S041. Establish a training Actor network and a training Critic network. The training Actor network is used to generate action strategies taken in a specific state, and the training Critic network is used to evaluate the value of a specific state-action pair (ie, Q value).

[0034] S042. Set a target Actor network for training the Actor network, and set a target Critic network for training the Critic network.

[0035] S043. Randomly initialize the parameter weights of the training Actor and the training Critic networks to set the neural network parameters.

[0036] S044. Copy the network parameters of the training Actor and the training Critic network to the target Actor network and the target Critic network.

[0037] S045. Set hyper parameters, wherein the hyper parameters include learning rate, discount factor γ, experience replay pool size, batch size, and soft update coefficient.

[0038] S046. Set and initialize an experience replay pool for storing experience samples obtained during the interaction between the intelligent agent and the environment.

[0039] In an optional implementation, step S05 specifically includes step S051 to step S055.

[0040] S051. Obtain a target position on a target trajectory.

[0041] S052, according to the target position, obtaining the position deviation and heading deviation between the real-time position of the tracked vehicle and the target position, and obtaining the speed deviation between the real-time track speed of the tracked vehicle and the target track speed, to form the current state of the tracked vehicle .

[0042] S053, input the current state into the training Actor network to obtain the current action that conforms to the action space .

[0043] S054, after the tracked vehicle performs the current moment action, obtain the position deviation and heading deviation between the real-time position of the tracked vehicle and the target position, and obtain the speed deviation between the real-time track speed of the tracked vehicle and the target track speed, to form the next moment state of the tracked vehicle .

[0044] Obtain an immediate reward based on the next moment state, the current moment action and the reward function .

[0045] S055, the current state, the next state, the instant reward and the current action are combined into a four-tuple experience sample , stored in the experience replay pool.

[0046] In an optional implementation, step S06 specifically includes steps S061 to S063.

[0047] S061. Randomly sample from an experience replay pool storing four-tuple experience samples to obtain a data set for network training.

[0048] S062. Calculate the target Q value through the target Actor network and the target Critic network according to the data set. The calculation model of the target Q value is:

[0049] ;

[0050] In the formula, For the data set The target Q value of samples, For instant rewards, is the discount factor, is the output of the target Critic network, For the next moment state, is the target Actor network's strategy, Represents the parameters of the target Critic network.

[0051] S063. Update the training critic network according to the target Q value. The update model of the training critic network is:

[0052] ;

[0053] In the formula, is the updated training parameters of the Critic network, is the size of the dataset, is the serial number of the sample in the dataset, For the data set The target Q value of samples, is the action value function, is the current state, Indicates the current action, It is the parameter for training the Critic network.

[0054] In an optional implementation, step S07 specifically includes: updating the training Actor network using a gradient ascent method according to the updated training Critic network and the experience sample. The update model of the training Actor network is:

[0055] ;

[0056] In the formula, Represents the updated parameters of the training Actor network, represents the gradient, Indicates action The gradient of the action-value function, represents the policy gradient of the output of the training Actor network, is the action value function, Indicates status, Indicates action, is the updated training parameters of the Critic network, Action functions for training Actor networks, are the parameters for training the Actor network, is the size of the dataset, is the sequence number of the samples in the dataset.

[0057] In an optional implementation, the update model of the target Critic network in step S08 is:

[0058] ;

[0059] In the formula, Represents the parameters of the target Critic network, is the update coefficient, are the updated parameters of the training Critic network.

[0060] In an optional implementation, the update model of the target Actor network in step S09 is:

[0061] ;

[0062] In the formula, Parameters of the target Actor network, is the update coefficient, Represents the updated parameters of the trained Actor network.

[0063] In an optional implementation, step S11 specifically includes steps S111 to S115.

[0064] S111, obtaining a target trajectory of the tracked vehicle, and repeatedly performing the following actions according to the target trajectory to control the movement of the tracked vehicle.

[0065] S112: Acquire a target position according to the target trajectory.

[0066] S113, obtaining the current position of the tracked vehicle, obtaining the position deviation and heading deviation between the current position and the target position of the tracked vehicle according to the target trajectory, the target position and the current position, and obtaining the speed deviation between the current track speed of the tracked vehicle and the target track speed, thereby constituting the current state of the tracked vehicle.

[0067] S114, inputting the current state into the optimal Actor network to obtain the current action that conforms to the action space.

[0068] S115. Control the movement of the tracked vehicle according to the current action.

[0069] In the second aspect, the present invention provides a tracked vehicle adaptive terrain trajectory tracking device based on deep reinforcement learning, which includes an action space construction module, a state space definition module, a reward function design module, a control machine network establishment module, an experience sample acquisition module, a first update module, a second update module, a third update module, a fourth update module, an iteration module and a control module.

[0070] Action space construction module, used to construct the action space of tracked vehicles.

[0071] The state space definition module is used to define the state space of the tracked vehicle.

[0072] Reward function design module, used to design reward functions for tracked vehicles.

[0073] The control machine network establishment module is used to establish the training Actor network and the training Critic network, as well as the corresponding target Actor network and the target Critic network.

[0074] The experience sample acquisition module is used to control the tracked vehicle to interact with the environment and acquire experience samples according to the state space, the action space, the reward function and the trained Actor network.

[0075] The first updating module is used to update the trained Critic network according to the experience samples.

[0076] The second updating module is used to update the target critic network according to the updated training critic network.

[0077] The third updating module is used to update the training Actor network according to the updated training Critic network and the experience sample.

[0078] The fourth update module is used to update the target Actor network according to the updated training Actor network.

[0079] The iteration module is used to repeat the above steps of interacting with the environment to obtain experience samples, updating the training Critic network, updating the target Critic network, updating the training Actor network, and updating the target Actor network until the preset convergence standard is reached or the total number of training rounds reaches the preset value to obtain the optimal Actor network.

[0080] The control module is used to obtain the target trajectory of the tracked vehicle and control the tracked vehicle to move along the target trajectory through the optimal Actor network.

[0081] In a third aspect, the present invention provides a tracked vehicle adaptive terrain trajectory tracking device based on deep reinforcement learning, comprising a processor, a memory, and a computer program stored in the memory. The computer program can be executed by the processor to implement a tracked vehicle adaptive terrain trajectory tracking method based on deep reinforcement learning as described in any paragraph of the first aspect.

[0082] In a fourth aspect, the present invention provides a computer-readable storage medium. The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute a method for adaptive terrain trajectory tracking of a tracked vehicle based on deep reinforcement learning as described in any paragraph of the first aspect.

[0083] By adopting the above technical solution, the present invention can achieve the following technical effects:

[0084] The tracked vehicle adaptive terrain trajectory tracking method based on deep reinforcement learning proposed in this invention significantly improves the adaptability and robustness of tracked vehicles in variable terrain. Through deep learning strategies, this method can autonomously learn and generate optimal control actions, adjust vehicle motion parameters in real time, and ensure that the vehicle stably follows the predetermined trajectory. In addition, this technology does not need to rely on accurate environmental models or prior knowledge, and exhibits strong adaptive capabilities and continuous optimization characteristics. It is suitable for many fields such as mechanical engineering, and realizes efficient trajectory tracking control in complex terrain. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the specific implementation methods of the present invention. It should be understood that the following drawings only show certain specific implementation methods of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0086] Figure 1 It is a flow chart of a method for adaptive terrain trajectory tracking of a tracked vehicle.

[0087] Figure 2 This is the logic block diagram of step S05.

[0088] Figure 3 It is a schematic diagram of optimizing the reward function.

[0089] Figure 4is a flowchart of the offline training process (step S05 to step S10).

[0090] Figure 5 is a flowchart of the line application process (step S11 ). DETAILED DESCRIPTION

[0091] The technical solutions in the embodiments of the present invention will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0092] Example 1, please refer to Figures 1 to 5 The first embodiment of the present invention provides a method for adaptive terrain trajectory tracking of a tracked vehicle based on deep reinforcement learning, which can be performed by a tracked vehicle adaptive terrain trajectory tracking device based on deep reinforcement learning (hereinafter referred to as: trajectory tracking device). In particular, it is performed by one or more processors in the trajectory tracking device to implement steps S01 to S11.

[0093] S01, constructing the action space of the tracked vehicle. Preferably, the action space includes the left track speed and right track speed The action variables constituted by the speed adjustment range of the left and right tracks, and the action range constituted by the speed adjustment range of the left and right tracks [ , ].

[0094] The construction of the action space determines the range of control instructions that the agent (i.e., the track tracking device + the tracked vehicle controlled by the track tracking device) can execute in different states. In order to achieve effective control of the tracked vehicle, the definition of the action space should cover all the adjustment measures that the vehicle may need to take in different states.

[0095] First, determine the type and range of control inputs. Then, define the dimensions of the action space based on these control inputs to ensure that all possible control operations are covered. Next, create an action selection mechanism, including the left and right track speeds of the vehicle. Finally, store these action information in a reasonable data structure so that they can be called during the decision-making process, and finally form a complete action space model.

[0096] The specific construction process is as follows: 1. Set the speed of the left and right tracks and As the action variable. By controlling the speed of the left and right tracks, the vehicle's steering and forward direction can be adjusted to accurately follow the target trajectory. 2. Define the action space consisting of the speed adjustment range of the left and right tracks. The control strategy is adjusted by and 3. Set the range of the action space to ensure that the output speed value is within the physically achievable range and control the speed within the maximum speed of the vehicle design. and minimum speed 4. Through the normalization operation of linear transformation, the speed range is changed from [ , ] maps to , ensuring the consistency of control instructions in algorithm training and actual application.

[0097] S02. Define the state space of the tracked vehicle. Preferably, the state space It includes position state variables, heading state variables and speed state variables. The position state variables are determined by the deviation of the vehicle from the target trajectory in the lateral direction. , and the vehicle's current deviation from the target trajectory in the longitudinal direction The heading state variable is the deviation between the current vehicle heading angle and the expected heading angle of the target trajectory. The speed state variable is the difference between the current left track speed of the vehicle and the target speed , and the difference between the vehicle's current right track speed and the target speed constitute.

[0098] In trajectory tracking control, the construction of the state space needs to be able to fully reflect the current state and dynamic characteristics of the system and provide sufficient information for the decision-making process of the intelligent agent. An ideal state representation should include the geometric properties (position, speed) of the mobile system and the deviation information between the target trajectory and the current trajectory.

[0099] First, the state variables related to trajectory tracking control, including the current position, speed, and direction angle, are extracted and converted into a state vector. Then, the dimension and range of the state space are set according to these state variables to ensure that the state of the vehicle in different scenarios can be fully described. Subsequently, a data structure is created to store this state information so that it can be effectively used by the learning algorithm to form a complete state space model.

[0100] The specific construction process is as follows: 1. The relative position deviation between the current vehicle position and the target trajectory is used as the position state variable. Specifically, and Respectively represent the current deviation of the vehicle from the target track in the lateral and longitudinal directions. By using relative position deviation, the system can more accurately capture the distance relationship between the vehicle and the target track without relying on the absolute coordinate system. 2. Select the vehicle's heading deviation As the heading state variable. That is, the deviation between the current vehicle heading angle and the expected heading angle of the target trajectory, which helps guide the agent to adjust the vehicle direction to drive along the target trajectory more accurately. 3. The deviation of the left and right track speeds is used as the speed state variable. That is, the difference between the current left and right track speeds and the target speed, and Represent the speed deviation of the left track and the right track respectively. These variables reflect the dynamic changes of the vehicle speed state, which helps the controller to better adjust the track speed and optimize the trajectory tracking effect. 4. Based on the state variables selected above, define the state vector: The state vector comprehensively reflects the real-time state information of the vehicle relative to the target trajectory, including deviations in position, direction, and speed.

[0101] S03. Design a reward function for tracked vehicles.

[0102] In the trajectory tracking task, a reasonable reward function is designed according to the trajectory tracking goal. The design of the reward function needs to consider multiple factors to guide the agent to learn the optimal strategy (learn the optimal trajectory tracking strategy) towards the expected goal in an offline environment to ensure that the agent can achieve accurate trajectory tracking in a complex environment.

[0103] Based on the above embodiment, in an optional embodiment of the present invention, the reward function includes a trajectory error reward function, a control cost reward function and an optimization reward function.

[0104] Trajectory Error Reward Function :

[0105] ;

[0106] In the formula, is the weight of the Euclidean distance error, is the Euclidean distance error, is the weight of heading deviation, is the heading deviation, is the weight of speed deviation, is the speed deviation.

[0107] Specifically, the trajectory error reward is used to encourage the agent to adjust its behavior to reduce the deviation. The position error reward is used to encourage the reduction of position deviation. The heading deviation reward is used to encourage the correct adjustment of the direction. Each reward function is assigned an appropriate weight, and by adjusting these weights, it is ensured that the reward function can effectively guide the behavior of the agent in different scenarios.

[0108] Control Cost Reward Function :

[0109] ;

[0110] In the formula, is the action at the current moment, It is the action at the previous moment (ie control signal).

[0111] Specifically, the control cost reward is used to ensure the continuity and stability of the control signal, evaluate the rate of change of the control signal, and encourage smooth and stable control. This formula encourages the agent to take smoother control actions by penalizing the absolute change of the control signal, thereby improving the stability and smoothness of trajectory tracking.

[0112] Optimizing the reward function By designing a complex reward function, it can more accurately reflect the actual state changes of the system, thereby improving the learning efficiency of the agent and the effect of strategy optimization. In order to strengthen the control of position error and heading deviation, effectively guide the agent to learn a better trajectory tracking strategy, and improve the convergence speed of the algorithm, the error coupling term is introduced to optimize the reward function, and the position error and heading deviation are combined for evaluation. The specific steps are as follows.

[0113] Determine the straight-line distance between the agent's current position and the target trajectory point, and use this distance as the basis for the error. Calculate the angle between this straight-line distance and the current heading angle, and use this angle as part of the heading deviation.

[0114] First, determine the straight-line distance between the current position of the agent and the target trajectory point

[0115] ;

[0116] In the formula, is the current deviation of the vehicle from the target trajectory in the lateral direction, is the current deviation of the vehicle from the target trajectory in the longitudinal direction.

[0117] Then, calculate the angle between the straight-line distance and the current heading angle, and set the heading deviation to , then the absolute value of the deviation Reflects the degree of deviation of the heading angle.

[0118] Then, build the geometry and calculate the area of ​​the coupling region :like Figure 3 As shown. Connect the current position and the forward point, then this straight line has an angle with the current heading, and this angle is the heading deviation. The length of this line segment is the position error. Take the line segment as the chord and the current heading as the tangent to make a circle, then the area formed by the chord and the circle is Figure 3 The area of ​​the green part in the middle is the coupling area: The coupling area can be regarded as a part of the area of ​​a circle with the position error as the boundary and the heading angle as the tangent angle: With the orientation of the agent as the tangent and the target trajectory point as the point on the circle, an arc geometry model is constructed. Calculate the area of ​​the coupling area between the line connecting the current position of the agent to the target trajectory point and the arc.

[0119] The calculation model of the coupling area is:

[0120] ;

[0121] In the formula, is the straight-line distance between the current position and the target trajectory point, The deviation between the current vehicle heading angle and the desired heading angle of the target trajectory.

[0122] Specifically, in order to evaluate the position error and heading deviation comprehensively, an arc geometric model is constructed to connect the current position and target position of the intelligent body. The area of ​​the coupling area formed by the geometric model is calculated, which reflects the comprehensive degree of the position error and heading deviation of the vehicle in the current state. The larger the area, the more serious the comprehensive deviation of the position error and heading deviation.

[0123] Introduce a coupling term into the reward function and calculate the area of ​​the coupling region The reward function is introduced to more accurately reflect the overall error. The size of the coupling area is used to update the reward function. A larger coupling area indicates a larger overall error, so a negative reward is given. A smaller coupling area indicates a smaller error, so a positive reward is given. In order to strengthen the effect of the reward, an exponential function can be used to adjust the immediate reward to enhance the guidance of the agent's behavior.

[0124] Optimizing the Reward Function :

[0125] ;

[0126] In the formula, is a proportional coefficient used to adjust the strength of the reward, is the base of natural logarithms, is the area of ​​the coupling region.

[0127] During training, the scaling factor and other parameters can be adjusted dynamically based on the agent’s performance to optimize the learning process.

[0128] Next, train the "Deep Deterministic Policy Gradient Controller (ie: Actor Network)" offline

[0129] S04, establishing a training Actor network and a training Critic network, as well as a corresponding target Actor network and a target Critic network. Preferably, step S04 specifically includes steps S041 to S046.

[0130] S041. Establish a training Actor network and a training Critic network. The training Actor network is used to generate action strategies taken in a specific state, and the training Critic network is used to evaluate the value of a specific state-action pair (ie, Q value).

[0131] S042. Set the target Actor network for training the Actor network and the target Critic network for training the Critic network. Specifically, in order to enhance the stability of training, create a target network for each main network (Actor and Critic). These target networks will update parameters at a slower speed during training to avoid violent shocks during model training and stabilize the training process.

[0132] S043. Randomly initialize the parameter weights of the training Actor and the training Critic networks to set the neural network parameters.

[0133] S044. Copy the network parameters of the training Actor and the training Critic network to the target Actor network and the target Critic network.

[0134] S045. Set hyper parameters, wherein the hyper parameters include learning rate, discount factor γ, experience replay pool size, batch size, and soft update coefficient.

[0135] Specifically, the learning rate is used for the learning rate of the Actor and Critic networks, the discount factor γ is used to weigh the relative importance of current rewards and future rewards in reinforcement learning, the experience replay pool size, the batch size is the number of samples drawn from the experience replay pool during each training, and the soft update coefficient is used to update the parameters of the target network.

[0136] S046. Set and initialize an experience replay pool for storing experience samples (state, action, reward, and next state) obtained during the interaction between the agent and the environment. The experience replay pool is used to train the network through random sampling, thereby reducing the correlation between data and improving training stability.

[0137] S05, according to the state space, the action space, the reward function and the trained Actor network, control the tracked vehicle to interact with the environment and obtain experience samples. Preferably, step S05 specifically includes steps S051 to S055.

[0138] S051. Obtain a target position on a target trajectory.

[0139] Specifically, a look-ahead value is set according to factors such as the vehicle's speed and turning radius. A point on the target trajectory that is a distance from the vehicle equal to the look-ahead value is selected as the look-ahead point, which is the target point that the vehicle needs to track next.

[0140] In this embodiment, the forward-looking The calculation formula is:

[0141] ;

[0142] In the formula, is the current speed of the vehicle, is the reaction time, i.e. the time required from recognizing the path change to executing the control command, is the maximum steering angular velocity of the vehicle.

[0143] S052, according to the target position, obtaining the position deviation and heading deviation between the real-time position of the tracked vehicle and the target position, and obtaining the speed deviation between the real-time track speed of the tracked vehicle and the target track speed, to form the current state of the tracked vehicle .

[0144] S053, input the current state into the training Actor network to obtain the current action that conforms to the action space .

[0145] S054, after the tracked vehicle performs the current moment action, obtain the position deviation and heading deviation between the real-time position of the tracked vehicle and the target position, and obtain the speed deviation between the real-time track speed of the tracked vehicle and the target track speed, to form the next moment state of the tracked vehicle .

[0146] Obtain an immediate reward based on the next moment state, the current moment action and the reward function .

[0147] S055, the current state, the next state, the instant reward and the current action are combined into a four-tuple experience sample , stored in the experience replay pool.

[0148] Specifically, during the training process, the agent performs a series of actions in the environment, and each action is generated by the training Actor network based on the current state (the agent decides to adjust the speed of the left and right tracks to follow the target trajectory). Every time the agent performs an action, the environment returns a feedback, including the next state and the immediate reward. The current state, the current action performed, the immediate reward obtained, and the next state (the relative position deviation, heading deviation, and speed deviation of the vehicle) are stored as experience data in the experience replay pool, and these data will be used for subsequent training.

[0149] Specifically, according to the current state of the agent Generate an action by training the Actor network , the agent performs the action , and observe the next state in the environment and instant rewards Then store the experience samples and convert the four-tuple samples Stored in the experience replay pool.

[0150] S06: Update the training critic network according to the experience samples. Preferably, step S06 specifically includes steps S061 to S063.

[0151] S061, randomly sample from the experience playback pool storing the four-tuple experience samples to obtain a data set for network training. The size of the data set is set when setting the hyperparameters in step S045, and may be tens or hundreds, which is not specifically limited in the present invention.

[0152] S062. Calculate the target Q value through the target Actor network and the target Critic network according to the data set. The calculation model of the target Q value is:

[0153] ;

[0154] In the formula, For the data set The target Q value of samples, For instant rewards, is the discount factor, is the output of the target Critic network, For the next moment state, is the target Actor network's strategy, Represents the parameters of the target Critic network.

[0155] Specifically, the target Q value is calculated through the target Critic network using the extracted sample data to update the parameters of the trained Critic network. Its goal is to predict the long-term return of the current state-action combination and adjust its own parameters to more accurately evaluate the value of different state-action combinations.

[0156] S063. Update the training critic network according to the target Q value. The update model of the training critic network is:

[0157] ;

[0158] In the formula, is the updated training parameters of the Critic network, is the size of the dataset, is the serial number of the sample in the dataset, For the data set The target Q value of samples, is the action value function, is the current state, Indicates the current action, It is the parameter for training the Critic network.

[0159] Specifically, the parameters are updated by minimizing the update model of the trained Critic network.

[0160] S07. Update the training Actor network according to the updated training Critic network and the experience samples.

[0161] After the training Critic network is updated, the feedback provided by the training Critic network is used to update the training Actor network. The goal is to learn a strategy, output an optimal action in each state (how to adjust the speed of the left and right tracks), adjust its own parameters according to the evaluation results of the training Critic network, and gradually learn how to take the best action in each state to maximize long-term rewards.

[0162] Based on the above embodiment, in an optional embodiment of the present invention, step S07 is specifically: according to the updated training Critic network and the experience sample, the training Actor network is updated using the gradient ascent method. The update model of the training Actor network is:

[0163] ;

[0164] In the formula, Represents the updated parameters of the training Actor network, represents the gradient, Indicates action The gradient of the action-value function, represents the policy gradient of the output of the training Actor network, is the action value function, Indicates status, Indicates action, is the updated training parameters of the Critic network, Action functions for training Actor networks, are the parameters for training the Actor network, is the size of the dataset, is the sequence number of the samples in the dataset.

[0165] Specifically, based on the feedback from the training Critic network, the gradient ascent method is used to update the Actor network parameters and optimize the strategy so that the actions it generates can maximize the expected cumulative reward. The gradient ascent method optimizes the parameters of the training Actor network through the above update model. The update of the parameters of the training Actor network is based on the gradient estimation of the current strategy by the training Critic network. The training Actor network adjusts the strategy according to the feedback from the training Critic network to maximize the Q value.

[0166] S08. Update the target critic network according to the updated training critic network. Preferably, the update model of the target critic network in step S08 is:

[0167] ;

[0168] In the formula, Represents the parameters of the target Critic network, is the update coefficient, are the updated parameters of the training Critic network.

[0169] S09, update the target Actor network according to the updated training Actor network. Preferably, the update model of the target Actor network in step S09 is:

[0170] ;

[0171] In the formula, Parameters of the target Actor network, is the update coefficient, Represents the parameters of the updated training Actor network.

[0172] In order to maintain the stability of training, the target network (i.e., the target Critic network and the target Actor network) will not be updated synchronously immediately, but a "soft update" method will be adopted. That is, the parameter values ​​of the main network are gradually integrated into the target network with a smaller step size. The soft update strategy is adopted to gradually adjust the parameters of the target network to make the update of the target network more stable, so as to prevent instability caused by drastic fluctuations in parameters during training.

[0173] Specifically, the parameters of the target network are not directly copied from the parameters of the main network every time, but are soft updated with a smaller step size through the above target network update model. is a small update coefficient, here set to 0.001, which is used to slowly integrate the parameters of the main network into the target network.

[0174] S10, repeat the above steps of interacting with the environment to obtain experience samples, updating the training Critic network, updating the target Critic network, updating the training Actor network, and updating the target Actor network (i.e., steps S05 to S09) until the preset convergence standard is reached, or the total number of training rounds reaches a preset value, and the optimal Actor network is obtained.

[0175] like Figure 4 The implementation steps of offline training of the deep deterministic policy gradient controller include the interaction between the agent and the environment, data collection, network optimization and policy update process.

[0176] By continuously repeating steps S05 to S09, interacting with the environment and extracting data from the experience replay pool for training, multiple iterations are performed, and the agent gradually learns how to choose the optimal action under different states until the policy network (training the Actor network) and the value network (training the Critic network) reach the preset convergence standard, or the total number of training rounds reaches the preset value, and the model parameters of the optimal Actor network and the optimal Critic network are obtained.

[0177] When the training reaches convergence or obtains satisfactory performance, the parameters of the trained optimal Actor network and the optimal Critic network are saved for subsequent online application and actual testing. These parameters will be used in the online application stage to control the tracked vehicle in the actual environment to achieve the best trajectory tracking effect under complex terrain conditions.

[0178] S11, obtaining the target trajectory of the tracked vehicle, and controlling the tracked vehicle to move along the target trajectory through the optimal Actor network. Specifically, the model parameters of the trained Actor network and Critic network are applied to the actual control system, the state information of the vehicle is collected in real time, the optimal control action is generated through the Actor network, and the vehicle is controlled to track the trajectory.

[0179] The online application of the deep deterministic policy gradient controller is the process of applying the offline trained control strategy (i.e. the model parameters of the optimal actor network and the optimal critic network) to the actual environment. In this stage, the previously offline trained model is used. In the actual environment, the controller makes the best decision and control instructions based on the real-time collected state information, such as Figure 5 shown.

[0180] On the basis of the above embodiment, in an optional embodiment of the present invention, step S11 specifically includes steps S111 to S115.

[0181] S111, obtaining a target trajectory of the tracked vehicle, and repeatedly performing the following actions according to the target trajectory to control the movement of the tracked vehicle.

[0182] Specifically, the model parameters of the Actor network of the optimal strategy model obtained in the offline training phase are loaded into the actual control system, and the control system is initialized to ensure that all sensors, actuators and network connections in the control system are working properly. The online control system can use these parameters to generate control actions in the actual environment.

[0183] It should be noted that, in this embodiment, only the optimal Actor network is used in the reasoning stage. In other embodiments, the optimal Critic network can also be applied to the reasoning stage. The present invention does not make specific limitations on this, and this solution also falls within the protection scope of the present invention.

[0184] S112, obtaining the target position according to the target trajectory. Specifically, this step is the same as step S051, except that S051 is a training stage and step S112 is an application stage.

[0185] S113, obtaining the current position of the tracked vehicle, obtaining the position deviation and heading deviation between the current position and the target position of the tracked vehicle according to the target trajectory, the target position and the current position, and obtaining the speed deviation between the current track speed of the tracked vehicle and the target track speed, thereby constituting the current state of the tracked vehicle.

[0186] Real-time collection of vehicle status information is used to reflect the current operating status of the vehicle. The state vector is defined as: ,in, is the current deviation of the vehicle from the target trajectory in the lateral direction, is the current deviation of the vehicle from the target trajectory in the longitudinal direction, The deviation between the current vehicle heading angle and the desired heading angle of the target trajectory, is the difference between the vehicle's current left track speed and the target speed, is the difference between the current right track speed of the vehicle and the target speed. Then the real-time collected state information is preprocessed (normalized and standardized) to make it conform to the input format of the network (the format of offline training data).

[0187] S114, inputting the current state into the optimal Actor network to obtain the current action that conforms to the action space.

[0188] The control system collects the vehicle's status information in real time, including the vehicle's current position, speed, and heading angle. The real-time collected status data is preprocessed to match the data format in the training phase. All of these data will be transmitted to the deep deterministic policy gradient controller as input.

[0189] The preprocessed state vector is input into the optimal Actor network, and the network calculates the optimal action for the current state (i.e., the speed of the left and right tracks) based on the current state. and ):

[0190] ;

[0191] In the formula, is the action at the current moment, Action functions for training Actor networks, is the current state, Parameters for training the Actor network.

[0192] The control action includes speed adjustment of the left and right tracks to optimize the vehicle's steering and forward direction to follow the target trajectory. Then, the action output by the optimal Actor network is restricted to ensure that it is within the physically feasible range or safe operating range. The speed of the left and right tracks is controlled at the maximum speed designed for the vehicle. and minimum speed Between, then, as Figure 5 As shown in , the data output by the optimal Actor network also needs to be processed by the activation function (Tanh function) and data scaling (scale function). Through the normalization operation of linear transformation, the speed range is changed from [ , ] maps to , ensuring the consistency of control instructions in algorithm training and actual application.

[0193] S115. Control the movement of the tracked vehicle according to the current action.

[0194] Based on the optimal action (the speed of the left and right tracks) calculated by the Actor network, a control command is generated and sent to the vehicle's drive system via a control signal (PWM signal or CAN bus signal). Then, the vehicle's execution is monitored in real time.

[0195] Based on real-time feedback data, the control strategy is dynamically adjusted to respond to environmental changes and ensure that the vehicle continues to effectively track the target trajectory. The controller continuously collects real-time feedback information from the vehicle, including position, speed, and direction, which will be used to evaluate whether the current control strategy is effective. If the vehicle deviates from the predetermined trajectory or exhibits unexpected behavior, the control system will adjust the action in real time. The above process of action calculation, execution, and feedback collection is repeated continuously, and the vehicle's motion state is continuously monitored and adjusted to ensure that it can complete tasks stably and efficiently in various environments.

[0196] It is understandable that the trajectory tracking device may be an electronic device with computing capabilities, such as a portable notebook computer, a desktop computer, a server, a smart phone, or a tablet computer.

[0197] The invention discloses a tracked vehicle adaptive terrain trajectory tracking method based on deep reinforcement learning, which significantly improves the track tracking capability of tracked vehicles in complex environments through deep learning technology. The core of the method is to use a deep deterministic policy gradient algorithm to autonomously optimize the control strategy through interactive learning between the intelligent agent and the environment to adapt to the changing terrain and environmental conditions. This method not only improves the adaptability and robustness of tracked vehicles, but also enhances their real-time performance and accuracy in unknown or dynamically changing environments.

[0198] By constructing the state and action space, the method of the present invention can fully capture the real-time state of the tracked vehicle and generate the optimal control action based on this state information. In addition, by designing a reasonable reward function, the intelligent agent can be guided to learn to reduce position deviation, heading deviation and control cost, thereby achieving more accurate trajectory tracking. The adaptability of this method is reflected in the ability to dynamically adjust the vehicle's motion parameters according to different terrain conditions and environmental changes, ensuring that the vehicle can stably follow the predetermined trajectory.

[0199] In the offline training phase, the state data is collected through interaction with the simulation environment, and the data is used to train and optimize the Actor network and the Critic network. In the online application phase, the trained model is applied to the actual control system, the vehicle state information is collected in real time, and the optimal control action is generated to achieve efficient trajectory tracking.

[0200] In general, the beneficial effect of this patent is to provide a method that can achieve efficient trajectory tracking control under complex terrain conditions, improving the autonomy and robustness of tracked vehicles. This method not only reduces the dependence on accurate environmental models or prior knowledge, but also enhances the vehicle's self-learning ability and continuous optimization ability when facing unknown environments, enabling it to achieve optimal performance in a variety of scenarios.

[0201] Embodiment 2: The present invention provides a tracked vehicle adaptive terrain trajectory tracking device based on deep reinforcement learning, which includes an action space construction module, a state space definition module, a reward function design module, a control machine network establishment module, an experience sample acquisition module, a first update module, a second update module, a third update module, a fourth update module, an iteration module and a control module.

[0202] Action space construction module, used to construct the action space of tracked vehicles.

[0203] The state space definition module is used to define the state space of the tracked vehicle.

[0204] Reward function design module, used to design reward functions for tracked vehicles.

[0205] The control machine network establishment module is used to establish the training Actor network and the training Critic network, as well as the corresponding target Actor network and the target Critic network.

[0206] The experience sample acquisition module is used to control the tracked vehicle to interact with the environment and acquire experience samples according to the state space, the action space, the reward function and the trained Actor network.

[0207] The first updating module is used to update the trained Critic network according to the experience samples.

[0208] The second updating module is used to update the target critic network according to the updated training critic network.

[0209] The third updating module is used to update the training Actor network according to the updated training Critic network and the experience sample.

[0210] The fourth update module is used to update the target Actor network according to the updated training Actor network.

[0211] The iteration module is used to repeat the above steps of interacting with the environment to obtain experience samples, updating the training Critic network, updating the target Critic network, updating the training Actor network, and updating the target Actor network until the preset convergence standard is reached or the total number of training rounds reaches the preset value to obtain the optimal Actor network.

[0212] The control module is used to obtain the target trajectory of the tracked vehicle and control the tracked vehicle to move along the target trajectory through the optimal Actor network.

[0213] Embodiment 3, the present invention provides a tracked vehicle adaptive terrain trajectory tracking device based on deep reinforcement learning, which includes a processor, a memory, and a computer program stored in the memory. The computer program can be executed by the processor to implement a tracked vehicle adaptive terrain trajectory tracking method based on deep reinforcement learning as described in any paragraph of Embodiment 1.

[0214] Embodiment 4: The present invention provides a computer-readable storage medium. The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute a method for adaptive terrain trajectory tracking of a tracked vehicle based on deep reinforcement learning as described in any paragraph of Embodiment 1.

[0215] In several embodiments provided in the embodiments of the present invention, it should be understood that the disclosed apparatus and method can also be implemented in other ways. The apparatus and method embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the apparatus, method and computer program product according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, program segment or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0216] In addition, the functional modules in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0217] If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer, an electronic device, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code. It should be noted that in this article, the term "include", "include" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements that are not explicitly listed, or also includes elements inherent to such process, method, article or device. Without more constraints, an element defined by the phrase "comprising a..." does not exclude the existence of other identical elements in the process, method, article or apparatus comprising the element.

[0218] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention are also intended to include plural forms, unless the context clearly indicates other meanings.

[0219] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0220] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0221] The "first\second" mentioned in the embodiments is only to distinguish similar objects, and does not represent a specific order for the objects. It is understandable that the "first\second" can be interchanged with the specific order or sequence where permitted. It should be understood that the objects distinguished by "first\second" can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than those illustrated or described herein.

[0222] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A tracked vehicle adaptive terrain trajectory tracking method based on deep reinforcement learning, characterized in that: Include: Constructing the action space of tracked vehicles; Define the state space of the tracked vehicle; Design reward functions for tracked vehicles; Establish training Actor network and training Critic network, as well as corresponding target Actor network and target Critic network; According to the state space, the action space, the reward function and the trained Actor network, the tracked vehicle is controlled to interact with the environment to obtain experience samples; According to the experience samples, update the training Critic network; Update the target critic network based on the updated training critic network; According to the updated training Critic network and the experience sample, the training Actor network is updated; Update the target Actor network based on the updated training Actor network; Repeat the above steps of interacting with the environment to obtain experience samples, updating the training Critic network, updating the target Critic network, updating the training Actor network, and updating the target Actor network until the preset convergence standard is reached or the total number of training rounds reaches the preset value, and the optimal Actor network is obtained; Acquire a target trajectory of the tracked vehicle, and control the tracked vehicle to move along the target trajectory through the optimal Actor network; The reward function includes a trajectory error reward function, a control cost reward function and an optimization reward function; Trajectory Error Reward Function : ； In the formula, is the weight of the Euclidean distance error, is the Euclidean distance error, is the weight of heading deviation, is the heading deviation, is the weight of speed deviation, is the speed deviation; Control Cost Reward Function : ； In the formula, is the action at the current moment, The action at the previous moment; Optimizing the Reward Function : ； ； ； In the formula, is a proportionality factor, is the base of natural logarithms, is the area of ​​the coupling region, is the straight-line distance between the current position and the target trajectory point, The deviation between the current vehicle heading angle and the desired heading angle of the target trajectory, is the current deviation of the vehicle from the target trajectory in the lateral direction, is the current deviation of the vehicle from the target trajectory in the longitudinal direction.

2. The method for adaptive terrain trajectory tracking of a tracked vehicle based on deep reinforcement learning according to claim 1, characterized in that: According to the state space, the action space, the reward function and the training Actor network, the tracked vehicle is controlled to interact with the environment and obtain experience samples, specifically including: Get the target position on the target trajectory; According to the target position, the position deviation and heading deviation between the real-time position of the tracked vehicle and the target position are obtained, and the speed deviation between the real-time track speed of the tracked vehicle and the target track speed is obtained to form the current state of the tracked vehicle. ; Input the current state into the training Actor network to obtain the current action that conforms to the action space ; After the tracked vehicle performs the current action, the position deviation and heading deviation between the real-time position of the tracked vehicle and the target position are obtained, and the speed deviation between the real-time track speed of the tracked vehicle and the target track speed is obtained to form the next state of the tracked vehicle. ; Obtain an immediate reward based on the next moment state, the current moment action and the reward function ; The current state, the next state, the immediate reward and the current action are combined into a four-tuple experience sample. , stored in the experience replay pool.

3. The method for adaptive terrain trajectory tracking of a tracked vehicle based on deep reinforcement learning according to claim 1, characterized in that: According to the experience samples, the critic network is updated and trained, specifically including: Randomly sample from the experience replay pool storing four-tuple experience samples to obtain a data set for network training; According to the data set, the target Q value is calculated through the target Actor network and the target Critic network; the calculation model of the target Q value is: ； In the formula, For the data set The target Q value of samples, For instant rewards, is the discount factor, is the output of the target Critic network, For the next moment state, is the target Actor network's strategy, Represents the parameters of the target Critic network; According to the target Q value, the training critic network is updated; the update model of the training critic network is: ； In the formula, is the updated training parameters of the Critic network, is the size of the dataset, is the serial number of the sample in the dataset, For the data set The target Q value of samples, is the action value function, is the current state, Indicates the current action, It is the parameter for training the Critic network; According to the updated training critic network, update the target critic network, including: According to the updated training critic network, the target critic network is updated; the update model of the target critic network is: ； In the formula, is the update coefficient.

4. The method for adaptive terrain trajectory tracking of a tracked vehicle based on deep reinforcement learning according to claim 1, characterized in that: According to the updated training Critic network and the experience sample, the training Actor network is updated, specifically including: According to the updated training Critic network and the empirical samples, the training Actor network is updated using the gradient ascent method; the update model of the training Actor network is: ； In the formula, Represents the updated parameters of the training Actor network, represents the gradient, Indicates action The gradient of the action-value function, represents the policy gradient of the output of the training Actor network, is the action value function, Indicates status, Indicates action, is the updated training parameters of the Critic network, Action functions for training Actor networks, are the parameters for training the Actor network, is the size of the dataset, is the serial number of the sample in the data set; According to the updated training Actor network, update the target Actor network, including: The update model of the target Actor network is: ； In the formula, Parameters of the target Actor network, is the update coefficient.

5. The method for adaptive terrain trajectory tracking of a tracked vehicle based on deep reinforcement learning according to claim 1, characterized in that: Obtaining a target trajectory of the tracked vehicle, and controlling the tracked vehicle to move along the target trajectory through the optimal Actor network, specifically includes: Obtaining a target trajectory of the tracked vehicle, and repeatedly performing the following actions according to the target trajectory to control the movement of the tracked vehicle; Acquire a target position according to the target trajectory; Acquire the current position of the tracked vehicle, acquire the position deviation and heading deviation between the current position and the target position of the tracked vehicle according to the target trajectory, the target position and the current position, and acquire the speed deviation between the current track speed of the tracked vehicle and the target track speed, so as to form the current state of the tracked vehicle; Input the current state into the optimal Actor network to obtain the current action that conforms to the action space; The tracked vehicle is controlled to move according to the current action.

6. The method for adaptive terrain trajectory tracking of a tracked vehicle based on deep reinforcement learning according to claim 1, characterized in that: The action space consists of the left track speed and right track speed The action variables constituted by the speed adjustment range of the left and right tracks, and the action range constituted by the speed adjustment range of the left and right tracks [ , ]; The state space It includes position state variables, heading state variables and speed state variables; the position state variables are determined by the deviation of the vehicle from the target trajectory in the lateral direction. , and the current deviation of the vehicle from the target trajectory in the longitudinal direction The heading state variable is the deviation between the current vehicle heading angle and the desired heading angle of the target trajectory. The speed state variable is the difference between the current left track speed of the vehicle and the target speed , and the difference between the vehicle's current right track speed and the target speed constitute.

7. A tracked vehicle adaptive terrain trajectory tracking device based on deep reinforcement learning, characterized in that: Include: Action space construction module, used to construct the action space of tracked vehicles; A state space definition module, used to define the state space of the tracked vehicle; Reward function design module, used to design reward functions for tracked vehicles; The control machine network establishment module is used to establish the training Actor network and the training Critic network, as well as the corresponding target Actor network and target Critic network; An experience sample acquisition module, used to control the tracked vehicle to interact with the environment and acquire experience samples according to the state space, the action space, the reward function and the trained Actor network; A first updating module, used for updating the training Critic network according to the experience samples; The second updating module is used to update the target critic network according to the updated training critic network; A third updating module is used to update the training Actor network according to the updated training Critic network and the experience sample; The fourth update module is used to update the target Actor network according to the updated training Actor network; Iteration module, used to repeat the above steps of obtaining experience samples through environmental interaction, updating the training Critic network, updating the target Critic network, updating the training Actor network, and updating the target Actor network, until the preset convergence standard is reached or the total number of training rounds reaches the preset value, and the optimal Actor network is obtained; A control module, used for obtaining a target trajectory of the tracked vehicle, and controlling the tracked vehicle to move along the target trajectory through the optimal Actor network; The reward function includes a trajectory error reward function, a control cost reward function and an optimization reward function; Trajectory Error Reward Function : ； In the formula, is the weight of the Euclidean distance error, is the Euclidean distance error, is the weight of heading deviation, is the heading deviation, is the weight of speed deviation, is the speed deviation; Control Cost Reward Function : ； In the formula, is the action at the current moment, The action at the previous moment; Optimizing the Reward Function : ； ； ； In the formula, is a proportionality factor, is the base of natural logarithms, is the area of ​​the coupling region, is the straight-line distance between the current position and the target trajectory point, The deviation between the current vehicle heading angle and the desired heading angle of the target trajectory, is the current deviation of the vehicle from the target trajectory in the lateral direction, is the current deviation of the vehicle from the target trajectory in the longitudinal direction.

8. A tracked vehicle adaptive terrain trajectory tracking device based on deep reinforcement learning, characterized in that: It comprises a processor, a memory, and a computer program stored in the memory; the computer program can be executed by the processor to implement a tracked vehicle adaptive terrain trajectory tracking method based on deep reinforcement learning as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute a tracked vehicle adaptive terrain trajectory tracking method based on deep reinforcement learning as described in any one of claims 1 to 6.

Images