Crawler vehicle trajectory tracking method and system based on fusion of rl and extension optimization, and medium

Abstract

This invention relates to a tracked vehicle trajectory tracking method, system, and medium that integrates RL and extension optimization. The trajectory tracking control method includes: a line-of-sight navigation law that generates a desired longitudinal velocity u based on the vehicle's reference trajectory and a trajectory tracking error model. d And the expected yaw angle; u d The longitudinal total driving force F is generated through an active disturbance rejection control algorithm. sum And the force difference between the left and right tracks F diff ; F sum F diff The torque commands are mapped to the left and right tracks to achieve reference trajectory tracking. This invention integrates a cascaded control architecture that combines line-of-sight navigation law and active disturbance rejection control. It also introduces extension superiority evaluation theory to extract evaluation matter elements from the inner and outer loops, quantifies the real-time comprehensive superiority, and infers scaling coefficients through a reinforcement learning agent in each control cycle based on the real-time comprehensive superiority. This is then fed back to the cascaded control architecture for execution, thereby improving the trajectory tracking accuracy of tracked vehicles in complex dynamic environments.

Description

Technical Field

[0001] This invention relates to the field of vehicle control technology, and in particular to a method, system, and medium for tracking the trajectory of tracked vehicles that integrates RL and extension optimization. Background Technology

[0002] Track tracking of tracked vehicles in complex terrain environments is one of the key technologies for achieving unmanned operations, and it has significant application value in fields such as military reconnaissance, field exploration, and precision agriculture. However, in real-world complex dynamic environments, the motion control of tracked vehicles still faces the following severe bottlenecks: First, in the control methods of unmanned tracked vehicles, the control input is usually a speed command rather than a direct torque command. Under high speed and heavy load conditions, kinematic control cannot cope with transient dynamic disturbances, resulting in a significant decrease in tracking accuracy. Second, the resistance torque and changes in ground adhesion experienced by tracked vehicles in complex off-road environments are difficult to describe accurately. This inaccuracy in physical modeling and the complexity of road interaction make it difficult for traditional control methods based on precise models to achieve ideal tracking accuracy in dynamic environments. Third, controller parameters usually rely on manual experience for tuning. Fixed controller parameters lack adaptability to dynamically changing vehicle speeds and road curvatures, making it difficult to guarantee optimal control performance under all operating conditions. Fourth, traditional deep reinforcement learning faces challenges in error source tracing and multi-objective weight adjustment in cascaded control architectures. When the final trajectory deviates, the agent lacks an effective mechanism to distinguish whether the error originates from an inaccurate outer-loop guidance strategy or insufficient execution capability of the inner loop. This strong coupling leads to blindness in reinforcement learning when exploring parameters, ultimately affecting the convergence speed and global optimality of training. Summary of the Invention

[0003] To address the technical problem of poor trajectory tracking accuracy and adaptive capability of existing tracked vehicles in complex dynamic environments, this invention provides a tracked vehicle trajectory tracking method, system, and medium that integrates RL and extension optimization.

[0004] This invention is achieved through the following technical solution: In a first aspect, the present invention provides a tracked vehicle trajectory tracking method that integrates RL and extension optimization, comprising: A cascaded control architecture integrating the line-of-sight navigation law and active disturbance rejection control algorithm is constructed. The line-of-sight navigation law generates the desired longitudinal velocity u based on the vehicle's reference trajectory and trajectory tracking error model. d And expected yaw angle u d , The longitudinal total driving force F is generated by the active disturbance rejection control algorithm. sum And the force difference between the left and right tracks F diff F sumF diff The torque commands are mapped to the left and right tracks to achieve reference trajectory tracking.

[0005] An evaluation matter-element model integrating the line-of-sight navigation law and active disturbance rejection control algorithm is constructed. (U) d , With the vehicle's longitudinal velocity u and yaw angle The vehicle speed tracking residual e u and yaw angle tracking residual As a characteristic quantity, the inner-loop comprehensive goodness G of the active disturbance rejection control algorithm is calculated. inner The lateral prediction error e between the vehicle's predicted pose and the reference trajectory. nlos and longitudinal prediction error e slos As a characteristic quantity, the outer-loop comprehensive goodness G of the line-of-sight navigation law is calculated. outer .

[0006] Will include G inner G outer The real-time vehicle status information is input into the RL agent, which infers several scaling factors, respectively used to adjust the forward sight distance Δ and the longitudinal error feedback gain k of the line-of-sight navigation law. s The bandwidth of the state observer is expanded in the active disturbance rejection control algorithm to update the line-of-sight navigation law and the active disturbance rejection control algorithm.

[0007] Secondly, this invention proposes a tracked vehicle trajectory tracking system integrating RL and extension optimization, which utilizes the tracked vehicle trajectory tracking method integrating RL and extension optimization described in the first aspect. The tracked vehicle trajectory tracking control system includes: a cascaded control module, an evaluation matter-element module, and an optimization module. The cascaded control module is used to construct a cascaded control architecture integrating the line-of-sight navigation law and the active disturbance rejection control algorithm. The line-of-sight navigation law generates the desired longitudinal velocity u based on the vehicle's reference trajectory and trajectory tracking error model. d And expected yaw angle u d , The longitudinal total driving force F is generated by the active disturbance rejection control algorithm. sum And the force difference between the left and right tracks F diff F sum F diff The torque commands are mapped to the left and right tracks to achieve reference trajectory tracking. The evaluation matter-element module is used to construct an evaluation matter-element model that integrates the line-of-sight navigation law and the active disturbance rejection control algorithm. (U...) d , With the vehicle's longitudinal velocity u and yaw angle The vehicle speed tracking residual e u and yaw angle tracking residual As a characteristic quantity, the inner-loop comprehensive goodness G of the active disturbance rejection control algorithm is calculated. inner The lateral prediction error e between the vehicle's predicted pose and the reference trajectory. nlos and longitudinal prediction error e slos As a characteristic quantity, the outer-loop comprehensive goodness G of the line-of-sight navigation law is calculated. outer The optimization module is used to include G inner G outer The real-time vehicle status information is input into the RL agent, which infers several scaling factors, respectively used to adjust the forward sight distance Δ and the longitudinal error feedback gain k of the line-of-sight navigation law. s The bandwidth of the state observer is expanded in the active disturbance rejection control algorithm to update the line-of-sight navigation law and the active disturbance rejection control algorithm.

[0008] Thirdly, the present invention provides a computer-readable storage medium storing a computer program / instructions. When the computer program / instructions are executed by a processor, they implement the steps of the fused RL and extension-optimized tracked vehicle trajectory tracking method in the first aspect.

[0009] The beneficial effects of this invention are as follows: 1. This invention effectively reduces lateral and longitudinal errors by integrating the line-of-sight navigation law with active disturbance rejection control into a cascaded control architecture. At the same time, it introduces extension superiority evaluation theory into the cascaded control architecture, extracts the evaluation objects of the inner and outer loops respectively, and quantifies the real-time comprehensive superiority. Based on the real-time comprehensive superiority, the scaling coefficient is inferred by the agent in each control cycle and fed back to the cascaded control architecture for execution. This can improve the trajectory tracking accuracy and adaptive capability of tracked vehicles in complex dynamic environments.

[0010] 2. This invention transforms the extension superiority evaluation results into dynamic weight coefficients for each sub-objective in the reinforcement learning composite reward function. When the inner loop execution error exceeds the extension domain boundary, the agent increases the weight of the inner loop tracking accuracy reward term, focusing the optimization efforts on the bandwidth and gain tuning of ADRC to ensure the stability of the inner loop. When the inner loop execution is stable but the outer loop trajectory diverges, the reward weights tilt towards the performance indicators of the line-of-sight navigation law. Combined with a composite reward function that includes extension non-domain penalties, this effectively avoids blind exploration in reinforcement learning, greatly improving model convergence efficiency and global adaptive optimization capabilities. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a flowchart of a tracked vehicle trajectory tracking method that integrates RL and extension optimization; Figure 2 This is the training flowchart for the SAC agent; Figure 3 This is a comparison chart of the reference trajectory and the actual tracking trajectory under the dual-track simulation condition; Figure 4 This is a diagram showing the expected vehicle speed tracking effect under the dual lane change simulation condition; Figure 5 This is a diagram showing the expected yaw angle tracking effect under the dual-track simulation condition; Figure 6 This is a comparison chart of longitudinal tracking errors under different control strategies; Figure 7 This is a comparison chart of lateral tracking errors under different control strategies. Detailed Implementation

[0013] The technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0014] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the specification of this invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "or / and" as used herein includes any and all combinations of one or more of the associated listed items.

[0015] This invention constructs a tracked vehicle trajectory tracking method integrating reinforcement learning (RL) and extension optimization. First, a line-of-sight (LOS) navigation law is used as the geometric guidance layer to ensure efficient trajectory convergence. Second, Active Disturbance Rejection Control (ADRC) technology is used as the dynamic execution layer, employing an Extended State Observer (ESO) to perform real-time estimation and robust compensation of lumped disturbances, addressing the tracked vehicle model mismatch problem. Based on this, this invention innovatively introduces extension superiority evaluation theory to evaluate the navigation accuracy of the geometric guidance layer and the tracking performance of the dynamic execution layer. Finally, combined with reinforcement learning (RL) algorithms, leveraging their powerful policy evolution capabilities, the weight coefficients of each sub-objective in the composite reward function are dynamically adjusted based on the extension evaluation results, autonomously seeking optimal control parameters under complex and variable operating conditions. Based on the above logic, this embodiment proposes a tracked vehicle trajectory tracking method integrating RL and extension optimization. Please refer to [reference needed]. Figure 1It includes the following steps: First, a multi-dimensional mathematical model of the tracked vehicle is established. A dynamic model based on the traction input of the left and right tracks is constructed, along with a kinematic model describing the vehicle's position and heading in the geodetic coordinate system. Next, a trajectory tracking error model in the path tangential coordinate system is established. Specifically, the tracked vehicle dynamic model is as follows:

[0016] Where u is the longitudinal velocity of the vehicle (in the vehicle coordinate system). Let v be the derivative of u, and v be the lateral velocity of the vehicle (in the vehicle coordinate system). Let v be the derivative of v, and r be the yaw rate. F is the derivative of r. tl F tr These represent the traction forces of the left and right tracks, R. l R r The resistance values ​​for the left and right tracks are F, respectively. y Let m be the lateral force, b be the vehicle mass, and b be the center distance between the left and right tracks. z Let M be the moment of inertia about the z-axis. r This is the yaw resistance torque.

[0017] The kinematic model of a tracked vehicle is used to output the state from the vehicle dynamics model. Describes the vehicle's pose state in the global coordinate system. This is used for error calculation in the trajectory tracking error model. The tracked vehicle kinematic model is described in the geodetic coordinate system as follows:

[0018] Where x and y represent the longitudinal and transverse positions in the geodetic coordinate system, and θ is the heading angle. , , These are longitudinal velocity, lateral velocity, and directional angular velocity, respectively.

[0019] In the path tangential coordinate system, the direction of the tangent at any point on the reference trajectory is the reference heading angle θ. ref Longitudinal error e s (Along the path) and lateral error e n (Vertical path direction) can be represented as: ; ; Therefore, the trajectory tracking error equation is established as follows: ; ; Among them, e θFor the heading angle error, x ref For reference, the vertical position, For reference longitudinal velocity, y ref For reference horizontal position, θ ref For reference heading angle, For reference heading angular velocity.

[0020] After determining the vehicle's dynamics model, kinematics model, and trajectory tracking error model, a cascaded control architecture integrating the line-of-sight navigation law and active disturbance rejection control algorithm is constructed. This line-of-sight navigation law generates the desired longitudinal velocity u based on the vehicle's reference trajectory and trajectory tracking error model. d And expected yaw angle Desired longitudinal velocity u d And expected yaw angle The longitudinal total driving force F is generated by the active disturbance rejection control algorithm. sum And the force difference between the left and right tracks F diff The total longitudinal driving force F sum And the force difference between the left and right tracks F diff The torque commands are mapped to the left and right tracks to achieve reference trajectory tracking. The Line of Sight (LOS) navigation law, based on the principle of minimizing the distance between the actual position and the target point on the desired trajectory, generates the desired longitudinal velocity u in real time. d And expected yaw angle This achieves the goal of trajectory tracking. In vehicle trajectory tracking tasks, instead of directly guiding the vehicle back to the nearest point on the path, the vehicle is directed to a point ahead of the path, i.e., the forward-looking point. This results in better convergence and dynamic performance. The expression for the line-of-sight navigation law is: ; ; ; ; Where, k s Here, θ represents the longitudinal error feedback gain, Δ represents the forward sight distance, and θ represents the longitudinal error feedback gain. los Let β be the line-of-sight angle from the vehicle to the forward viewpoint, and β be the sideslip angle of the vehicle's center of gravity. Since the forward view distance Δ directly affects tracking error and convergence speed, an adaptive adjustment strategy for the forward view distance Δ is considered, taking into account tracking error and vehicle speed.

[0021] Where Δ0 is the minimum forward sight distance. uThe vehicle speed coefficient is used to adaptively adjust the forward sight distance Δ based on the longitudinal speed u. That is, the higher the vehicle speed, the larger the forward sight distance Δ, to enhance foresight. The lower the vehicle speed, the smaller the forward sight distance Δ, to improve cornering agility. n The coefficient for the lateral error term in the adaptive forward look distance is used to control the lateral error e. n The degree of attenuation of the forward sight distance Δ, and the lateral error e n The larger the value, the smaller the forward sight distance Δ, thus allowing the vehicle to approach the reference trajectory more quickly.

[0022] An active disturbance rejection control (ADRC) algorithm is then applied to the output of the line-of-sight navigation law. This ADRC algorithm includes: a tracking differentiator, an extended state observer, an error feedback law, and a disturbance compensation control law. The tracking differentiator is used to prevent abrupt changes in the input signal by arranging a transient process to obtain a smooth input signal. The extended state observer is used to calculate the estimated value of the input signal and the estimated value of the total disturbance of the ADRC algorithm. The error feedback law is used to calculate the original control input without disturbance compensation based on the estimated value of the input signal. The disturbance compensation control law is used to calculate the output signal based on the original control input and the estimated value of the total disturbance. Specifically, an ADRC controller is designed based on the tracked vehicle's dynamics model. This is applied to the desired longitudinal speed u. d And expected yaw angle To track the vehicle, a speed control algorithm (first-order ADRC controller) and a yaw angle control algorithm (second-order ADRC controller) are designed to achieve the desired longitudinal speed u. d And expected yaw angle Accurate tracking. The longitudinal and lateral dynamic differential equations in the tracked vehicle's dynamic model are: ; ; Among them, F sum For the total longitudinal driving force, its F sum =F tl + F tr F diff Due to the difference in force between the left and right tracks, its F diff = F tl -F tr Define the unknown system dynamics and external disturbances as lumped disturbances, and expand the lumped disturbances to a new state dimension. The above equation can be written as a first-order system and a second-order system.

[0023] (1) First-order vehicle speed control algorithm.

[0024] For the longitudinal dynamic equation: ; Divide both sides of the above equation by the mass m, and define the total disturbance f of the system. u It can be rewritten in the standard first-order system form of ADRC: ; Here, the system state variable x1 is defined as the longitudinal velocity u. u To control the input gain, it is as follows: U u To control the input, the vehicle speed control algorithm uses the required total longitudinal driving force F. sum Total disturbance f u It includes known track rolling resistance and unknown dynamic uncertainties, and is expressed as: ; w u (t) represents the unmodeled dynamics and external disturbances. Therefore, this longitudinal dynamic process is normalized to the classical first-order system state equations: ; The tracking differentiator in the vehicle speed control algorithm is used to perform a transition process to prevent abrupt changes in the input signal, resulting in a smooth input signal. This allows for a wider adjustable gain range, improves the robustness of the controller, and resolves the contradiction between speed and overshoot in control. Its expression is: ; Where v1 is the desired longitudinal velocity u d The transition process output is v2, which is the derivative of v1. R u Let be the velocity factor, and take . T s To control the cycle.

[0025] The extended state observer in the vehicle speed control algorithm is located in the vehicle speed channel, and its expression is: ; ; ; Where e1 is the observation residual of the vehicle speed channel, z1 is the estimated value of the longitudinal velocity u, and z2 is the total disturbance f. u The estimated value of β. 01 β 02 ω is the error gain coefficient for the corresponding term in the vehicle speed channel expansion state observer. u The bandwidth of the vehicle speed channel expansion state observer.

[0026] The expression for the error feedback law is: ; U u0For the original control input without disturbance compensation, k p_u This is the proportional gain.

[0027] The expression for the disturbance compensation control law is: ; Control input U u That is, the required total longitudinal driving force F sum .

[0028] (2) Second-order yaw angle control algorithm.

[0029] Similarly, for the yaw dynamics equation: ; Given the yaw angle The relationship with the yaw rate r is as follows Then the above formula can be rewritten as about the yaw angle. The second-order differential equation: ; By defining the total disturbance of this second-order system This can be simplified to the ADRC standard second-order system form: ; Define the system state variable x3 as the yaw angle. The system state variable x4 is the yaw rate r. r To control the input, the yaw angle control algorithm is based on the required left and right track force difference F. diff b r To control the input gain, it is Total disturbance It includes unknown modeling errors, road resistance, and external lateral disturbances, and its expression is: ; The system's dynamics and external disturbances are not modeled. Therefore, the dynamic process of the yaw channel is described in the following standard form, serving as the design basis for the second-order linear extended state observer: ; The expression for the tracking differentiator in the yaw angle control algorithm is: ; Where v3 is the desired yaw angle. The transition process outputs v4 as the first derivative of v3 and v5 as the second derivative of v3. Let be the velocity factor, and take . .

[0030] The yaw angle channel expansion state observer in the yaw angle control algorithm is designed as follows: ; ; ; ; Where e2 is the observation residual of the yaw angle channel, and z3 is the yaw angle. The estimated values ​​are z4 and z5, respectively, z4 being the estimated yaw rate r and z5 being the total disturbance. The estimated value, β 03 β 04 β 05 ω is the error gain coefficient for the corresponding term in the yaw angle channel expansion state observer. r The bandwidth of the yaw angle channel expansion state observer.

[0031] Error feedback laws use transient process variables and observed variables to represent expected and actual values, respectively, and are used for feedback control. ; U r0 This is the original control input without disturbance compensation. k p_r k d_r These are the proportional gain and the derivative gain, respectively.

[0032] The expression for the disturbance compensation control law is:

[0033] Control input U r That is, the required difference in force between the left and right tracks, F. diff .

[0034] After obtaining the total longitudinal driving force F sum Left and right track force difference F diff Then, the desired traction force required for the left and right tracks of the vehicle can be calculated. Combining the drive wheel radius R, transmission ratio i, and transmission efficiency η, the desired traction force is converted into motor torque command T for the left and right tracks. L T R :

[0035] Finally, the torque command T will be... L T R The command is sent to the motor driver to execute the reference trajectory tracking.

[0036] One of the innovations of this invention lies in the real-time updating of control parameters for the cascaded control architecture through reinforcement learning that integrates extension superiority evaluation. The updated control parameters are then fed back to the line-of-sight navigation law and the active disturbance rejection control algorithm, further improving trajectory tracking in complex and variable environments. Specifically, an evaluation matter-element model is used to decouple and evaluate the outer loop (line-of-sight navigation law) and inner loop (active disturbance rejection control algorithm) of the cascaded control architecture to quantify the degree of operational superiority.

[0037] (1) Inner loop evaluation matter-element model.

[0038] Select vehicle speed tracking residual e u and yaw angle tracking residual As a characteristic quantity, construct the inner loop evaluation matter element R. inner : ; ; Where, N inner N is the name of the matter element, specifically the name of the matter element in the active disturbance rejection control algorithm at the inner loop. ADRC C represents the set of features, specifically the feature parameters selected for the inner-loop evaluation: vehicle speed tracking residual e. u and yaw angle tracking residual V is the set of values, that is, the values ​​V corresponding to each characteristic quantity. eu , Among them, V eu =|e u |, .

[0039] Next, we set the classical domain X of the i-th feature. 0,I =[0,a I The extensional domain X of the I-th feature quantity ext,I =[0,b I ]. a I b I This represents the boundary of the i-th characteristic quantity. The value V... eu , Let x represent the characteristic value x and the classical domain X. 0,I The distance function ρ(x, X) 0,I And establish the correlation function K(x): ; The value of K(x) ranges from (-∞, 1]. When K(x) < -1, it indicates that the error is in the non-domain and the tracking performance is poor. When -1 ≤ K(x) < 0, it indicates that the error is in the extension domain and belongs to the acceptable range. When 0 ≤ K(x) ≤ 1, it indicates that the error is in the classical domain and belongs to the satisfactory range. At this time, the inner loop overall goodness G of the inner loop execution performance is calculated by using the correlation function K(x).inner : ; α1 is the weighting coefficient for vehicle speed tracking error, and α2 is the weighting coefficient for yaw angle tracking error. Both correspond to the proportion of each feature quantity in the overall goodness, and their overall goodness ranges from (-∞, 1], quantitatively reflecting the ADRC controller's ability to track upper-level commands.

[0040] (2) Outer ring evaluation matter-element model.

[0041] To decouple the evaluation of the outer loop's superiority, the current command of the line-of-sight navigation law is input into the kinematic model to obtain its error compared to the reference trajectory. The lateral prediction error e is selected. nlos and longitudinal prediction error e slos As a characteristic quantity, the outer loop evaluation matter element R is constructed. outer Similarly, calculate the overall goodness G of the outer ring. outer This is used to evaluate the guiding effect of the line-of-sight navigation law, and its expression is: ; ; ; Where, x pred The predicted longitudinal position derived from the kinematic model, y pred N represents the predicted lateral position derived from the kinematic model. outer The name of the object element, specifically N, is the name of the object element in the line-of-sight navigation law. LOS Quantity V enlos =|e nlos |, Quantity V eslos =|e slos |. Set the corresponding classical and extensional domains for each feature quantity, establish the correlation function K(x), and then calculate the outer loop overall goodness G of the outer loop execution performance through the correlation function K(x). outer : G outer =α3K(e nlos )+ α4K(e slos ); Where α3 is the horizontal prediction error weighting coefficient and α4 is the vertical prediction error weighting coefficient.

[0042] After obtaining the comprehensive goodness G of the inner ring... inner Outer ring road comprehensive quality G outer Then, it will include the inner ring comprehensive excellence G. inner Outer ring road comprehensive quality G outer The real-time vehicle status information is input into the RL agent, which infers four scaling factors: forward look distance scaling factor delta_scale, longitudinal error feedback gain scaling factor k.s _scale, bandwidth ω of the vehicle speed channel expansion state observer u The scaling factor u_scale, the yaw angle channel expansion state observer bandwidth ω r The scaling factor r_scale. The output range of all four scaling factors is limited to [0.5, 2.5] to prevent the parameters from deviating too far from the reference value, which could lead to system instability. In each control cycle, the RL agent outputs the four scaling factors delta_scale, k, and r_scale based on the real-time vehicle status information. s _scale, u_scale, and r_scale are respectively related to the forward look distance Δ and the longitudinal error feedback gain k. s Bandwidth ω u Bandwidth ω r The parameters are multiplied to obtain real-time control parameters, which are then fed back to the line-of-sight navigation law and ADRC controller for execution.

[0043] On the other hand, the RL agent needs to be used after training. Another innovative design of this invention is: during the training process of the RL agent, a reward weight adjustment mechanism based on extension superiority evaluation is designed. First, based on the inner-loop comprehensive superiority G... inner Outer ring road comprehensive quality G outer Based on the real-time state, a continuous weight adjustment strategy based on exponentially normalized mapping (Softmax) is constructed to determine the dynamic weight coefficient W in the reinforcement learning reward function. ADRC W LOS : ; ; τ is an adjustment coefficient; in this embodiment, τ=2 is used to adjust the sensitivity of weight allocation to differences in superiority. Next, the state space S is defined as: ; κ represents the curvature of the reference trajectory. This state space S integrates error characteristics and extension evaluation metrics, improving the RL agent's perception depth of the system's operating state.

[0044] Define the action space A as: A={delta_scale, k s _scale, u_scale, r_scale}.

[0045] Design a composite reward function: J = J1 + J2 + J3 + J4 + J5; Where J1 represents the trajectory tracking reward: ; When the lateral error e nLongitudinal error e s The smaller the value, the better the trajectory tracking, and the closer J1 is to 1. When the error is large, it tends to be 0.

[0046] J2 indicates a high-precision tracking bonus: ; When the lateral error e n Longitudinal error e s Less than the set threshold ε n ε s At this time, J2 provides additional positive rewards to incentivize the entry into high-precision tracking mode.

[0047] J3 represents dynamic weighted reward: ; The inner ring's overall excellence G inner Outer ring road comprehensive quality G outer Using real-time values ​​as input, when the inner loop is subjected to transient disturbances that cause the tracking deviation to increase, the overall goodness G of the inner loop is... inner Decrease, corresponding to the negative exponent term This will increase the weight of the reward term related to inner loop tracking accuracy in the reinforcement learning reward function, thereby guiding the RL agent to prioritize optimizing the parameters of ADRC. Conversely, if the inner loop tracking is stable but the outer loop error is large, the overall goodness G of the outer loop will be reduced. outer As the weight of the outer loop decreases, the reward weight increases, and the optimization focus of the RL agent shifts to adjusting the parameters of the line-of-sight navigation law. The optimal balance point is found by balancing the parameter correction intensity of the inner and outer loops through dynamic weight coefficients and leveraging the strong nonlinear mapping capability of reinforcement learning.

[0048] J4 indicates a non-domain penalty: ; When the overall excellence of the inner ring is G inner Outer ring road comprehensive quality G outer When entering a non-domain, J4 provides a significant negative reward, effectively constraining the blind exploration process of reinforcement learning.

[0049] J5 indicates a motion smoothing penalty: When the action outputs a t The smoother the surface, the closer J5 is to 0.

[0050] Where k1 to k6 are six adjustable coefficients in the composite reward function. In this embodiment, k1 is the trajectory tracking basic reward decay coefficient, controlling the lateral error e. n Longitudinal error e sThe sensitivity to tracking reward is set to 1. K2 is the high-precision extra reward value, which provides a fixed positive incentive when the tracking error is below a threshold, and its value is 2. K3 is the outer loop prediction error reward decay coefficient, used to adjust the influence of the line-of-sight navigation law prediction error on the dynamic weight reward term, and its value is 1. K4 is the inner loop tracking error reward decay coefficient, used to adjust the influence of the active disturbance rejection control algorithm tracking residual on the dynamic weight reward term, and its value is 1. K5 is the non-domain penalty value, a larger negative reward applied when the combined superiority of the inner and outer loops enters the non-domain, and its value is 100. K6 is the action smoothing penalty coefficient, used to suppress abrupt changes in action output between consecutive control cycles, and its value is 0.1. After the composite reward function is designed, based on the training samples, the RL agent is trained with the goal of maximizing the cumulative reward of the composite reward function, in order to learn how to move from state s in the state space S. t Action a in action space A t The mapping strategy. RL agents can employ similar agents, such as SAC, DDPG (Deep Deterministic Policy Gradient Algorithm), and PPO (Proximal Policy Optimization Algorithm). In this embodiment, the RL agent can be the SAC agent, which is a heterogeneous policy deep reinforcement learning algorithm based on a maximum entropy framework. Specifically, the SAC agent is based on the Actor-Critic framework and mainly includes: a policy network (Actor) for outputting the action probability distribution and an evaluation network (Critic) for evaluating the action value. For example... Figure 2 As shown, the training environment is initialized. During training, the current state s is used to initialize the training environment. t Action a is obtained by randomly sampling from the probability distribution output by the policy network. t It is then sent to the cascading control architecture for execution. After execution, the environment returns to the next state s. t+1 With reward r t (Reward r) t (This refers to the reward for the current training step calculated based on the aforementioned composite reward function). The evaluation network and policy network are continuously updated through interaction with the environment until the cumulative reward converges, ultimately resulting in a well-trained SAC agent.

[0051] To verify the technical effectiveness of this invention, a dual-track trajectory scenario with a target vehicle speed of 15 m / s was set up, and a tracked vehicle trajectory tracking method integrating RL and extension optimization was used for simulation. The main physical parameters of the tracked vehicle are shown in Table 1. The extension optimization evaluation parameters are shown in Table 2.

[0052] Table 1: Main physical parameters of tracked vehicles.

[0053]

[0054] Table 2: Parameter settings for extension quality evaluation.

[0055]

[0056] In the simulation results, the reference trajectory and the actual trajectory are as follows: Figure 3 As shown. The tracking effect for desired vehicle speed and desired yaw angle is as follows. Figure 4 , Figure 5 As shown. Please refer to... Figure 6 , Figure 7 An ablation experiment was conducted on longitudinal and lateral tracking errors to verify the tracking errors under different control strategies. Simulation results show that the tracked vehicle trajectory tracking method fused with RL and extension optimization proposed in this invention can enable tracked vehicles to maintain extremely high trajectory tracking accuracy in complex dynamic environments.

[0057] In another embodiment, a tracked vehicle trajectory tracking system integrating RL and extension optimization is proposed, which uses the tracked vehicle trajectory tracking method integrating RL and extension optimization described in the above embodiments. The tracked vehicle trajectory tracking control system includes: a cascaded control module, an evaluation matter-element module, and an optimization module. The cascaded control module is used to construct a cascaded control architecture integrating the line-of-sight navigation law and the active disturbance rejection control algorithm. The line-of-sight navigation law generates the desired longitudinal velocity u based on the vehicle's reference trajectory and trajectory tracking error model. d And expected yaw angle u d , The longitudinal total driving force F is generated by the active disturbance rejection control algorithm. sum And the force difference between the left and right tracks F diff F sum F diff The torque commands are mapped to the left and right tracks to achieve reference trajectory tracking. The evaluation matter-element module is used to construct an evaluation matter-element model that integrates the line-of-sight navigation law and the active disturbance rejection control algorithm. (U...) d , With the vehicle's longitudinal velocity u and yaw angle The vehicle speed tracking residual e u and yaw angle tracking residual As a characteristic quantity, the inner-loop comprehensive goodness G of the active disturbance rejection control algorithm is calculated. inner The lateral prediction error e between the vehicle's predicted pose and the reference trajectory. nlos and longitudinal prediction error e slos As a characteristic quantity, the outer-loop comprehensive goodness G of the line-of-sight navigation law is calculated. outer The optimization module is used to include G inner G outer The real-time vehicle status information is input into the RL agent, which infers several scaling factors, respectively used to adjust the forward sight distance Δ and the longitudinal error feedback gain k of the line-of-sight navigation law. sThe bandwidth of the state observer is expanded in the active disturbance rejection control algorithm to update the line-of-sight navigation law and the active disturbance rejection control algorithm.

[0058] In another embodiment, a computer-readable storage medium is also proposed, which stores a computer program / instructions. When the computer program / instructions are executed by a processor, the steps of the fusion RL and extension-optimized tracked vehicle trajectory tracking method in the above embodiments are implemented. The computer-readable storage medium may include, but is not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0059] In another embodiment, a computer program product is also proposed, comprising a computer program / instructions. This computer program / instructions are used to cause a computer to perform the steps of the tracked vehicle trajectory tracking method integrating RL and extension optimization described in the above embodiments. The computer program / instructions exist in a computer-readable medium in forms including, but not limited to, source files, executable files, and installation package files. Accordingly, the computer program / instructions can be executed by a computer in ways including, but not limited to: the computer directly executing the instructions; the computer compiling the instructions and then executing the corresponding compiled program; the computer reading and executing the instructions; or the computer reading and installing the instructions and then executing the corresponding installed program.

[0060] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0061] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.

Claims

1. A method for tracking the trajectory of tracked vehicles that integrates RL and extension optimization, characterized in that, It includes: Construct a cascaded control architecture that integrates line-of-sight navigation law and active disturbance rejection control algorithm; The line-of-sight navigation law generates the desired longitudinal velocity u based on the vehicle's reference trajectory and trajectory tracking error model. d And expected yaw angle ;u d , The longitudinal total driving force F is generated by the active disturbance rejection control algorithm. sum And the force difference between the left and right tracks F diff ; F sum F diff The torque commands are mapped to the left and right tracks to achieve reference trajectory tracking; Construct an evaluation matter-element model that integrates the line-of-sight navigation law and the active disturbance rejection control algorithm; with u d , With the vehicle's longitudinal velocity u and yaw angle The vehicle speed tracking residual e u and yaw angle tracking residual As a characteristic quantity, the inner-loop comprehensive goodness G of the active disturbance rejection control algorithm is calculated. inner The lateral prediction error e between the vehicle's predicted pose and the reference trajectory nlos and longitudinal prediction error e slos As a characteristic quantity, the outer-loop comprehensive goodness G of the line-of-sight navigation law is calculated. outer ; Will contain G inner G outer The real-time vehicle status information is input into the RL agent, which infers several scaling factors, respectively used to adjust the forward sight distance Δ and the longitudinal error feedback gain k of the line-of-sight navigation law. s The bandwidth of the state observer is expanded in the active disturbance rejection control algorithm to update the line-of-sight navigation law and the active disturbance rejection control algorithm.

2. The tracked vehicle trajectory tracking method integrating RL and extension optimization according to claim 1, characterized in that, The expression for the line-of-sight navigation law is: ； ； In the formula, For reference longitudinal velocity, e s For longitudinal error, e θ For the heading angle error, e n For the lateral error, θ ref For reference heading angle, θ los β is the line-of-sight angle from the vehicle to the forward viewpoint, and β is the sideslip angle of the vehicle's center of gravity. Among them, the forward sight distance Δ is based on the vehicle's longitudinal speed u and lateral error e. n Adaptive adjustment.

3. The tracked vehicle trajectory tracking method integrating RL and extension optimization according to claim 1, characterized in that, The active disturbance rejection control algorithm includes: tracking differentiator, extended state observer, error feedback law, and disturbance compensation control law; Among them, the tracking differentiator is used to process the transition process of the input signal to obtain a smooth input signal; The extended state observer is used to calculate the estimated value of the input signal and the estimated value of the total disturbance of the active disturbance rejection control algorithm; Error feedback laws are used to calculate the uncompensated raw control input based on estimates of the input signal; The disturbance compensation control law is used to calculate the output signal based on the original control input and the estimated value of the total disturbance.

4. The tracked vehicle trajectory tracking method integrating RL and extension optimization according to claim 3, characterized in that, For the desired longitudinal velocity u d And expected yaw angle The tracking and active disturbance rejection control algorithms include: vehicle speed control algorithm and yaw angle control algorithm; The vehicle speed channel expansion state observer in the vehicle speed control algorithm is designed as follows: ； ； ； In the formula, e1 is the observation residual of the vehicle speed channel, z1 is the estimated value of the longitudinal speed u, z2 is the estimated value of the total disturbance of the vehicle speed control algorithm, and β 01 β 02 b is the error gain coefficient for the corresponding term in the vehicle speed channel expansion state observer. u To control the input gain, U u For control input; in the vehicle speed control algorithm, the control input U u The total longitudinal driving force F sum ;ω u The bandwidth of the vehicle speed channel expansion state observer; The yaw angle channel expansion state observer in the yaw angle control algorithm is designed as follows: ； ； ； ； In the formula, e2 is the observation residual of the yaw angle channel, and z3 is the yaw angle. The estimated values ​​are z4, z5, and β. 03 β 04 β 05 b is the error gain coefficient for the corresponding term in the yaw angle channel expansion state observer. r To control the input gain, U r To control the input; in the yaw angle control algorithm, the control input U r The difference in force between the left and right tracks, F diff ;ω r The bandwidth of the yaw angle channel expansion state observer.

5. The tracked vehicle trajectory tracking method integrating RL and extension optimization according to claim 4, characterized in that, The RL agent is used after training, and its training methods include: Define the state space S as: ； Among them, e n For lateral error, e s κ represents the longitudinal error, and κ represents the curvature of the reference trajectory. Define the action space A as: A={delta_scale、k s _scale、u_scale、r_scale}; Where delta_scale is the forward sight distance scaling factor, k s _scale is the longitudinal error feedback gain scaling factor, and u_scale is the bandwidth ω of the vehicle speed channel expansion state observer. u The scaling factor, r_scale, is the bandwidth ω of the yaw angle channel expansion state observer. r The scaling factor; Design a composite reward function; during training, the RL agent continuously interacts with the vehicle system environment to obtain state transition data, aiming to maximize the cumulative reward calculated by the composite reward function, and learns to construct a mapping strategy from state space S to action space A.

6. The tracked vehicle trajectory tracking method integrating RL and extension optimization according to claim 5, characterized in that, The reward value J of the composite reward function is the sum of its sub-items: J = J1 + J2 + J3 + J4 + J5; Where J1 represents the trajectory tracking reward; when the lateral error e n Longitudinal error e s The smaller J1 is, the closer it is to 1; J2 represents the high-precision tracking reward; when the lateral error e n Longitudinal error e s When the value is less than the set threshold, J2 provides an additional positive reward; J3 represents dynamic weighted reward; among which, the inner ring comprehensive excellence G inner Outer ring road comprehensive quality G outer After normalization, the corresponding dynamic weight coefficient W is obtained. ADRC W LOS Then through W ADRC W LOS J3 is obtained by weighting and summing the tracking errors. J4 indicates entering a non-domain penalty; when the overall superiority of the inner loop is G... inner Outer ring road comprehensive quality G outer When entering a non-domain, J4 provides a significant negative reward; J5 represents the motion smoothing penalty; when the motion output is a... t The smoother the surface, the closer J5 is to 0.

7. The tracked vehicle trajectory tracking method integrating RL and extension optimization according to claim 1, characterized in that, Calculate the overall goodness G of the inner loop inner Or the overall goodness of the outer ring G outer First, the classical domain of each feature quantity is set, and the correlation function K(·) between the value of the feature quantity and the classical domain is defined; then, the function value of each feature quantity is calculated through the correlation function K(·) and weighted summed to obtain the corresponding inner-loop comprehensive goodness G. inner Or the overall goodness of the outer ring G outer .

8. The tracked vehicle trajectory tracking method integrating RL and extension optimization according to claim 1, characterized in that, The expression for the trajectory tracking error model is: ； ； Among them, e s For longitudinal error, e n Let v be the lateral error, and e be the lateral velocity of the vehicle. θ For heading angle error, For reference longitudinal velocity, For reference heading angular velocity.

9. A tracked vehicle trajectory tracking system integrating RL and extension optimization, characterized in that, It uses the tracked vehicle trajectory tracking method that integrates RL and extension optimization as described in any one of claims 1 to 8; It includes: Cascaded control module, which is used to construct a cascaded control architecture that integrates line-of-sight navigation law and active disturbance rejection control algorithm; The line-of-sight navigation law generates the desired longitudinal velocity u based on the vehicle's reference trajectory and trajectory tracking error model. d And expected yaw angle ;u d , The longitudinal total driving force F is generated by the active disturbance rejection control algorithm. sum And the force difference between the left and right tracks F diff ; F sum F diff The torque commands are mapped to the left and right tracks to achieve reference trajectory tracking; The evaluation matter-element module is used to construct an evaluation matter-element model that integrates the line-of-sight navigation law and the active disturbance rejection control algorithm; with u d , With the vehicle's longitudinal velocity u and yaw angle The vehicle speed tracking residual e u and yaw angle tracking residual As a characteristic quantity, the inner-loop comprehensive goodness G of the active disturbance rejection control algorithm is calculated. inner The lateral prediction error e between the vehicle's predicted pose and the reference trajectory nlos and longitudinal prediction error e slos As a characteristic quantity, the outer-loop comprehensive goodness G of the line-of-sight navigation law is calculated. outer ; The optimization module is used to include G inner G outer The real-time vehicle status information is input into the RL agent, which infers several scaling factors, respectively used to adjust the forward sight distance Δ and the longitudinal error feedback gain k of the line-of-sight navigation law. s The bandwidth of the state observer is expanded in the active disturbance rejection control algorithm to update the line-of-sight navigation law and the active disturbance rejection control algorithm.

10. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the tracked vehicle trajectory tracking method that integrates RL and extension optimization as described in any one of claims 1 to 8.

Images