Layered intelligent trajectory tracking control method and system for under-actuated vehicle

Abstract

The invention discloses a layered intelligent trajectory tracking control method and system for an under-actuated vehicle, and belongs to the technical field of intelligent control. In order to solve the problems that in the prior art, dependency on a model is high, a reward mechanism is rigid and perspective lacks, a double-loop framework combining outer-loop model predictive control (MPC) and inner-loop near-end strategy optimization (PPO) is adopted. The outer ring MPC generates a feasible speed reference instruction and a future prediction sequence based on a kinematic model; the inner ring PPO learns a non-linear inverse mapping of speed to thrust to compensate for uncertainty. The core innovation lies in that a look-ahead-response adaptive reward (PRAR) mechanism is designed, and the mechanism comprehensively utilizes real-time pose errors and a future prediction sequence of MPC to dynamically adjust the weight of a reward function, so that the controller has both the real-time error response capability and the future task complexity prediction capability. According to the invention, the trajectory tracking precision, robustness, learning efficiency and generalization ability of the under-actuated vehicle under complex disturbance are significantly improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of intelligent control and autonomous vehicle, and particularly relates to a layered intelligent trajectory tracking control method for underactuated vehicles, which is suitable for trajectory tracking tasks of autonomous systems with underactuated characteristics such as unmanned surface vehicles, unmanned underwater vehicles and double-propeller platforms in high dynamic and uncertain environments. BACKGROUND

[0002] Underactuated unmanned vehicles, such as unmanned surface vehicles and unmanned underwater vehicles, have attracted extensive attention due to their wide application in fields such as ocean monitoring, resource exploration and security patrol. Such systems usually have characteristics such as nonlinearity, strong coupling, time-varying parameters and limited control input. For example, a typical double-propeller USV can only provide longitudinal thrust and yaw moment, but has no direct driving ability in the lateral direction, belonging to a three-degree-of-freedom underactuated system. In actual marine environments, its motion is significantly affected by complex environmental disturbances such as wind, waves and currents, as well as model uncertainties, making it difficult for traditional control methods to achieve high-precision and high-robustness trajectory tracking.

[0003] Currently, trajectory tracking control methods for underactuated vehicles can be mainly divided into two categories: model-based control methods and data-driven control methods. Model-based control methods, such as sliding mode control, backstepping, linear quadratic regulation and nonlinear model predictive control, can achieve good performance when the model is accurate and the disturbance is weak. However, this type of method is strongly dependent on the model, and it is difficult to effectively cope with system parameter uncertainty, strong nonlinearity and external disturbance. In particular, when facing complex sea conditions, model mismatch can cause a significant decline in control performance. In addition, model-based methods usually involve complex optimization problem solving, which has a heavy real-time computing burden, limiting their applicability in dynamic environments. On the other hand, data-driven control methods, especially deep reinforcement learning methods, learn control strategies autonomously through the interaction between agents and the environment, and do not rely on accurate system models, having strong environmental adaptability. Representative algorithms such as proximal policy optimization and deep deterministic policy gradient have made certain progress in USV control. However, existing deep reinforcement learning methods still have the following problems in continuous control tasks: the reward function is usually fixed and designed, making it difficult to adapt to the dynamic changes in task complexity; the randomness in the exploration process easily leads to oscillation or instability of the control output; lack of perception ability for future task trends, unable to achieve forward-looking control adjustment. These problems limit the robustness, stability and training efficiency of deep reinforcement learning in complex trajectory tracking tasks.

[0004] Although there are studies that attempt to combine model predictive control with reinforcement learning to balance the predictive ability of the model and the adaptability of reinforcement learning, existing methods mostly use reinforcement learning for model compensation, parameter tuning or high-level decision making, and fail to effectively construct a reward mechanism that deeply integrates future prediction information and real-time errors, nor does it form a stable and reliable double-loop control architecture.

[0005] Therefore, there is an urgent need for an intelligent control method that can deeply integrate the forward-looking planning ability of MPC and the nonlinear compensation ability of reinforcement learning to improve the trajectory tracking performance of underactuated vehicles in complex dynamic environments. SUMMARY

[0006] The present application aims to overcome the above-mentioned defects of existing underactuated vehicle trajectory tracking control techniques, such as strong dependence on the model, poor environmental adaptability, rigid reward mechanism, and lack of forward-looking decision-making ability, and provides a hierarchical intelligent trajectory tracking control method and system for underactuated vehicles based on model predictive control (MPC), proximal policy optimization (PPO) and prospective-reactive adaptive reward (PRAR), which deeply integrates MPC-PPO double-loop control architecture and PRAR reward shaping mechanism.

[0007] To achieve the above-mentioned purpose, the technical solution of the present application is as follows: A hierarchical intelligent trajectory tracking control method for underactuated vehicles, characterized in that the method comprises the following steps: S1, state perception and trajectory planning step: obtaining the current actual pose and current actual speed of the underactuated vehicle; based on the current actual pose and the predetermined reference trajectory, performing rolling optimization through an outer loop model predictive controller to solve the optimal control sequence in the future time domain, thereby generating a current time speed reference instruction delivered to an inner loop, and simultaneously generating a speed prediction sequence in the future prediction time domain; S2, dynamic reward shaping step: receiving the speed prediction sequence from step S1, while combining the current actual pose with the expected pose of the reference trajectory, calculating a dynamic reward value through a prospective-reactive adaptive reward module; wherein the dynamic reward value integrates a reactive penalty based on the current pose tracking error and a prospective penalty based on the future change complexity of the speed prediction sequence; S3, nonlinear dynamics compensation and control step: the current time speed reference instruction from step S1 and the current actual speed are combined to form a state vector, which is input to an inner loop reinforcement learning controller; the inner loop reinforcement learning controller takes the dynamic reward value calculated in step S2 as the training signal, and outputs a thruster thrust instruction for driving the vehicle through a policy network mapping; S4. Execute the interaction with the environment step: Apply the thrust command generated in step S3 to the thruster of the underactuated vehicle to drive the vehicle to move and complete one control cycle; The vehicle dynamics model and the environmental disturbance together constitute the environment, generate a new state and feed it back to step S1, thereby forming a closed-loop control.

[0008] Furthermore, step S2 specifically includes: S2.1 Response Factor Calculation Steps: Calculate the error between the current actual pose and the desired pose in the inertial coordinate system, and transform it to the error vector in the ship coordinate system through coordinate transformation; process the error vector using a nonlinear mapping function to obtain the response factor reflecting the degree of degradation of real-time tracking performance; S2.2, Calculation steps of the forward factor: For the velocity prediction sequence from step S1, calculate the difference between adjacent prediction steps, and introduce a weight coefficient that decays exponentially with time to emphasize recent predictions, calculate the time-weighted trajectory complexity index that characterizes the complexity of future tasks; process the index using a nonlinear activation function to obtain the forward factor that reflects the degree of future control challenges. S2.3 Dynamic reward synthesis step: Multiply the preset basic weight vector element-wise with the response factor obtained in step S2.1 and the prospective factor obtained in step S2.2 to synthesize dynamic reward weights; use the dynamic reward weights to weight the velocity tracking error at the current moment to generate the final dynamic reward value that acts on the inner loop reinforcement learning controller in step S3.

[0009] Furthermore, in step S2.1, the first nonlinear mapping function is: in, express Time of the first i The reaction factor in each degree of freedom direction For the response amplitude hyperparameter, This is the hyperparameter for reaction sensitivity. for The first time in the ship's hull coordinate system i Error in each degree of freedom direction.

[0010] Furthermore, in step S2.2, the second nonlinear mapping function is: in, express Time of the first i Forward factor in each degree of freedom direction For the look-ahead magnitude hyperparameter, This is the look-ahead sensitivity hyperparameter. For The moment i The time-weighted trajectory complexity in the direction of the first degree of freedom.

[0011] A hierarchical intelligent trajectory tracking control system for underactuated vehicles for implementing the above method, characterized by comprising: A state perception module for obtaining current state information of the underactuated vehicle; An outer loop model predictive control module for generating a speed reference command and a speed prediction sequence according to the current state information and a reference trajectory; A look-ahead-reactive adaptive reward module for dynamically calculating a reward value according to a pose error and the speed prediction sequence; An inner loop reinforcement learning control module for outputting a thrust command according to the speed reference command and an actual speed; An execution module including a propeller of the underactuated vehicle for driving the vehicle to move according to the thrust command.

[0012] The application also provides an electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the method of any one of claims 1 to 4 when executing the program.

[0013] A computer-readable storage medium having a computer program stored thereon, characterized in that the program is executed by a processor to implement the method of any one of claims 1 to 4.

[0014] Compared with the prior art, the application has the following beneficial effects: 1) The outer loop MPC provides a dynamically feasible reference command by considering system constraints and future states; the inner loop PPO accurately compensates for model errors and disturbances in a data-driven manner. The double-loop collaborative work ensures high-precision trajectory tracking under complex sea conditions and model uncertainties.

[0015] 2) The future prediction sequence of MPC is introduced into the reward function through the PRAR mechanism, enabling the control system to have a "foresight" capability. It can adjust the control strategy in advance before the trajectory curvature mutates or the command changes significantly, significantly reducing response overshoot and oscillation, and improving the smoothness and stability of control.

[0016] 3) The dynamic adaptive reward mechanism provides the agent with a clearer and more timely learning direction. The reaction factor accelerates error correction, and the forward-looking factor prevents the strategy from falling into a local optimum. Combined with the stability of the PPO algorithm itself, the entire system training process converges faster and more smoothly. The hierarchical structure clearly defines the responsibility boundaries, and the outer ring MPC ensures the feasibility and safety of the generated instructions, while the inner ring PPO performs high-performance tracking within the safety boundary. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 is a flowchart of the hierarchical intelligent trajectory tracking control method for underactuated vehicles according to the present application.

[0018] Figure 2 is a flowchart of the forward-looking-reaction adaptive reward PRAR in the embodiment of the present application.

[0019] Figure 3 is a double-loop control structure diagram in the embodiment of the present application.

[0020] Figure 4 is the comparative tracking result of the circular trajectory in the embodiment.

[0021] Figure 5 is the comparative tracking result of the sine trajectory in the embodiment. DETAILED DESCRIPTION

[0022] The present application will be described in detail below with reference to the accompanying drawings and embodiments. The present embodiment uses a circular trajectory and a sine combined trajectory to verify the method of the present application.

[0023] The present embodiment provides a hierarchical structure combining model predictive control (MPC) and proximal policy optimization (PPO), and introduces a forward-looking-reaction adaptive reward (PRAR) with future prediction ability and real-time response ability. The overall scheme consists of four modules: outer ring (MPC prediction and feasibility constraints); middle layer (PRAR reward shaping mechanism); inner ring (PPO reinforcement learning controller); and vehicle dynamics actuator. As shown in FIG. 1, the system of the present application adopts a double-loop structure, which consists of an outer ring MPC, a middle layer PRAR, an inner ring PPO, and an execution layer.

[0024] The following is a detailed description.

[0025] 1.1 Outer ring: future prediction speed generation based on MPC The dynamics model of the USV system mainly consists of two parts. The first part links the control input of the USV in three degrees of freedom (corresponding to forward force, lateral force and steering torque respectively) with the dynamics equation for controlling the speed of the USV, which can be represented as: where, is the inertia matrix, which reflects the inertia characteristics of the ship hull during acceleration process. The Coriolis matrix reflects the coupling between inertia and velocity, which is a typical anti-symmetric matrix, used to capture the Coriolis and centrifugal effects during motion, is used to describe the effects of fluid damping and frictional losses on the ship hull motion, including linear and nonlinear components, denotes the surge, sway and yaw angular velocities in the body coordinate system.

[0026] The second part of the dynamic model describes the kinematic mapping relationship between the USV in the body coordinate system and the inertial coordinate system, i.e. the conversion of the body coordinate system velocity to the inertial coordinate system position rate: where, denotes the position and heading angle in the inertial coordinate system, denotes the surge, sway and yaw angular velocities in the body coordinate system, is the coordinate transformation matrix, used to convert the velocity vectors between the fixed body reference system and the inertial reference system, In the dynamic model of the USV, , and The values of some elements in the three matrices are closely related to the structure of the ship, and and The values of each element in the matrix depend on the speed of the USV.

[0027] In order to predict the future speed for MPC, the present invention linearizes the above nonlinear dynamic equation near the current speed and attitude working point by Taylor expansion, obtaining: where, the Jacobian matrix and are respectively: After discretization by forward Euler, by introducing the sampling period seconds, then at the sampling time , , can be derived as: In order to obtain the prediction time domain and control time domain Find all state and control variables within the system and solve for the cost function. To find the optimal value, we constructed an augmented state space. : make: To extract the system's internal pose from the augmented state space for subsequent prediction and weighting in the cost function, an output matrix is ​​defined. : And augmented error state Mapping to output : This allows us to obtain a method for controlling the increment. The augmented prediction model for input: By iterating through the predictive model, the future can be established. The state at each moment and the current state And the future Control increment sequence at time 1 The relationship between them. At time , we will model for the next sampling time. Predicted value Set as: For the next sampling time Predicted value Set as: Will Substituting the expression into the above formula, we can obtain the current state. and Predicted value at time The relationship between them: By analogy, in The moment for the future The predicted value at each time point is: From arrive All the predicted state vectors are stacked into a long vector. : and from to all control increments within -1 are stacked into another long column vector : The state prediction equation can be derived as: where the prediction matrix and are given by: Since the cost function directly acts on the output , we can get the output prediction sequence of future sampling time by combining the prediction equation with the pose error output equation, which is defined as : To simplify the expression, the prediction sequence is redefined as: where: At each control period, the MPC controller solves a finite horizon optimization problem, which is constructed as a standard quadratic programming (QP) problem. The cost function is set as: where and are the error weight matrix and the control increment weight matrix, respectively; is the slack factor, is its penalty coefficient. If the optimization variable is defined as: then the original optimization problem can be represented in the form of the standard QP objective function as: The specific forms of the Hessian matrix and the gradient vector are: The optimization process needs to satisfy the amplitude constraints of the control amount and the control increment simultaneously: At each sampling time This QP problem will be solved online, and the first element of the optimal control increment sequence is used to update the control at the current time: The final controller output is the sum of this feedback control and the feedforward velocity, which serves as the reference signal for the inner-loop PPO controller.

[0028] 1.2 Intermediate layer: Prospective-Reactive Adaptive Reward (PRAR) mechanism This section includes: (1) Reactive factor : Pose error based on real-time error Convert to the body-fixed coordinate system: where is the rotation transformation matrix from the inertial coordinate system to the body-fixed coordinate system.

[0029] Use a nonlinear mapping: where, denotes the reactive factor in the direction of the i th degree of freedom at time , is the reactive amplitude hyperparameter, is the reactive sensitivity hyperparameter, is the pose error in the direction of the i th degree of freedom in the body-fixed coordinate system at time . This mapping can be used to enhance the punishment when the error increases.

[0030] (2) Prospective factor : Future prediction sequence based on Obtain the future velocity prediction sequence of the MPC, denoted as: Reconstruct it as a matrix , where the th behavior is the reference velocity in the th prediction step in three channels.

[0031] To highlight the importance of the near future prediction steps on the current decision, we define a time attention weight : and normalize it: Take the difference between the two adjacent samples for each channel: Time weighted trajectory complexity: By quantifying the dynamic complexity of the MPC future plan, it is nonlinearly mapped to a bounded, smooth gain signal.

[0032] where is the look-ahead factor for the i-th degree of freedom direction at time t, is the look-ahead amplitude hyperparameter, i is the look-ahead sensitivity hyperparameter, is the time weighted trajectory complexity for the i-th degree of freedom direction at time t. (3) Dynamic reward weight i

[0033] where denotes the multiplication of corresponding elements in matrices. (4) Final reward The inner loop velocity tracking error is defined as:

[0034] including longitudinal velocity, lateral velocity, and yaw angle velocity error. Where is the actual velocity of the USV at time t, is the velocity reference output by the MPC. Then the final reward function is: 1.3 Inner loop: PPO-based nonlinear velocity-thrust mapping The state space is designed as: The action space is designed as: ​​​​​and respectively represent the actual output thrust of the left and right thrusters of the controlled ship.

[0035] The loss function of PPO is designed as: where is the clipping width. Such clipping operation makes the PPO algorithm limit to the interval , thereby ensuring the stability of the update. This loss function is used to learn the speed and thrust conversion relationship, realizing model adaptive compensation.

[0036] 1.4 System execution layer According to the PPO output thrust instruction, the positions and attitudes of the double thrusters are updated. In the embodiment, random disturbances can be injected into the lateral force channel and the yaw moment channel for simulating real sea conditions and model uncertainties.

[0037] The disturbances act on the lateral force channel and the yaw moment channel for simulating wind and model bias.

[0038] The embodiment has the following technical effects: 1. Strong task adaptability: Through the prospective-reactive adaptive reward mechanism (PRAR), the reward weight is dynamically adjusted according to the task complexity. Especially combined with the double regulation of future speed prediction sequence and real-time error, the control strategy can keep consistent learning direction under different trajectory shapes, speed levels and task difficulties, significantly improving the task adaptability of the controller.

[0039] 2. Good trajectory stability: By introducing the future prediction information of MPC into the reward construction, the future trajectory change trend is effectively perceived in advance, and possible oscillation is inhibited in advance before the trajectory curvature mutates or the control amount is adjusted greatly, thereby improving the control smoothness and stability of the system in complex trajectories and dynamic environments.

[0040] 3. High learning efficiency: Through the synergistic effect of reward adaptive shaping and double-loop structure, the reinforcement learning strategy is more easily converged in continuous action space; the introduced feasible region constraint and nonlinear inverse dynamics compensation mechanism effectively reduce invalid actions in the exploration stage, making the training process more stable and faster in convergence speed.

[0041] 4. Strong anti-interference ability: through the joint control of dynamic weight adjustment and MPC-PPO double-loop control architecture, excellent control performance can still be maintained under the interference of wind and waves, noise, model deviation, etc., significantly enhancing the robustness and reliability of the system in real sea conditions or industrial disturbance environments.

[0042] 5. High structural feasibility: the outer loop provides a feasible control domain based on model constraints, and the inner loop performs nonlinear compensation based on reinforcement learning, enabling the control process to achieve higher precision execution control while meeting safety constraints and physical constraints, effectively ensuring the feasibility and engineering practicality of the overall system.

[0043] 6. Excellent migration ability: this method does not rely on accurate dynamic models, has good structural versatility and strategy generalization ability, and can be smoothly migrated to USV, AUV, UAV, and other types of underactuated autonomous systems, and can perform stable control in varying conditions, different platforms, or unseen tasks.

[0044] In an embodiment, random disturbances can be injected into the lateral force channel and the yaw moment channel to simulate real sea conditions and model uncertainties.

[0045] Disturbances act on the lateral force channel and the yaw moment channel to simulate wind and waves and model deviations.

[0046] Embodiment: This embodiment takes a certain autonomous unmanned surface vehicle (USV) in a certain sea area performing a two-dimensional trajectory tracking task in a multi-disturbance environment as an example, and applies the layered intelligent control method based on forward-looking-reactive adaptive reward proposed by the present application to perform path tracking control on the vehicle.

[0047] First, collect the running data of the USV in real sea conditions (wind speed 3-6 m / s, random small waves) to calibrate the parameters of the simulation environment, and build a simulation experiment platform for the USV based on a three-degree-of-freedom kinematics-dynamics model. The position and attitude state includes , and the velocity state includes .

[0048] Subsequently, the outer loop MPC planning module is constructed. The prediction step is set to 100, the control step is set to 30, the velocity increment constraints are [0.80, 0.50, 1.4], and the velocity hard constraints are [0.05, 0.05, 0.05]; the position error weight matrix in the cost function is set to a diagonal matrix , and the velocity increment weight matrix is set to MPC generates a speed prediction sequence of 100 steps in the future at each sampling period, and outputs the current reference speed .

[0049] Then, the PRAR reward module is constructed and trained. The specific parameters are as follows: the reaction factor is nonlinearly mapped by the hyperbolic tangent function, the reaction amplitude , the reaction sensitivity ; the time decay coefficient of the foresight factor is , the foresight amplitude , the foresight sensitivity ; the basic reward weight . The PRAR module receives the future speed prediction sequence output by the MPC and the USV pose error in real time, dynamically adjusts the reward weight and calculates the instantaneous reward.

[0050] At the same time, the inner loop thrust control model based on PPO is constructed. The Actor-Critic network contains two hidden layers, and the number of neurons is 512. The action space is the thrust of the left and right thrusters, which are normalized to the interval , the clipping parameter of PPO is , and the learning rate is set to 3x10 -4 . In the training stage, simulate real sea state disturbance, superimpose uniform noise disturbance with an amplitude of 100 in the channel to the force and the yaw moment .

[0051] After the model is trained, reasoning is performed on the test set. The results are shown in Figure 3 Compared with the baseline method (fixed reward PPO model), the method of the present application can more stably track the preset circular trajectory and complex sinusoidal trajectory, and effectively suppress the control jitter of the traditional model at the trajectory mutation position. Quantitative results show that under typical wave disturbance conditions, the attitude root mean square error (RMSE) of the control method of the present application is reduced by about 35%, the steady-state speed error is reduced by about 28%, and the control thrust fluctuation amplitude is reduced by about 22%, which is significantly better than the comparative model.

[0052] The present embodiment has the following technical effects: 1. Strong task adaptability: the reward weight is dynamically adjusted with the task complexity through the foresight-reaction adaptive reward mechanism, especially the dual adjustment of the future speed prediction sequence and the real-time error, so that the control strategy can maintain consistent learning direction under different trajectory shapes, speed levels and task difficulties, significantly improving the task adaptability of the controller.

[0053] 2. Good trajectory stability: By introducing the future prediction information of MPC into the reward construction, the future trajectory change trend can be effectively perceived in advance, and possible oscillation can be inhibited in advance before the trajectory curvature mutation or the control amount is adjusted greatly, thereby improving the control smoothness and stability of the system in complex trajectories and dynamic environments.

[0054] 3. High learning efficiency: Through the synergistic effect of the forward-reaction adaptive reward mechanism and the double-loop structure, the reinforcement learning strategy is more easily converged in the continuous action space; the feasible region constraint and the nonlinear inverse dynamics compensation mechanism effectively reduce invalid actions in the exploration phase, making the training process more stable and faster.

[0055] 4. Strong anti-interference ability: Through the joint control of dynamic weight adjustment and MPC-PPO double-loop structure, the system can still maintain excellent control performance under the interference of wind, waves, noise, model deviation, etc., significantly enhancing the robustness and reliability of the system in real sea conditions or industrial disturbance environments.

[0056] 5. High structural feasibility: The outer loop provides a feasible control domain based on model constraints, and the inner loop performs nonlinear compensation based on reinforcement learning, enabling the control process to meet safety constraints and physical constraints while achieving higher precision execution control, effectively ensuring the feasibility and engineering practicability of the overall system.

[0057] 6. Good migration ability: This method does not rely on an accurate dynamic model, has good structural universality and strategy generalization ability, and can be smoothly migrated to USV, AUV, UAV and other types of underactuated autonomous systems, and can perform stable control in changing conditions, different platforms or unseen tasks.

[0058] The specific embodiments of the present application are described above. It should be understood that the present application is not limited to the above specific embodiments, and those skilled in the art can make various modifications or modifications within the scope of the claims, which does not affect the essential content of the present application. In the case of no conflict, the above embodiments and the features in the embodiments can be combined with each other.

Claims

1. A hierarchical intelligent trajectory tracking control method for underactuated vehicles, characterized in that, The method comprises the following steps: S1, state perception and trajectory planning step: obtaining the current actual pose and the current actual speed of the underactuated vehicle; based on the current actual pose and the predetermined reference trajectory, the optimal control sequence in the future time domain is solved through the outer loop model predictive controller to generate the current time speed reference instruction transmitted to the inner loop, and the speed prediction sequence in the future prediction time domain is generated synchronously; S2, dynamic reward shaping step: receiving the speed prediction sequence from step S1, combining the current actual pose and the expected pose of the reference trajectory, and calculating a dynamic reward value through a look-ahead-reactive adaptive reward module; wherein the dynamic reward value integrates the reactive punishment based on the current pose tracking error and the look-ahead punishment based on the future change complexity of the speed prediction sequence; S3, nonlinear dynamics compensation and control step: the current time speed reference instruction from step S1 and the current actual speed are combined to form a state vector, which is input into the inner loop reinforcement learning controller; the inner loop reinforcement learning controller takes the dynamic reward value calculated in step S2 as the training signal, maps through the policy network, and outputs the thrust instruction for driving the vehicle; S4, execution and environment interaction step: the thrust instruction generated in step S3 is applied to the propeller of the underactuated vehicle to drive the vehicle to move, completing a control cycle; the vehicle dynamics model and environmental disturbance together constitute the environment, generate a new state and feedback to step S1, thereby forming a closed loop control.

2. The layered intelligent trajectory tracking control method of claim 1, wherein, The step S2 specifically comprises: S2.1, reaction factor calculation step: calculating the error of the current actual pose and the expected pose in the inertial coordinate system, and converting it to an error vector in the ship body coordinate system through coordinate transformation; the error vector is processed by a nonlinear mapping function to obtain a reaction factor reflecting the real-time tracking performance degradation; S2.2, look-ahead factor calculation step: for the speed prediction sequence from step S1, calculate the difference between adjacent prediction steps, and introduce a weight coefficient that decays exponentially with time to emphasize recent predictions, and calculate a time-weighted trajectory complexity index representing future task complexity; the index is processed by a nonlinear activation function to obtain a look-ahead factor reflecting the degree of future control challenge; S2.3, dynamic reward synthesis step: multiplying a preset basic weight vector with the reaction factor obtained in step S2.1 and the look-ahead factor obtained in step S2.2 element by element to synthesize a dynamic reward weight; the dynamic reward weight is used to weight the speed tracking error at the current time to generate a final dynamic reward value acting on the inner loop reinforcement learning controller in step S3.

3. The layered intelligent trajectory tracking control method of claim 2, wherein, In the step S2.1, the nonlinear mapping function is: wherein, represents the error in the direction of the i th degree of freedom in the body coordinate system at the time instant is a reaction amplitude hyperparameter, is a reaction sensitivity hyperparameter, is the error in the direction of the i th degree of freedom in the body coordinate system at the time instant 4. The layered intelligent trajectory tracking control method of claim 2, wherein, In the step S2.2, the nonlinear activation function is: wherein, represents the time of the i the time of the is a lookahead amplitude hyperparameter, is a lookahead sensitivity hyperparameter, is a the time of the i the time of the 5. A layered intelligent trajectory tracking control system for underactuated vehicles for implementing the method of any one of claims 1-4, characterized by, It comprises: a state perception module for obtaining the current state information of the underactuated vehicle; an outer loop model predictive control module for generating a speed reference instruction and a speed prediction sequence according to the current state information and a reference trajectory; a forward-looking reaction-adaptive reward module configured to dynamically calculate a reward value according to a pose error and the velocity prediction sequence; an inner-loop reinforcement learning control module configured to output a thrust instruction according to the velocity reference instruction and an actual velocity; an execution module including a propeller of the underactuated vehicle, configured to drive the vehicle to move according to the thrust instruction.

6. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to implement the method in any one of claims 1 to 4.

7. A computer-readable storage medium having stored thereon a computer program, characterized in that The program is executed by the processor to implement the method in any one of claims 1 to 4. The program is executed by the processor to implement the method in any one of claims 1 to 4.

Images