A tracking control method and system for underwater robots that integrates reinforcement learning navigation and model predictive control

Abstract

This application provides a tracking control method and system for an underwater robot that integrates reinforcement learning navigation and model predictive control, relating to the field of robot control technology. The method includes: based on a top-level reinforcement learning navigation model, treating the underwater robot as a point mass and taking the target path as input, outputting desired velocity and angular velocity; the top-level reinforcement learning navigation model is trained through a first Markov decision process and a reinforcement learning algorithm; based on an intermediate-level nonlinear model predictive tracking controller, taking the desired velocity and angular velocity as tracking targets, outputting desired force and torque; the intermediate-level nonlinear model predictive tracking controller combines the underwater robot's dynamics model to perform state prediction, and performs optimization control by solving a constrained optimization problem in each control cycle; the desired force and torque are applied to the underwater robot's thrusters to drive the underwater robot to perform path tracking control tasks.

Description

Technical Field

[0001] This application relates to the field of robot control technology, and in particular to an underwater robot tracking control method and system that integrates reinforcement learning navigation and model predictive control. Background Technology

[0002] Underwater robots are increasingly used in marine exploration, environmental monitoring, and other fields, and their path-tracking control accuracy is crucial for performing complex tasks. Traditional control methods such as PID (Proportional Integral Derivative) control and Model Predictive Control (MPC) have achieved effective control of underwater robots to a certain extent. However, the underwater environment is characterized by strong nonlinearity, time-varying parameters, and susceptibility to unknown water flow disturbances. Traditional methods based on accurate mechanistic models often experience a significant decrease in control accuracy and robustness when facing such complex dynamic environments.

[0003] Reinforcement learning (RL) has attracted attention in the field of robot control due to its advantages of not relying on precise environment models and possessing adaptive learning capabilities. End-to-end reinforcement learning methods attempt to directly map sensor inputs to control outputs. However, these methods face challenges in practical applications: on the one hand, their training process requires a large amount of interactive data, which is time-consuming and difficult to guarantee real-time performance; on the other hand, end-to-end models struggle to handle complex underwater tasks and lack structured designs to address multi-level control requirements. Although hierarchical reinforcement learning proposes to decompose tasks, existing technologies have not fully utilized the advantages of this architecture, and its adaptability and robustness in dynamic environments remain insufficient.

[0004] Therefore, how to achieve high-precision and robust path tracking for underwater robots in complex environments is a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0005] In view of the above problems, embodiments of this application provide an underwater robot tracking control method and system that integrates reinforcement learning navigation and model predictive control, so as to overcome the above problems or at least partially solve the above problems.

[0006] To solve the above-mentioned technical problems, this application is implemented as follows: A first aspect of this application discloses a tracking control method for an underwater robot that integrates reinforcement learning navigation and model predictive control, the method comprising: Based on the top-level reinforcement learning navigation model, the underwater robot is treated as a point mass, the target path is taken as input, and the desired velocity and angular velocity are output; the top-level reinforcement learning navigation model is obtained through training a first Markov decision process and a reinforcement learning algorithm. The intermediate-layer nonlinear model predictive tracking controller uses the desired velocity and angular velocity as tracking targets and outputs the desired force and torque. The intermediate-layer nonlinear model predictive tracking controller combines the underwater robot dynamics model to predict the state and performs optimized control by solving a constrained optimization problem in each control cycle. The desired force and torque are applied to the underwater robot's thrusters to drive the underwater robot to perform path tracking control tasks.

[0007] Optionally, the top-level reinforcement learning navigation model is constructed through the following steps: By introducing a line-of-sight navigation strategy, the target path tracking problem is transformed into the target path point tracking problem, so as to determine the target heading angle of the underwater robot; Based on the target heading angle and the current position of the underwater robot, the first state space, the first action space, and the first reward function of the first Markov decision process are defined. A reinforcement learning algorithm incorporating a priority experience replay mechanism is used to solve and train the first Markov decision process to obtain the top-level reinforcement learning navigation model.

[0008] Optionally, the first state space includes information on the position error and heading angle error of the underwater robot relative to the target path point; The first motion space includes the forward velocity and yaw rate of the underwater robot; The first reward function includes a penalty term for the position error to guide the underwater robot toward the target path.

[0009] Optionally, a reinforcement learning algorithm incorporating a priority experience replay mechanism is used to solve and train the first Markov decision process, including: In the experience replay buffer of the soft actor-critic algorithm, the temporal difference error is calculated for each sample, and the temporal difference error is used as the sampling priority of that sample; The sampling probability of each sample is calculated based on the sampling priority, so that samples with large temporal difference errors are preferentially extracted from the empirical replay buffer for model training. Importance sampling weights are calculated for the sampled samples to correct for learning bias introduced by non-uniform sampling. The policy network and value network parameters of the soft actor-critic algorithm are updated using samples corrected for importance sampling weights.

[0010] Optionally, the intermediate nonlinear model predictive tracking controller is constructed through the following steps: Define the system state vector and control input vector of the underwater robot, wherein the system state vector includes at least the velocity component and angular velocity component of the underwater robot; Based on the underwater robot dynamics model, a predictive model is established to describe the future state evolution of the system; Construct a cost function that includes a state tracking error term and a control input term. The state tracking error term is used to quantify the deviation between the system state vector and the desired velocity and angular velocity. The control input term is used to constrain the amplitude or rate of change of the control action. Based on the physical limitations and operational requirements of the underwater robot, system state constraints and control input constraints are set; In each control cycle, with the current system state as the initial condition, the system state at each future time is deduced based on the prediction model. Under the premise of satisfying the system state constraints and the control input constraints, the cost function is rolled over and optimized in the preset prediction time domain to solve for the optimal control input sequence. The first element of the optimal control input sequence is used as the expected force and torque output at the current time.

[0011] Optionally, the cost function includes weight parameters to be optimized; the intermediate nonlinear model prediction tracking controller is further constructed through the following steps: A weight optimization reinforcement learning model is constructed to optimize and adjust the weight parameters online.

[0012] Optionally, a weight-optimized reinforcement learning model is constructed, including: A second Markov decision process is constructed, defining a second state space, a second action space, and a second reward function; wherein, the second state space contains the tracking error between the actual state of the underwater robot and the desired velocity and angular velocity, the second action space contains the weight parameters to be optimized, and the second reward function is constructed based on the tracking error; A dual-delay deep deterministic gradient reinforcement learning algorithm is used to solve the second Markov decision process and train a weight optimization strategy. The weight optimization strategy is then used to dynamically adjust the weight parameters of the cost function based on the real-time tracking error.

[0013] Optionally, the underwater robot dynamics model is a mathematical model pre-established based on the underwater robot's mass, inertia, and hydrodynamic coefficients, used to describe the mapping relationship between control input and robot motion response.

[0014] Optionally, after applying the desired force and torque to the underwater robot's thruster, the method further includes: The position and velocity status information of the underwater robot are collected in real time and fed back to the top-level reinforcement learning navigation model and / or the intermediate-level nonlinear model prediction and tracking controller to form a closed-loop control.

[0015] A second aspect of this application discloses an underwater robot tracking and control system that integrates reinforcement learning navigation and model predictive control, comprising: The top-level navigation module is used to treat the underwater robot as a point mass and take the target path as input, based on the top-level reinforcement learning navigation model, and output the desired velocity and angular velocity; the top-level reinforcement learning navigation model is obtained through training a first Markov decision process and a reinforcement learning algorithm. The intermediate-layer predictive tracking control module is used to predict and track the controller based on the intermediate-layer nonlinear model, taking the desired velocity and angular velocity as the tracking targets, and outputting the desired force and torque; the intermediate-layer nonlinear model predictive tracking controller combines the underwater robot dynamics model to predict the state, and performs optimization control by solving a constrained optimization problem in each control cycle; The underlying execution module is used to apply the desired force and torque to the underwater robot's thrusters to drive the underwater robot to perform path tracking control tasks.

[0016] A third aspect of this application discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the underwater robot tracking control method integrating reinforcement learning navigation and model predictive control described in the first aspect of this application.

[0017] A fourth aspect of this application discloses a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the underwater robot tracking control method that integrates reinforcement learning navigation and model predictive control as described in the first aspect of this application.

[0018] A fifth aspect of this application discloses a computer program product, including a computer program that, when executed by a processor, implements the steps of the underwater robot tracking control method that integrates reinforcement learning navigation and model predictive control as described in the first aspect of this application.

[0019] The embodiments of this application have the following advantages: In the embodiments of this application, based on the top-level reinforcement learning navigation model, the underwater robot can obtain navigation strategies by interacting with the environment without relying on an accurate environmental model (the top-level reinforcement learning navigation model is trained through the first Markov decision process and reinforcement learning algorithm), thereby effectively coping with uncertainties such as water flow disturbances and parameter perturbations in the underwater environment, and overcoming the problem of insufficient anti-disturbance performance of traditional mechanism-based control methods in complex environments.

[0020] Furthermore, based on the intermediate-layer nonlinear model predictive tracking controller, the desired velocity and angular velocity output by the top-level navigation model are used as tracking targets. State prediction is performed using the underwater robot's dynamics model, and optimal control is achieved by solving a constrained optimization problem within each control cycle. The model predictive control's precise control capability in the short time domain ensures that the lower execution layer quickly and accurately tracks the top-level navigation commands, achieving high-precision path tracking control.

[0021] Thus, by decomposing the path tracking task into two layers—top-level kinematic decision-making and intermediate-level dynamic tracking—the top-level reinforcement learning navigation model simplifies the underwater robot into a point mass for navigation planning, avoiding the training difficulties of complex dynamic models and ensuring the exploration efficiency of the global navigation strategy. The intermediate-level nonlinear model predictive tracking controller fully utilizes the known dynamic model for precise control, compensating for the shortcomings of end-to-end reinforcement learning methods in handling complex dynamic tasks. The combination of these two approaches achieves a unification of globally optimal decision-making and locally precise control. Attached Figure Description

[0022] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart illustrating the steps of an underwater robot tracking control method that integrates reinforcement learning navigation and model predictive control, as provided in an embodiment of this application. Figure 2 This is an overall architecture diagram of an underwater robot tracking control method that integrates reinforcement learning navigation and model predictive control, provided in an embodiment of this application. Figure 3 This is a comparison chart of the reward curves of the method in this application embodiment and existing reinforcement learning methods; Figure 4 This is a comparison chart of tracking trajectories using different methods; Figure 5It is a reward curve plot of the intermediate nonlinear model trained by reinforcement learning method to predict the controller weight parameters; Figure 6 These are simulation results of the methods in the embodiments of this application; Figure 7 This is a speed tracking result diagram of the method in the embodiment of this application; Figure 8 This is a diagram illustrating the experimental process of tracking a straight line trajectory using the method described in this embodiment of the application. Figure 9 This is a diagram showing the experimental results of the linear trajectory of the method in the embodiments of this application; Figure 10 This is a diagram illustrating the experimental process of tracking an arc trajectory using the method described in this embodiment of the application. Figure 11 This is a diagram showing the experimental results of the arc trajectory of the method in the embodiments of this application; Figure 12 This is a schematic diagram of the structure of an underwater robot tracking and control system that integrates reinforcement learning navigation and model predictive control, provided in an embodiment of this application. Figure 13 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0024] To make the above-mentioned objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0025] The technical concept of this application is to adopt a hierarchical fusion control architecture to address the path tracking control problem of underwater robots in complex dynamic environments. This architecture decomposes the path tracking task into two layers: a top-level decision-making layer and an intermediate-level tracking layer, implemented using reinforcement learning and model predictive control respectively, to fully leverage the advantages of both. Specifically, based on the top-level reinforcement learning navigation model, the underwater robot is simplified to a point mass, learning a global navigation strategy through interaction with the environment and outputting desired velocities and angular velocities. This layer aims to address model uncertainty in complex underwater environments, utilizing the adaptive capabilities of reinforcement learning to generate robust navigation commands. The intermediate layer, based on a nonlinear model predictive tracking controller, uses the desired velocities and angular velocities output by the top-level model as tracking targets, combining the underwater robot's dynamics model for rolling optimization control. This layer utilizes the precise control capabilities of model predictive control in the short time domain to ensure rapid and accurate tracking of navigation commands. Through this hierarchical fusion structure, this application achieves adaptive optimization of the global navigation strategy and precise execution of local motion control, solving the problems of poor disturbance rejection performance of traditional model control methods and low training efficiency of end-to-end reinforcement learning methods in related technologies.

[0026] The underwater robot tracking and control method and system integrating reinforcement learning navigation and model predictive control, provided in this application, will be described in detail below with reference to the accompanying drawings and through specific embodiments and application scenarios.

[0027] Reference Figure 1 As shown, Figure 1 This is a flowchart illustrating the steps of an underwater robot tracking control method integrating reinforcement learning navigation and model predictive control, as provided in an embodiment of this application. Figure 1 As shown, the method may include steps S110 to S130: Step S110: Based on the top-level reinforcement learning navigation model, the underwater robot is regarded as a point mass, the target path is taken as input, and the desired velocity and angular velocity are output; the top-level reinforcement learning navigation model is obtained by training through the first Markov decision process and reinforcement learning algorithm.

[0028] In this step, global navigation commands (i.e., desired velocity and angular velocity) are generated based on a pre-built top-level reinforcement learning navigation model. This top-level reinforcement learning navigation model is obtained through training using a first Markov decision process and a reinforcement learning algorithm. The first Markov decision process is used to describe the interaction between the underwater robot and the environment, and typically includes three core elements: state space, action space, and reward function.

[0029] Specifically, the top-level reinforcement learning navigation model simplifies the underwater robot as a point mass, meaning that at the navigation decision level, complex dynamic characteristics such as the robot's specific attitude and moment of inertia are ignored, and only its position and direction of motion are considered. This simplification reduces the learning difficulty of the navigation strategy and avoids direct exploration in a high-dimensional dynamic space.

[0030] The model takes the target path as input, which is the predetermined trajectory that the underwater robot is expected to follow; the model outputs the desired velocity and angular velocity, which serve as the tracking targets for the next-layer controller. Given the target path and the underwater robot's current position, the model can output the corresponding desired velocity and angular velocity.

[0031] Step S120: Based on the intermediate nonlinear model predictive tracking controller, the desired velocity and angular velocity are used as tracking targets, and the desired force and torque are output; the intermediate nonlinear model predictive tracking controller combines the underwater robot dynamics model to predict the state, and performs optimization control by solving a constrained optimization problem in each control cycle.

[0032] In this step, a pre-built intermediate-layer nonlinear model predictive tracking controller is used to accurately execute top-level navigation commands. This controller combines the underwater robot's dynamics model for state prediction and performs optimal control by solving a constrained optimization problem within each control cycle. The core idea of ​​model predictive control is to use the system's dynamics model to predict the state evolution over a future period and solve a finite-time optimization problem within each control cycle to obtain the optimal control input sequence. Because model predictive control can explicitly handle system constraints (such as thruster limitations, state constraints, etc.) and has the characteristics of rolling optimization, it is particularly suitable for controlled objects with complex constraints and dynamic characteristics, such as underwater robots.

[0033] In practice, the intermediate-layer nonlinear model predictive tracking controller uses the desired velocity and angular velocity output in step S110 as the tracking targets. Combined with a pre-established underwater robot dynamics model, it solves a constrained optimization problem in each control cycle. The goal of this optimization problem is to make the predicted system state as close as possible to the desired state, while simultaneously making the control input as smooth or energy-efficient as possible. After solving this optimization problem, the first element of the optimal control sequence is the desired force and torque to be applied in the current control cycle.

[0034] Step S130: Apply the desired force and torque to the underwater robot's thrusters to drive the underwater robot to perform path tracking control tasks.

[0035] This step is the execution phase of the control commands. The desired force and torque output in step S120 are abstract control quantities that need to be converted into specific actuator commands. After receiving these desired force and torque signals, the underwater robot's control board uses a preset thrust distribution algorithm to calculate the thrust that each thruster should provide, and then converts it into thruster motor speed commands. The propeller blades rotate according to the commands to generate thrust, driving the underwater robot to move along the desired path. At this point, a complete control cycle ends. As the control cycle progresses, the underwater robot gradually approaches and eventually travels along the target path, completing the path tracking control task.

[0036] The technical solution adopted in this embodiment decomposes the underwater robot's path tracking task into two layers through a hierarchical control architecture: a top-level navigation decision-making layer and a middle-level motion control layer. The top-level reinforcement learning navigation model utilizes the adaptive capabilities of reinforcement learning to generate robust navigation commands in complex and uncertain underwater environments. The middle-level nonlinear model predictive tracking controller leverages the precise control capabilities of model predictive control, combined with a dynamic model for state prediction and rolling optimization, ensuring that the underlying actuators track navigation commands quickly and accurately. This integration enables the underwater robot to adapt to complex dynamic environments while achieving high-precision path tracking control, overcoming the problems of poor disturbance rejection performance in traditional model control methods and low training efficiency in end-to-end reinforcement learning methods in related technologies.

[0037] It is understood that the underwater robot in this application includes: a sealed cabin, a thruster, and an electrical system; wherein, the sealed cabin may be made of high-toughness nylon material, and its upper cover plate is statically sealed to the cabin with an O-ring; a thruster fixing plate may be embedded in the cabin to fix the thruster mounted on the outside of the cabin; the thruster may use four thrusters to control the robot's three-dimensional spatial movement, for example, one thruster is set at the front and rear of the cabin to control the robot's pitch and attitude, and one thruster is set on each of the left and right sides of the cabin to control the robot's forward and backward movement and yaw attitude; the thrusters may be configured as an underactuated configuration, with four degrees of freedom of motion capability; the electrical system includes the underwater robot's power supply unit, motion control unit, communication unit, and sensor configuration unit.

[0038] In an optional embodiment, after step S130 above, which involves "applying the desired force and torque to the underwater robot's thruster," step S140 is further included: Step S140: Collect the position and velocity status information of the underwater robot in real time and feed it back to the top-level reinforcement learning navigation model and / or the intermediate-level nonlinear model prediction and tracking controller to form a closed-loop control.

[0039] In this embodiment, a feedback mechanism is further introduced to form a complete closed-loop control. First, the current motion state information is collected in real time by various sensors carried by the underwater robot. Specifically, a Doppler velocimeter can be used to measure the robot's three-dimensional velocity information relative to the bottom or water layer, a depth sensor can be used to obtain the current depth data, and combined with the inertial measurement unit and dead reckoning algorithm, the precise position and attitude of the robot in the global coordinate system can be calculated. This real-time collected data constitutes the system's feedback.

[0040] Secondly, the collected position and velocity information is fed back, either separately or simultaneously, to the top-level reinforcement learning navigation model and the intermediate-level nonlinear model predictive tracking controller. The top-level reinforcement learning navigation model, upon receiving the underwater robot's position and velocity information, can reassess its current deviation from the target path and output adjusted desired velocities and angular velocities in the next decision cycle. This feedback mechanism allows the top-level reinforcement learning navigation model to dynamically adjust navigation commands (i.e., desired velocities and angular velocities) based on the actual tracking performance. For example, when the robot deviates from the predetermined path due to water flow interference, the desired velocities and angular velocities can generate larger steering commands to correct the deviation.

[0041] After receiving the position and velocity state information of the underwater robot, the intermediate-layer nonlinear model predictive tracking controller uses this information as the initial state for rolling optimization in the next control cycle. The essence of this model control is to predict and optimize the future based on the current measured state; therefore, real-time state feedback is an indispensable component. By continuously refreshing the optimization starting point with the measured state, the controller can promptly correct tracking errors caused by model mismatch or external disturbances.

[0042] Through the above feedback mechanism, a complete closed-loop control system is formed. The top-level reinforcement learning navigation model adjusts navigation commands according to position deviation, and the intermediate-level nonlinear model predicts and tracks the controller to correct the control force according to speed deviation. The two work together to enable the underwater robot to move continuously and stably along the target path.

[0043] The technical solution adopted in this embodiment, by introducing real-time state information feedback, enables the top-level reinforcement learning navigation model to dynamically adjust its navigation strategy based on the actual tracking performance, thereby improving the adaptability of path tracking. Simultaneously, it provides an accurate initial state for the intermediate-layer nonlinear model predictive tracking controller, fully leveraging the advantages of rolling optimization and effectively suppressing accumulated errors and external disturbances. Thus, it enhances the path tracking accuracy and system robustness of the underwater robot in complex environments.

[0044] In an optional embodiment, the "top-level reinforcement learning navigation model" in step S110 above is constructed through the following steps S110-1 to S110-3: Step S110-1: Introduce a line-of-sight navigation strategy to transform the target path tracking problem into a target path point tracking problem in order to determine the target heading angle of the underwater robot.

[0045] In this step, an intuitive navigation reference is introduced for the top-level reinforcement learning navigation model, namely line-of-sight navigation. Its core idea is to simplify the complex task of "traveling along a curved path" into the simple task of "chasing a dynamic point on the path".

[0046] In practice, a circle is drawn with the underwater robot's current position as the center and a preset line-of-sight radius as its length. This circle intersects the target path. The intersection point of this circle and the target path that is in front of the robot's direction of travel is the forward-looking point. This forward-looking point is considered the instantaneous target point that the underwater robot needs to chase at the current moment. Through this geometric transformation, the originally complex curve tracking problem is transformed into a series of point tracking problems.

[0047] Based on the position of the forward-looking point and the robot's own position, the target heading angle at the current moment can be calculated. This target heading angle indicates the direction the robot needs to move from its current position to the forward-looking point, providing a clear reference for subsequent navigation decisions.

[0048] For example, the tracking error of an underwater robot system is defined. as follows: , , Then the target heading angle The calculation is as follows: , in, This is the current location of the underwater robot. This is the target path point.

[0049] Step S110-2: Based on the target heading angle and the current position of the underwater robot, define the first state space, the first action space, and the first reward function of the first Markov decision process.

[0050] The first state space includes the position error and heading angle error information of the underwater robot relative to the target path point; the first action space includes the forward velocity and yaw angle velocity of the underwater robot; and the first reward function includes a penalty term for the position error to guide the underwater robot to approach the target path.

[0051] In this step, the target heading angle is introduced into the reinforcement learning framework to construct a complete first Markov decision process. The first state space describes the environmental conditions of the underwater robot at the current moment. It is constructed based on the target heading angle obtained in step S110-1 and the current position of the underwater robot, and includes at least information such as the robot's positional deviation and heading deviation relative to the target path point. This information constitutes the basis for the underwater robot's decision-making.

[0052] For example, the first state space It can be represented as: , in, This represents the Euclidean distance between the underwater robot's current position and the target path point. For heading angle error, This refers to the tracking error of the underwater robot system.

[0053] The first action space is used to define the navigation commands that the underwater robot can execute. Considering that the top-level reinforcement learning navigation model simplifies the underwater robot to a point mass, and that underwater robots typically have underactuated characteristics (such as the inability to move laterally independently), this embodiment designs the action space as a continuous action space that includes forward velocity and yaw angular velocity. This design not only conforms to the actual motion capabilities of the underwater robot, but also provides a clear tracking target for the intermediate nonlinear model predictive tracking controller.

[0054] For example, the first action space A can be represented as: , in, Indicates forward velocity, This indicates the yaw rate.

[0055] The first reward function is used to evaluate the quality of each action and guide the underwater robot to learn in the desired direction. The design of the reward function revolves around the core objective of "approaching the target path". When the underwater robot takes an action and gets closer to the target path or target point, it is given a positive reward; when it deviates, it is given a negative penalty.

[0056] For example, the first reward function It can be represented as: , in, and This represents the weighting parameter for the distance term.

[0057] Step S110-3: Use a reinforcement learning algorithm that combines a priority experience replay mechanism to solve and train the first Markov decision process to obtain the top-level reinforcement learning navigation model.

[0058] In this step, a reinforcement learning algorithm is used to solve the first Markov decision process to train and obtain the optimal navigation strategy. Specifically, a reinforcement learning algorithm combined with a priority experience replay mechanism is employed. The core idea of ​​priority experience replay is to no longer sample uniformly from the experience pool, but instead assign a higher sampling probability to samples with "higher learning value" (usually measured by temporal difference error). This mechanism enables the underwater robot to learn more frequently from experiences that have not yet been fully learned or are challenging, thereby accelerating the training process and improving the performance of the final strategy.

[0059] During training, the underwater robot continuously interacts with the environment in either a simulated or real-world setting, storing the resulting experiential data (including states, actions, rewards, and next states) in an experience replay buffer. Then, a batch of samples is sampled from this buffer according to a priority experience replay mechanism to update the policy and value network parameters of the reinforcement learning algorithm. As the number of training rounds increases, the underwater robot's navigation strategy gradually optimizes, eventually converging to a stable strategy that can output the optimal expected velocity and angular velocity based on the current state, thus obtaining the top-level reinforcement learning navigation model.

[0060] The technical solution of this embodiment transforms the complex path tracking problem into an easily manageable point tracking problem by introducing a line-of-sight navigation strategy, providing a clear navigation reference for the reinforcement learning model. Based on this, a Markov decision process is constructed that integrates the target heading angle and the underwater robot's position information into the state space, enabling the model to fully perceive the environment and task requirements. The action space design closely matches the actual motion capabilities of the underwater robot, and the reward function effectively guides the model to learn towards the target path. Finally, a reinforcement learning algorithm incorporating a priority experience replay mechanism is used for training, improving learning efficiency and the performance of the final navigation strategy. Through this embodiment, a top-level reinforcement learning navigation model that outputs robust navigation commands in complex underwater environments can be obtained.

[0061] In an optional embodiment, step S110-3 above, "using a reinforcement learning algorithm combined with a priority experience replay mechanism to solve and train the first Markov decision process," may include the following steps S110-3-1 to S110-3-4: Step S110-3-1: In the experience replay buffer of the soft actor-critic algorithm, calculate the temporal difference error for each sample and use the temporal difference error as the sampling priority of that sample.

[0062] The soft actor-critic algorithm is a reinforcement learning algorithm based on the maximum entropy framework. It introduces a policy entropy reward term into the traditional reinforcement learning objective (maximizing cumulative reward), encouraging underwater robots to maintain a certain degree of randomness during exploration. This mechanism gives the algorithm better exploration capabilities and robustness when facing complex tasks, making it particularly suitable for control problems involving underwater robots with continuous action spaces.

[0063] During the training of the soft actor-critic algorithm, the experiential data (including current state, actions, rewards, and next state) generated by the underwater robot's interaction with the environment are stored in an experience replay buffer. Traditional reinforcement learning methods typically train by uniformly and randomly sampling from the buffer, but this approach ignores the differences in the learning contributions of different samples.

[0064] This step improves upon the proposed priority experience replay mechanism. Specifically, whenever a new experience sample is stored in the buffer, its temporal difference is calculated. This temporal difference error measures the difference between the current value network estimate and the actual sampled return. The larger the temporal difference error, the less accurate the current model's prediction of that sample, and the higher the learning value or information contained in that sample. Therefore, this step uses the absolute value of the temporal difference error as the sampling priority for that sample; the larger the error, the higher the priority.

[0065] Step S110-3-2: Calculate the sampling probability of each sample according to the sampling priority, so as to preferentially extract samples with large temporal difference errors from the empirical replay buffer for model training.

[0066] In this step, the priority of each sample is converted into an actual sampling probability. Specifically, a priority-based probability distribution sampling method is used, where the sampling probability of a sample is proportional to its priority; that is, samples with higher priority have a greater probability of being selected for training. This non-uniform sampling strategy allows the underwater robot to learn more frequently from "unlearned" or "challenging" experiences, thereby accelerating the convergence process of the value function and policy function and improving sample utilization efficiency.

[0067] For example, the sampling probability of the i-th sample It can be represented as: , in, For hyperparameters, Let be the priority of the i-th sampling, i.e., the sequential difference error; This indicates the priority of the k-th sampling.

[0068] Step S110-3-3: Calculate the importance sampling weights for the extracted samples to correct for learning bias introduced by non-uniform sampling.

[0069] In this step, we consider that non-uniform sampling alters the original distribution of the data, causing a shift in the expected gradient during training; this shift is called learning bias. Directly using biased samples to update network parameters may lead to the model converging to a suboptimal solution or even diverging. To address this issue, this step introduces importance sampling weights. For each sample drawn, an importance sampling weight is calculated based on its sampling probability. This weight is inversely proportional to the sample's sampling probability; that is, samples with higher sampling probabilities have smaller weights. This weakens the excessive influence of the sample during gradient updates, restoring an unbiased estimate of the original data distribution.

[0070] For example, importance sampling weights It can be represented as: , in, This indicates the capacity of the experience playback buffer. This is a constant parameter used to adjust the sampling weights.

[0071] Step S110-3-4: Update the policy network and value network parameters of the soft actor-critic algorithm using the samples corrected by importance sampling weights.

[0072] In this step, after calculating the importance sampling weights, the weighted samples are used for actual network parameter updates. Specifically, the state, action, reward, and next state data from the samples are input into each network of the soft actor-critic algorithm, and the corresponding loss function is calculated. When calculating the gradient and updating the network parameters, the contribution of each sample is multiplied by its corresponding importance sampling weight, thereby achieving bias correction. In this way, the sample efficiency improvement brought by priority experience replay is utilized, while avoiding the training instability problems that may be caused by non-uniform sampling.

[0073] As the training process iterates, the samples in the experience replay buffer are continuously updated, the priorities are dynamically adjusted, and the parameters of the policy network and value network are gradually optimized, ultimately resulting in a high-performance top-level reinforcement learning navigation model.

[0074] The technical solution adopted in this embodiment improves the training efficiency and final performance of the top-level reinforcement learning navigation model by introducing a priority experience replay mechanism into the soft actor-critic algorithm. Specifically, by using temporal difference error as the sampling priority, the model can pay more attention to difficult samples with inaccurate predictions, thus accelerating the learning process. At the same time, the bias introduced by non-uniform sampling is corrected by importance sampling weights, ensuring the stability and convergence of training. This embodiment, while maintaining the exploration capabilities of the soft actor-critic algorithm, further improves the sample utilization efficiency, enabling the top-level navigation model to learn a better navigation strategy within a limited number of interactions.

[0075] In an optional embodiment, the "intermediate layer nonlinear model prediction tracking controller" in step S120 above is constructed through the following steps S120-1 to S120-5: Step S120-1: Define the system state vector and control input vector of the underwater robot. The system state vector includes at least the velocity component and angular velocity component of the underwater robot.

[0076] In this step, the system state vector is a set of variables describing the underwater robot's motion at a certain moment. The velocity components can include forward velocity, lateral velocity, etc., while the angular velocity component mainly refers to the yaw rate. These variables can comprehensively characterize the robot's real-time motion state, providing necessary input information for subsequent prediction and control.

[0077] The control input vector is a set of variables that describe the control actions applied to an underwater robot, corresponding to the forces and torques that the propulsion system can generate, such as forward thrust and yaw torque. These control inputs serve as a bridge connecting the controller algorithm and the underlying actuators; the desired forces and torques obtained in subsequent steps are the specific values ​​of the control input vectors.

[0078] Step S120-2: Based on the underwater robot dynamics model, establish a predictive model to describe the future state evolution of the system.

[0079] The underwater robot dynamics model is a mathematical model pre-established based on the underwater robot's mass, inertia, and hydrodynamic coefficients, used to describe the mapping relationship between control input and robot motion response.

[0080] Based on this underwater robot dynamics model, a predictive model can be established to deduce the system's state evolution trajectory over a future period, based on the measured state at the current moment and the assumed control input sequence applied over a future period within each control cycle. The length of this deduction time is called the prediction time domain.

[0081] For example, the system state vector at time i+1 It can be represented as: , in, This is the dynamic model of the underwater robot, where Np is the prediction time domain, and I here means that the value of i ranges from 0 to... .

[0082] Step S120-3: Construct a cost function that includes a state tracking error term and a control input term. The state tracking error term is used to quantify the deviation between the system state vector and the desired velocity and angular velocity. The control input term is used to constrain the amplitude or rate of change of the control action.

[0083] Specifically, the state tracking error term measures the deviation between the predicted state and the desired state, where the desired state refers to the expected velocity and angular velocity output by the top-level reinforcement learning navigation model. A larger tracking error results in a larger value for this term, indicating poorer control performance. Minimizing this term can drive the predicted state to be as close as possible to the desired state.

[0084] Control inputs are used to penalize the control action itself to avoid generating excessively large or abrupt control signals, thereby saving energy, protecting the actuator, and ensuring the smoothness of the control process. Control inputs can be constrained based on the magnitude of the control input or the rate of change of the control input.

[0085] The cost function is a weighted sum of the state tracking error term and the control input term, where the weight coefficients of each term determine the relative importance of the control objectives. For example, increasing the weight of the state tracking error term will make the system more aggressive in pursuing tracking accuracy, while increasing the weight of the control input term will make the control action smoother.

[0086] For example, the cost function can be expressed as: , Among them, the state tracking error term and control input items They are represented as follows: , , in, , and These represent the state error weight matrix, the control input weight matrix, and the terminal state error weight matrix, respectively. Let represent the system state vector at time i. This represents the desired system state vector at time i (i.e., the desired velocity and angular velocity output in step S110 above). Indicates the first The system state vector at time t. Indicates the first The expected system state vector at time t. The control input vector at time i.

[0087] Step S120-4: Based on the physical limitations and operational requirements of the underwater robot, set system state constraints and control input constraints.

[0088] In this step, system state constraints stem from the physical limitations or operational safety requirements of the underwater robot. For example, the maximum forward speed of the underwater robot is limited by the thruster power, the maximum diving depth is limited by the hull's pressure resistance, and the attitude angle is limited by stability requirements. Violating these constraints during control may lead to equipment damage or mission failure.

[0089] Control input constraints stem from the physical limitations of the actuators. For example, the maximum thrust that each thruster can provide is finite, and the rate of thrust change is also limited by the motor's response speed. These constraints must be taken into account when generating control commands; otherwise, the generated commands will be unexecutable in the actual system.

[0090] By explicitly incorporating these constraints into the formulation of the optimization problem, the model predictive controller can ensure that the generated control commands are always within the feasible range, thereby improving the safety and practicality of the control system.

[0091] For example, system state constraints and control input constraints can be expressed as: , , in, This represents the system state vector at time i. The control input vector at time i; and These represent the prediction time domain and the control time domain, respectively. For the system state constraint set, This represents the system state vector at time i+1. To control the set of input constraints, Indicates the first The control input vector at time t; I here means that the value of i ranges from... arrive .

[0092] Step S120-5: In each control cycle, with the current system state as the initial condition, the system state at each future time is deduced based on the prediction model. Under the premise of satisfying the system state constraints and the control input constraints, the cost function is rolled over in the preset prediction time domain to solve for the optimal control input sequence. The first element of the optimal control input sequence is used as the expected force and torque output at the current time.

[0093] Specifically, at the beginning of each control cycle, the controller first acquires the position and velocity state information of the underwater robot (obtained through sensor measurements) as the initial conditions for the optimization problem. Then, based on the prediction model (i.e., the underwater robot dynamics model) established in step S120-2, the controller deduces the state evolution trajectory of the system at each time point in the future prediction time domain under different assumed control input sequences.

[0094] Based on this, the controller, under the premise of satisfying the various constraints set in step S120-4, solves an optimization problem in a finite time domain: finding a set of optimal control input sequences for a future period (i.e., within the control time domain) such that the cost function defined in step S120-3 is minimized in the prediction time domain. The calculation of the cost function here depends on the system state at each future time point deduced by the prediction model.

[0095] The optimal control input sequence obtained by the solution contains control commands for multiple future time points. However, model predictive control does not execute the entire sequence, but only takes the first element of the sequence (i.e., the control command to be applied at the current time) and outputs it to the underlying actuator as the desired force and torque.

[0096] When the next control cycle arrives, the controller will reacquire the latest measured state and repeat the above optimization process. This control method of re-optimizing in each cycle and rolling forward enables model predictive control to make timely use of the latest state feedback information to effectively compensate for model mismatch and external disturbances, thus exhibiting strong robustness.

[0097] The technical solution of this embodiment constructs a complete nonlinear model predictive tracking controller. First, the system state and control input are clearly defined, establishing a clear variable foundation for the controller. Second, a predictive model is established based on the underwater robot's dynamics model, enabling the controller to predict future states. Then, a cost function including tracking error terms and control input terms is constructed, clarifying the quantification method of the control objective. Next, state and input constraints are set to ensure the feasibility of control commands. Finally, by extrapolating future states based on the predictive model and performing rolling optimization in each control cycle, accurate and robust tracking of top-level navigation commands is achieved. This embodiment fully leverages the advantages of model predictive control in handling constraints and optimizing control, providing reliable technical support for high-precision path tracking of underwater robots in complex environments.

[0098] In one optional embodiment, the cost function includes weight parameters to be optimized; the intermediate nonlinear model prediction tracking controller is further constructed through the following steps: constructing a weight optimization reinforcement learning model to optimize and adjust the weight parameters online.

[0099] In this step, the core idea of ​​the weight optimization reinforcement learning model is to model the online adjustment problem of weight parameters as another reinforcement learning problem, forming a two-layer optimization architecture: the inner layer is the model predictive controller that performs rolling optimization based on the current weight parameters and outputs control commands; the outer layer is the weight optimization reinforcement learning model that dynamically adjusts the weight parameters of the inner layer controller according to the current control effect, so as to optimize the overall control performance.

[0100] The technical solution of this embodiment, by constructing a weighted optimization reinforcement learning model, achieves online optimization and adjustment of the weight parameters of the model predictive controller cost function, overcoming the limitations of traditional methods that rely on offline manual tuning and are difficult to adapt to dynamic environments. Specifically, this model can dynamically adjust the weight distribution between the state tracking error term and the control input term based on real-time tracking error information, ensuring that the controller maintains optimal control performance under different operating conditions. This adaptive weight adjustment mechanism further improves the path tracking accuracy, response speed, and control robustness of underwater robots in complex and changing environments, representing an important optimization and supplement to the aforementioned hierarchical control architecture.

[0101] Furthermore, a weight optimization reinforcement learning model is constructed, including steps A1 and A2: Step A1: Construct a second Markov decision process, defining a second state space, a second action space, and a second reward function; wherein, the second state space contains the tracking error between the actual state of the underwater robot and the desired velocity and angular velocity, the second action space contains the weight parameters to be optimized, and the second reward function is constructed based on the tracking error.

[0102] In this step, unlike the first Markov decision process in the top-level navigation model, the second Markov decision process describes a meta-level optimization problem: the goal of the weight optimization reinforcement learning model is to optimize the control effect of the inner controller by adjusting the parameters of the inner controller.

[0103] The design of the second state space needs to reflect the current control effect and provide a basis for weight adjustment decisions. In this embodiment, the second state space is constructed based on the tracking error between the actual state and the desired state of the underwater robot. Specifically, it includes forward velocity error, lateral velocity error, yaw rate error, etc. This error information reflects the performance of the nonlinear model predictive tracking controller under given weight parameters: if the tracking error is large, it indicates that the current weight parameter settings may not be ideal and need to be adjusted; if the tracking error is small, it indicates that the current weight parameters are appropriate and should be maintained or fine-tuned.

[0104] For example, the second state space It can be represented as: , in, These are forward velocity error, lateral velocity error, and yaw rate error, respectively.

[0105] The second action space corresponds to the weight parameters to be optimized. The cost function of the nonlinear model predictive tracking controller includes a state error weight matrix, a control input weight matrix, and a terminal state error weight matrix. The elements in these matrices are the weight parameters to be optimized as described in this step. The second action space defines the range of actions that the weight optimization model can output, i.e., the feasible value interval for each weight parameter. By outputting specific weight values, the weight optimization model directly intervenes in the behavior of the inner model predictive controller.

[0106] For example, the second action space It can be represented as: , in, The weights corresponding to the forward velocity error, lateral velocity error, and yaw rate error are respectively. The weights correspond to the system state vector and the control input vector.

[0107] The second reward function evaluates the effectiveness of each weight adjustment, guiding the weight optimization model to learn in a direction that improves control performance. The design of the reward function should align with the control objective of path tracking and be based on the tracking error: a positive reward is given when the weight adjustment reduces subsequent tracking errors; a negative reward or penalty is given when the tracking error increases. Furthermore, to balance tracking accuracy and control energy consumption, a penalty term for the control input amplitude can also be introduced into the reward function.

[0108] For example, the second reward function r can be expressed as: , in, These are the weighting coefficients of the reward function. Forward propulsion force for underwater robot thrusters; For the steering torque of the underwater robot's thruster; These are forward velocity error, lateral velocity error, and yaw rate error, respectively.

[0109] Based on the above definition, this step transforms a weight parameter optimization problem into a standard reinforcement learning problem, laying the foundation for subsequent solution and training.

[0110] Step A2: The second Markov decision process is solved using a dual-delay deep deterministic gradient reinforcement learning algorithm to train a weight optimization strategy, which is then used to dynamically adjust the weight parameters of the cost function based on the real-time tracking error.

[0111] In this step, after defining the second Markov decision process, a dual-delay deep deterministic policy gradient reinforcement learning algorithm is used to solve it, training a weight optimization policy that can output optimal weight parameters based on real-time state. This algorithm is an advanced reinforcement learning algorithm designed for continuous action spaces, possessing advantages such as deterministic policy, dual-value network, and delayed updates. It can effectively suppress value overestimation problems and improve training stability.

[0112] During training, the weight optimization model continuously interacts with an interactive system comprising an inner model predictive controller and the underwater robot's environment. At each decision point, the weight optimization model outputs a set of weight parameters (second action) based on the currently observed tracking error (second state). The inner model predictive controller performs rolling optimization based on these weight parameters, outputs control commands, and drives the underwater robot's movement. After one evaluation cycle, the reward value is calculated based on the tracking error, and this series of interaction data (state, action, reward, next state) is stored in an experience replay buffer. Then, samples are sampled from the buffer, and the parameters of the policy network and value network are progressively optimized according to the update rules of the dual-delay deep deterministic policy gradient.

[0113] As training progresses, the weight optimization strategy gradually learns to output the optimal combination of weight parameters that yields the best subsequent control performance under different tracking error states. Once training is complete, the strategy can be deployed to the actual control system. In each control cycle (or every few cycles), the weight parameters in the model predictive controller cost function are dynamically adjusted based on the real-time feedback tracking error, achieving adaptive optimization of control performance.

[0114] The technical solution of this embodiment constructs a second Markov decision process and solves it using a dual-delay deep deterministic policy gradient algorithm, thereby realizing a reinforcement learning model specifically designed to optimize the weight parameters of a model predictive controller. This model can dynamically adjust the weight parameters in the cost function based on real-time tracking errors, ensuring that the inner controller always operates at an optimal or near-optimal operating point. Compared to model predictive control with fixed weight parameters, the adaptive weight adjustment mechanism of this embodiment significantly improves the adaptability and robustness of the control system under different operating conditions. Compared to traditional offline tuning methods, this embodiment achieves online automatic optimization of weights, reducing the workload of manual parameter tuning and enabling it to cope with performance degradation caused by environmental changes.

[0115] like Figure 2 As shown, Figure 2 This is an overall architecture diagram of an underwater robot tracking control method that integrates reinforcement learning navigation and model predictive control, provided in an embodiment of this application. This embodiment adopts a layered fusion control architecture, which is divided into three layers: a top-level reinforcement learning navigation layer, an intermediate nonlinear model predictive tracking control layer, and a bottom-level thruster control layer.

[0116] I. Model construction methods for each layer: The top-level reinforcement learning navigation layer (top-level reinforcement learning navigation model) is constructed as follows: First, a line-of-sight navigation strategy is introduced to transform the target path tracking problem into a target path point tracking problem to determine the target heading angle of the underwater robot; second, based on the target heading angle and the current position of the underwater robot, the first state space, the first action space, and the first reward function of the first Markov decision process are defined; finally, a reinforcement learning algorithm combined with a priority experience replay mechanism is used to solve and train the first Markov decision process to obtain the top-level reinforcement learning navigation model.

[0117] The intermediate nonlinear model predictive tracking control layer (intermediate nonlinear model predictive tracking controller) is constructed through the following steps: First, a cost function containing state tracking error terms and control input terms is constructed, and system state constraints and control input constraints are set according to the physical limitations and operational requirements of the underwater robot; second, a weighted optimization reinforcement learning model is constructed to optimize and adjust the weight parameters contained in the cost function online; finally, in each control cycle, with the current system state as the initial condition, the system state at each future moment is deduced based on the underwater robot dynamics model (prediction model), and under the premise of satisfying the system state constraints and control input constraints, the cost function is rolled over in the preset prediction time domain to solve for the optimal control input sequence, and the first element of the optimal control input sequence is used as the expected force and torque output at the current moment.

[0118] II. The specific control process of the underwater robot is as follows: In the top-level reinforcement learning navigation layer, the underwater robot is simplified into a point mass and processed using the target path as input. This layer generates the desired velocity and angular velocity based on a pre-built top-level reinforcement learning navigation model, which are then passed down to the intermediate layer controller as the tracking target. This layer mainly utilizes the adaptive capabilities of reinforcement learning to solve the model uncertainty problem in complex underwater environments and generate robust global navigation commands.

[0119] The intermediate nonlinear model predictive tracking control layer receives the desired velocity and angular velocity output from the top layer as the tracking target. Within each control cycle, this layer performs state prediction based on the current system state (including the underwater robot's position and velocity information) and the underwater robot's dynamics model. Under the premise of satisfying system state constraints and control input constraints, it performs rolling optimization on the cost function to solve for the optimal control sequence. The first element of this sequence is then output to the bottom execution layer as the desired force and torque. This layer utilizes the precise control capability of model predictive control in the short time domain to ensure fast and accurate tracking of the top-level navigation commands.

[0120] The bottom thruster control layer receives the desired force and torque output from the middle layer, and uses a thrust distribution algorithm to calculate them into thrust commands for each thruster, which are then converted into speed commands for the thruster motors, driving the underwater robot to move along the desired path.

[0121] Furthermore, during its movement, the underwater robot's onboard sensors (such as Doppler velocimeters, depth sensors, and inertial measurement units) collect real-time position and velocity information, which is then fed back to the top-level reinforcement learning navigation model and the intermediate-level nonlinear model predictive tracking controller, forming a closed-loop control system. The top-level reinforcement learning navigation model adjusts the desired velocity and angular velocity based on the feedback position information, while the intermediate-level nonlinear model predictive tracking controller updates the initial conditions for rolling optimization based on the feedback velocity information, thereby effectively compensating for model mismatch and external disturbances.

[0122] Through the aforementioned layered fusion architecture, this application achieves the unification of adaptive optimization of global navigation strategy and precise execution of local motion control, enabling underwater robots to achieve high-precision and robust path tracking control in complex dynamic environments.

[0123] To verify the effectiveness of the technical solution presented in this application, multiple sets of comparative experiments were conducted in both a simulation environment and a real water tank. The following section combines... Figures 3 to 11 The performance of the solution in this application is described in detail.

[0124] Figure 3 This is a comparison of the reward curves of the method in this application and existing reinforcement learning methods. Existing reinforcement learning methods include DDPG (Deep Deterministic Policy Gradient), TD3 (Twin Delayed Deep Deterministic Policy Gradient), and SAC (Soft Actor Critic). It can be seen that the method in this application (i.e., SAC-PER in the figure) has the fastest convergence speed and the highest cumulative reward, indicating that the introduction of the priority experience replay mechanism in this application significantly improves the sample efficiency and learning performance of reinforcement learning, enabling the top-level navigation model to obtain a better navigation policy in a shorter training time.

[0125] Figure 4 The image shows a comparison of tracking trajectories using different methods. It is clear that the navigation trajectory generated by the method presented in this application (SAC-PER) is closest to the target path, with the smallest tracking error. This result verifies that the desired velocity and angular velocity generated by the navigation algorithm in this application have higher accuracy and robustness, providing a better tracking target for the intermediate layer controller.

[0126] Figure 5This is a reward curve plotting the weight parameters of the intermediate-layer nonlinear model predictive tracking controller trained using a reinforcement learning method. It shows that the reward value gradually increases with the number of training epochs and eventually converges. This indicates that the weight optimization reinforcement learning model has successfully learned the optimal strategy for adjusting the cost function weight parameters under different operating conditions. Through this online adaptive weight adjustment mechanism, the intermediate-layer nonlinear model predictive tracking controller can dynamically optimize its control behavior based on real-time tracking errors, avoiding the limitations of traditional methods that rely on offline manual tuning.

[0127] Figure 6 The simulation results of the method in this embodiment are shown in the figure. It can be seen that after starting from the initial position, the underwater robot can quickly converge to the target path and maintain a small tracking error throughout the subsequent tracking process. Even in areas where the curvature of the target path changes significantly, the actual trajectory of the underwater robot still closely matches the target path, verifying the effectiveness of the hierarchical control architecture of this application.

[0128] Figure 7 The diagram shows the velocity tracking results of the method in this embodiment. It can be seen that, regardless of forward velocity or yaw rate, the underwater robot's actual response can quickly and accurately track the expected value output by the top-level navigation model. Even when velocity changes drastically, although the actual velocity shows a slight lag, it can complete tracking in a very short time, with a steady-state error close to zero. This result verifies the accurate tracking capability of the intermediate-layer nonlinear model predictive tracking controller.

[0129] Figure 8 This is an experimental process diagram of the method for tracking a straight line trajectory according to the embodiment of this application. As can be seen from the diagram, the underwater robot maintains a stable posture throughout the entire movement process and does not exhibit significant yaw. Figure 9 This is a diagram showing the experimental results of the straight-line trajectory of the method in the embodiment of this application. As can be seen from the diagram, after the underwater robot starts from the initial position, it accurately tracks the target straight line after a short adjustment, and the tracking error remains within a small range throughout the entire process.

[0130] Figure 10 This is a diagram of the experimental process of tracking an arc trajectory using the method of this application embodiment. As can be seen from the diagram, the underwater robot can still maintain stable tracking on an arc path with continuously changing curvature. Figure 11 This is an experimental result diagram of the arc trajectory of the method in the embodiment of this application. It can be seen from the diagram that the actual trajectory of the underwater robot is highly consistent with the target arc, and there is no obvious deviation even in the region of maximum curvature.

[0131] The above-mentioned experimental results fully demonstrate the feasibility and effectiveness of the proposed method in practical applications. Whether on straight or complex curved paths, the underwater robot can achieve high-precision path tracking control, verifying the excellent transferability of the proposed method from simulation to real-world systems.

[0132] comprehensive Figures 5 to 11 Based on simulation and experimental results, the proposed method exhibits the following superior performance: Through a priority experience replay mechanism, the training convergence speed of the top-level navigation model is significantly faster than that of traditional reinforcement learning algorithms; in both simulation environments and real-world pool experiments, the actual trajectory of the underwater robot closely matches the target path; through weight-optimized reinforcement learning models, the intermediate-layer controller can dynamically adjust control parameters according to real-time operating conditions, adapting to complex and changing working environments; under dynamic conditions such as changes in path curvature and speed, the system maintains stable tracking performance; and experimental results verify the deployability and effectiveness of the proposed method in practical underwater robot systems.

[0133] Therefore, the method of this application effectively overcomes the problems of poor anti-disturbance performance of traditional model control methods and low training efficiency of end-to-end reinforcement learning methods, and realizes high-precision and high-robust path tracking control of underwater robots in complex dynamic environments.

[0134] This application also provides an underwater robot tracking and control system that integrates reinforcement learning navigation and model predictive control, referring to... Figure 12 As shown, Figure 12 This is a schematic diagram of a tracking and control system for an underwater robot that integrates reinforcement learning navigation and model predictive control, provided in an embodiment of this application. The system includes: The top-level navigation module 1210 is used to construct a top-level reinforcement learning navigation model, treating the underwater robot as a point mass, taking the target path as input, and outputting the desired velocity and angular velocity; the top-level reinforcement learning navigation model is obtained through training a first Markov decision process and a reinforcement learning algorithm. The intermediate layer predictive tracking control module 1220 is used to construct an intermediate layer nonlinear model predictive tracking controller, which takes the desired velocity and angular velocity as the tracking targets and outputs the desired force and torque. The intermediate layer nonlinear model predictive tracking controller combines the underwater robot dynamics model to predict the state and performs optimization control by solving a constrained optimization problem in each control cycle. The underlying execution module 1230 is used to apply the desired force and torque to the thruster of the underwater robot to drive the underwater robot to perform path tracking control tasks.

[0135] It is understood that the underwater robot tracking control system that integrates reinforcement learning navigation and model predictive control in the embodiments of this application can realize the underwater robot tracking control method that integrates reinforcement learning navigation and model predictive control in the above embodiments. The underwater robot tracking control system that integrates reinforcement learning navigation and model predictive control has the same advantages as the underwater robot tracking control method that integrates reinforcement learning navigation and model predictive control compared with the prior art, and will not be repeated here.

[0136] This application also provides an electronic device, which is described in reference to... Figure 13 , Figure 13 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. For example... Figure 13 As shown, the electronic device 1300 includes a memory 1310 and a processor 1320. The memory 1310 and the processor 1320 are connected via a bus for communication. The memory 1310 stores a computer program, which can run on the processor 1320 to implement the steps of the underwater robot tracking control method that integrates reinforcement learning navigation and model predictive control as described in the embodiments of this application.

[0137] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the underwater robot tracking control method that integrates reinforcement learning navigation and model predictive control as described in this application.

[0138] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the underwater robot tracking control method that integrates reinforcement learning navigation and model predictive control as described in this application.

[0139] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0140] This application describes embodiments of methods and apparatus according to flowchart illustrations and / or block diagrams. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1A device that provides the functions specified in one or more boxes.

[0141] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0142] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0143] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0144] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0145] The above provides a detailed description of an underwater robot tracking control method and system that integrates reinforcement learning navigation and model predictive control. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and its core ideas. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A tracking control method for an underwater robot that integrates reinforcement learning navigation and model predictive control, characterized in that, include: Based on the top-level reinforcement learning navigation model, the underwater robot is treated as a point mass, with the target path as input, and the expected velocity and angular velocity are output. The top-level reinforcement learning navigation model is obtained through training a first Markov decision process and a reinforcement learning algorithm. The intermediate-layer nonlinear model predictive tracking controller uses the desired velocity and angular velocity as tracking targets and outputs the desired force and torque. The intermediate-layer nonlinear model predictive tracking controller combines the underwater robot dynamics model to predict the state and performs optimized control by solving a constrained optimization problem in each control cycle. The desired force and torque are applied to the underwater robot's thrusters to drive the underwater robot to perform path tracking control tasks.

2. The underwater robot tracking control method integrating reinforcement learning navigation and model predictive control according to claim 1, characterized in that, The top-level reinforcement learning navigation model is constructed through the following steps: By introducing a line-of-sight navigation strategy, the target path tracking problem is transformed into the target path point tracking problem, so as to determine the target heading angle of the underwater robot; Based on the target heading angle and the current position of the underwater robot, the first state space, the first action space, and the first reward function of the first Markov decision process are defined. A reinforcement learning algorithm incorporating a priority experience replay mechanism is used to solve and train the first Markov decision process to obtain the top-level reinforcement learning navigation model.

3. The underwater robot tracking control method integrating reinforcement learning navigation and model predictive control according to claim 2, characterized in that, The first state space includes the position error and heading angle error information of the underwater robot relative to the target path point; The first motion space includes the forward velocity and yaw rate of the underwater robot; The first reward function includes a penalty term for the position error to guide the underwater robot toward the target path.

4. The underwater robot tracking control method integrating reinforcement learning navigation and model predictive control according to claim 2, characterized in that, A reinforcement learning algorithm incorporating a priority experience replay mechanism is used to solve and train the first Markov decision process, including: In the experience replay buffer of the soft actor-critic algorithm, the temporal difference error is calculated for each sample, and the temporal difference error is used as the sampling priority of that sample; The sampling probability of each sample is calculated based on the sampling priority, so that samples with large temporal difference errors are preferentially extracted from the empirical replay buffer for model training. Importance sampling weights are calculated for the sampled samples to correct for learning bias introduced by non-uniform sampling. The policy network and value network parameters of the soft actor-critic algorithm are updated using samples corrected for importance sampling weights.

5. The underwater robot tracking control method integrating reinforcement learning navigation and model predictive control according to claim 1, characterized in that, The intermediate-layer nonlinear model predictive tracking controller is constructed through the following steps: Define the system state vector and control input vector of the underwater robot, wherein the system state vector includes at least the velocity component and angular velocity component of the underwater robot; Based on the underwater robot dynamics model, a predictive model is established to describe the future state evolution of the system; Construct a cost function that includes a state tracking error term and a control input term. The state tracking error term is used to quantify the deviation between the system state vector and the desired velocity and angular velocity. The control input term is used to constrain the amplitude or rate of change of the control action. Based on the physical limitations and operational requirements of the underwater robot, system state constraints and control input constraints are set; In each control cycle, with the current system state as the initial condition, the system state at each future time is deduced based on the prediction model. Under the premise of satisfying the system state constraints and the control input constraints, the cost function is rolled over and optimized in the preset prediction time domain to solve for the optimal control input sequence. The first element of the optimal control input sequence is used as the expected force and torque output at the current time.

6. The underwater robot tracking control method integrating reinforcement learning navigation and model predictive control according to claim 5, characterized in that, The cost function includes weight parameters to be optimized; the intermediate nonlinear model prediction tracking controller is further constructed through the following steps: A weight optimization reinforcement learning model is constructed to optimize and adjust the weight parameters online.

7. The underwater robot tracking control method integrating reinforcement learning navigation and model predictive control according to claim 6, characterized in that, Constructing a weight-optimized reinforcement learning model includes: A second Markov decision process is constructed, defining a second state space, a second action space, and a second reward function; wherein, the second state space contains the tracking error between the actual state of the underwater robot and the desired velocity and angular velocity, the second action space contains the weight parameters to be optimized, and the second reward function is constructed based on the tracking error; A dual-delay deep deterministic gradient reinforcement learning algorithm is used to solve the second Markov decision process and train a weight optimization strategy. The weight optimization strategy is then used to dynamically adjust the weight parameters of the cost function based on the real-time tracking error.

8. The underwater robot tracking control method integrating reinforcement learning navigation and model predictive control according to any one of claims 1-7, characterized in that, The underwater robot dynamics model is a mathematical model pre-established based on the underwater robot's mass, inertia, and hydrodynamic coefficients, used to describe the mapping relationship between control input and robot motion response.

9. The underwater robot tracking control method integrating reinforcement learning navigation and model predictive control according to any one of claims 1-7, characterized in that, After applying the desired force and torque to the underwater robot's thruster, the process further includes: The position and velocity status information of the underwater robot are collected in real time and fed back to the top-level reinforcement learning navigation model and / or the intermediate-level nonlinear model prediction and tracking controller to form a closed-loop control.

10. A tracking and control system for an underwater robot that integrates reinforcement learning navigation and model predictive control, characterized in that, include: The top-level navigation module is used to treat the underwater robot as a point mass and take the target path as input, based on the top-level reinforcement learning navigation model, and output the desired velocity and angular velocity. The top-level reinforcement learning navigation model is obtained through training a first Markov decision process and a reinforcement learning algorithm. The intermediate-layer predictive tracking control module is used to predict and track the controller based on the intermediate-layer nonlinear model, taking the desired velocity and angular velocity as the tracking targets, and outputting the desired force and torque; the intermediate-layer nonlinear model predictive tracking controller combines the underwater robot dynamics model to predict the state, and performs optimization control by solving a constrained optimization problem in each control cycle; The underlying execution module is used to apply the desired force and torque to the underwater robot's thrusters to drive the underwater robot to perform path tracking control tasks.

Images