Underwater robot path planning and control method based on reinforcement learning

Abstract

The application relates to the field of intelligent robot control, and discloses a path planning and control method for an underactuated underwater robot based on reinforcement learning. The method comprises the following steps: determining the current position of a target robot as a starting point, selecting a plurality of path points in a three-dimensional space between the starting point and an ending point based on the kinematic constraints of the target robot, so as to generate a target path with the minimum comprehensive cost; in the process of the target robot tracking the target path, calculating a tracking error based on the target path, the current pose and speed of the target robot; inputting the tracking error and the current speed of the target robot into a tracking controller to obtain a corresponding PID parameter adjustment amount; and adjusting the thrust of each propeller of the target robot based on the PID parameter adjustment amount, so that the target robot tracks the target path. The method can generate a high-quality tracking trajectory for the underwater robot, improve tracking accuracy and save energy consumption.

Description

Technical Field

[0001] This application relates to the field of intelligent robot control, specifically to a path planning and control method for underactuated underwater robots based on reinforcement learning. Background Technology

[0002] Autonomous Underwater Vehicles (AUVs) are capable of autonomously performing environmental perception, localization, analysis, and decision-making tasks in complex environments, playing a vital role in marine scientific research and resource exploration. Underactuated AUVs offer advantages such as low mechanical complexity, light weight, and low energy consumption; however, their underactuated structure leads to significant nonlinear coupling during motion, complicating path planning and trajectory tracking.

[0003] Current underactuated AUV motion planning and control schemes suffer from low planning efficiency, high energy consumption, and poor path planning quality, resulting in low tracking accuracy and difficulty in adapting to the ever-changing and unknown underwater environment. Therefore, improving the tracking accuracy of underactuated underwater robots and enhancing their adaptability and robustness in unknown underwater environments are problems that need to be addressed. Summary of the Invention

[0004] The purpose of this application is to provide a path planning and control method for underactuated underwater robots based on reinforcement learning, so as to improve the tracking accuracy of underactuated underwater robots and enhance their adaptability and robustness in unknown underwater environments.

[0005] To achieve the above objectives, the technical solution of this application is as follows: In a first aspect, embodiments of this application provide a path planning and control method for an underactuated underwater robot based on reinforcement learning, the method comprising: The current position of the target robot is determined as the starting point. Based on the kinematic constraints of the target robot, multiple path points are selected in the three-dimensional space between the starting point and the ending point to generate the target path with the minimum comprehensive cost. The comprehensive cost includes at least the path length cost and the energy consumption cost. During the process of the target robot tracking the target path, the tracking error is calculated based on the target path, the current pose and speed of the target robot; The tracking error and the current speed of the target robot are input into the tracking controller to obtain the corresponding PID parameter adjustment. The tracking controller is trained through reinforcement learning based on the tracking error and speed of the target robot, with the optimization objective of maximizing the comprehensive reward. The comprehensive reward is related to the tracking error and energy consumption of the target robot. Based on the adjustment of the PID parameters, the target robot is controlled to track the target path.

[0006] Secondly, embodiments of this application provide a reinforcement learning-based underactuated underwater robot tracking and control device for implementing the steps in the method provided in the first aspect of this application. The device includes: The path planning module is configured to determine the current position of the target robot as the starting point, and select multiple path points in the three-dimensional space between the starting point and the ending point based on the kinematic constraints of the target robot, so as to generate the target path with the minimum comprehensive cost; the comprehensive cost includes at least: path length cost and energy consumption cost; The monitoring module is configured to calculate the tracking error based on the target path, the current pose and speed of the target robot during the process of the target robot tracking the target path; The control module is configured to input the tracking error and the current speed of the target robot into the tracking controller to obtain the corresponding PID parameter adjustment amount; the tracking controller is obtained through reinforcement learning training based on the tracking error and speed of the target robot, with the optimization objective of maximizing the comprehensive reward; the comprehensive reward is related to the tracking error and energy consumption of the target robot; based on the PID parameter adjustment amount, the target robot is controlled to track the target path.

[0007] Thirdly, embodiments of this application provide an electronic device including a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps in the method provided in the first aspect of embodiments of this application.

[0008] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps in the method provided in the first aspect of embodiments of this application.

[0009] The reinforcement learning-based path planning and control method for underactuated underwater robots provided in this application incorporates the kinematic constraints of the target robot into the path planning process. With the goal of minimizing path length and energy consumption, multiple path points conforming to the kinematic constraints of the underactuated robot are selected in the three-dimensional space between the starting and ending points to generate a dynamically feasible target path. By combining kinematic constraints with minimizing the overall cost, path points that the underactuated robot cannot reach can be avoided. This ensures that each step the target robot takes is within reach during its subsequent movement along the target path, thereby improving the tracking accuracy. In the path tracking stage, maximizing the overall reward of the target robot during navigation is the optimization objective. PID parameter tuning is optimized through reinforcement learning, thereby simultaneously optimizing tracking accuracy and energy consumption, and enhancing the adaptability and robustness of the underactuated robot in unknown underwater environments. Attached Figure Description

[0010] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a flowchart of a path planning and control method for underactuated underwater robots based on reinforcement learning, proposed in an embodiment of this application; Figure 2 This is a flowchart of path planning and tracking control for an underactuated AUV in one embodiment of this application; Figure 3 This is a schematic diagram of the thruster output direction of an underdriven AUV in one embodiment of this application; Figure 4 This is a flowchart of training an agent through reinforcement learning in one embodiment of this application; Figure 5(a) shows the tracking results of the path planned using this method; Figure 5(b) shows the tracking results of the path planned using the traditional interpolation algorithm; Figure 6 This is a schematic diagram of a reinforcement learning-based underactuated underwater robot tracking and control device proposed in one embodiment of this application. Detailed Implementation

[0012] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0013] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments.

[0014] In the various embodiments of this application, it should be understood that the sequence number of each process described below does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0015] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects as detailed in this application.

[0016] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other.

[0017] Current path planning for autonomous underwater vehicles (AUVs) primarily employs schemes based on classical graph structures and rapid exploration random trees based on sampling techniques. These schemes neglect the vehicle's kinematic constraints, and the dynamically infeasible reference trajectories generated in practice lead to low tracking accuracy and high energy consumption for underactuated AUVs. Related schemes utilize learning-based planning methods, but these still suffer from excessive computational load and insufficient interpretability, resulting in significantly increased planning costs and poor controllability.

[0018] In the tracking and control of underwater vehicles, the classic PID control scheme is widely used in industrial scenarios due to its simple structure and ease of implementation. However, its performance degrades in the complex and uncertain underwater environment due to difficulties in parameter tuning. Sliding mode control employs a nonlinear control strategy and can be used to handle dynamic uncertainties and unmodeled external disturbances, but its control signal is prone to jitter and high-frequency oscillations, resulting in low tracking accuracy and poor robustness. Model predictive control relies on model accuracy for its control performance, but suffers from high computational costs and insufficient real-time performance. Related end-to-end reinforcement learning schemes based on neural networks lack stability guarantees, require large amounts of data for training, are costly, and cannot adapt well to unknown underwater environments and changing influencing factors, exhibiting poor robustness.

[0019] The present application will now be described in detail with reference to the accompanying drawings and embodiments.

[0020] Figure 1 This is a flowchart of a path planning and control method for underactuated underwater robots based on reinforcement learning, proposed in an embodiment of this application. Figure 1 As shown, the method includes: S1: Determine the current position of the target robot as the starting point. Based on the kinematic constraints of the target robot, select multiple path points in the three-dimensional space between the starting point and the ending point to generate the target path with the minimum comprehensive cost. The comprehensive cost includes at least: path length cost and energy consumption cost. S2: During the process of the target robot tracking the target path, the tracking error is calculated based on the target path, the current pose and speed of the target robot; S3: Input the tracking error and the current speed of the target robot into the tracking controller to obtain the corresponding PID parameter adjustment amount; the tracking controller is obtained through reinforcement learning training based on the tracking error and speed of the target robot, with the optimization objective of maximizing the comprehensive reward; the comprehensive reward is related to the tracking error and energy consumption of the target robot; S4: Based on the adjustment amount of the PID parameters, control the target robot to track the target path.

[0021] The underactuated underwater robot tracking and control method provided in this application can track the navigation state of an underactuated robot and plan the optimal path in real time. First, based on the current position of the target robot and the target position to be reached, the start and end points of the planned path are determined. Based on the dynamic constraints of the target robot and the objective of minimizing both path length and energy consumption costs, multiple dynamically reachable path points are selected in the three-dimensional space between the start and end points, thereby establishing the target path with the minimum overall cost from the start to the end point.

[0022] Then, during the target robot's navigation along the target path, its current pose (including its position in 3D space and its deflection angle relative to the world fixed coordinate system) and velocity are recorded. The real-time tracking error of the target robot is calculated using the target path as a reference trajectory. A pre-trained tracking controller processes the real-time tracking error and outputs the current PID parameter adjustment. Next, based on the PID parameter adjustment, a derivation and transformation are performed to ultimately achieve real-time adjustment of the thrust of each thruster of the target robot, controlling the target robot to accurately track the target path during navigation.

[0023] In the path planning phase, the kinematic constraints of the target robot are integrated to ensure that all points on the target path generated based on the selected path points are within the dynamic reachability of the target robot. This means the generated target path serves as a dynamically feasible reference trajectory for the underactuated AUV. Simultaneously, the selection of path points considers the combined minimum cost of path length and energy consumption, minimizing the target path length while reducing energy consumption. This improves the efficiency of the vehicle's movement to the target location (destination) while conserving energy. In the subsequent trajectory tracking phase, a tracking controller pre-trained through reinforcement learning is used to monitor the target robot's pose and velocity in real time and adaptively adjust the PID parameters. This enhances the adaptability to unknown factors in the underwater environment while maintaining the interpretability and robustness of PID control.

[0024] Figure 2 This is a flowchart illustrating online dynamic path planning and tracking control of an underactuated AUV in one embodiment of this application. Figure 2 As shown, a target path (reference trajectory) is planned based on the endpoint position required by the target robot (underactuated AUV), using a method combining the AUV's kinematic model. Based on the planned target path and the actual position of the target robot collected in real-time by sensors during navigation, the real-time tracking error is calculated. Based on this real-time tracking error, the tracking controller continuously generates PID parameter adjustments using reinforcement learning methods to drive the target robot to accurately track the target path.

[0025] As one embodiment of this application, the current position of the target robot is determined as the starting point. Based on the kinematic constraints of the target robot, multiple path points are selected in the three-dimensional space between the starting point and the ending point to generate a target path with the minimum overall cost, including: Construct a dynamic model of the target robot; Initialize the random tree and use the starting point as the root node; Perform the following path point search steps: Select a bias point in three-dimensional space, and start from the first path point in the random tree that is currently closest to the bias point, heuristically sample the action space to obtain multiple candidate points; calculate the comprehensive cost from the first path point to each candidate point, determine the candidate point with the smallest comprehensive cost as the new path point, and add it to the random tree; After each new path point is added to the random tree, it is determined whether the distance between the new path point and the destination is greater than a first threshold. If the distance between the new path point and the destination is greater than the first threshold, the path point search step is repeated. If the distance between the new path point and the destination is less than or equal to the first threshold, a target path is generated based on all path points contained in the random tree.

[0026] Figure 3 This is a schematic diagram of the thruster output direction of an underactuated AUV in one embodiment of this application. Figure 2 As shown, this AUV possesses six degrees of freedom of motion in three-dimensional space and is equipped with thrusters in four directions. Specifically, the thrusters provide control for the AUV to perform forward, backward, upward, downward, and horizontal rotation along the Z-axis of the world fixed coordinate system, while roll and pitch movements are passively stabilized due to its underactuated design. In this embodiment, the dynamic model of the target robot is first constructed. Specifically, the kinematic model of this underactuated AUV in the world fixed coordinate system can be represented as follows: ;in, ; In the above expression, This is the transformation matrix between the carrier coordinate system and the world fixed coordinate system; It is a 3x3 rotation matrix used to transform the linear velocity in the vehicle coordinate system to the world coordinate system; It is a 3x3 transformation matrix used to convert the angular velocity in the carrier coordinate system to the Euler angle change rate in the world coordinate system; The velocity vector in the carrier coordinate system; p , q , r These are the angular velocities of roll, pitch, and rotation in the carrier coordinate system, respectively. Velocity in the world coordinate system; Indicates the pose of the AUV; , , These represent the deflection angles of the target robot relative to the fixed world coordinate system around the x-axis, y-axis, and z-axis, respectively.

[0027] In practical applications, AUVs need to maintain attitude stability, i.e., roll yaw angle. With pitch and yaw angle Approaching zero, the corresponding angular velocity p=q=0. Furthermore, due to the design characteristics of an underactuated AUV, its lateral velocity v during navigation is negligible, i.e., v=0. Therefore, the above transformation matrix... It can be simplified to: .

[0028] In the path planning phase, the kinematic model of the target robot is combined with the Fast Random Tree Search (RRT) algorithm to select path points in the three-dimensional space between the starting point and the ending point. Traditional RRT algorithms explore the state space by randomly sampling and progressively building a search tree. However, in high-dimensional spaces, classic RRT algorithms often produce dynamically infeasible paths. In this embodiment, kinematic constraints are incorporated into the sampling and tree expansion process to overcome this deficiency.

[0029] First, a random tree is initialized with the starting point as the root node. A pathfinding step is then performed, expanding the random tree by heuristically sampling candidate points in 3D space. During this expansion, the candidate point with the lowest overall cost is added to the random tree each time, and the corresponding step count is incremented by 1. After each addition of a new path point, the distance between the new path point and the destination is checked against a first threshold. If the distance is less than or equal to the first threshold, the path point is considered very close to the destination, and the minimum-cost path planning from the starting point to the destination is considered complete. Conversely, if the distance between the newly added path point and the destination is greater than the first threshold, further pathfinding is needed to approach the destination. The planned target path is a discrete set of points. During subsequent path tracking, the various thrusters of the target robot are controlled to track these discrete points.

[0030] Specifically, each time the random tree is expanded, a bias point is selected in 3D space as a reference. Starting from the first path point in the current random tree that is closest to the bias point, multiple actions are heuristically sampled from the action space. The position of the target robot after performing the corresponding action at the first path point is taken as a candidate point. The comprehensive cost required to reach each candidate point from the first path point is calculated. The optimal candidate point is selected and added to the random tree based on the principle of minimizing the combined path length cost and energy consumption cost.

[0031] In one embodiment, the path point lookup process is as follows: (1) Set the starting point ,end Maximum number of sampling times Time step Based on the root node Initialize a random tree; (2) Randomly sample a state And determine its corresponding position in three-dimensional space as the bias point. ; (3) Find the distance bias point in the random tree The nearest first path point Starting from the first path point, the action space is heuristically sampled, and multiple candidate points are determined based on multiple actions. (4) Calculate the comprehensive cost of each candidate point and select the candidate point with the minimum comprehensive cost as the new path point. Expand the random tree. The pose of the new path point is calculated using the following expression: ; in, The pose of the first path point; These are pose smoothing parameters; It is the Jacobian matrix corresponding to the first path point, and S is the velocity coordinate transformation matrix (defining the positive direction of velocity). The speed corresponding to the current action; Let S be the time step. Matrix S is used for coordinate transformation and has the following form: .

[0032] As one embodiment of this application, a bias point is selected in three-dimensional space, and starting from the first path point in the random tree that is currently closest to the bias point, the action space is heuristically sampled to obtain multiple candidate points, including: The endpoint is selected as the bias point according to a first probability, and a point is randomly sampled in three-dimensional space as the bias point according to a second probability; the sum of the first probability and the second probability is 1. Calculate the distance error between the first path point and the deviation point, and input the distance error into the P controller to obtain the first speed control command; The first speed control command is used as the sampling center for Gaussian sampling to obtain multiple candidate action commands; wherein, the range of the Gaussian sampling distribution decreases as the distance between the first path point and the deviation point decreases; Based on each candidate action command and the speed of the target robot at the first path point, the posture and speed of the candidate point corresponding to each candidate action command are determined.

[0033] Traditional RRT algorithms use random sampling to select candidate points, which leads to huge computational costs and low efficiency. This embodiment does not use traditional random sampling; instead, it combines heuristic sampling with the expansion of the random tree to select candidate points. This accelerates pathfinding, saves computational resources, and further improves online response efficiency. Heuristic sampling introduces a target bias strategy, increasing the probability of sampling points near the target region, thereby reducing unnecessary exploration.

[0034] Before each expansion, the endpoint is first selected as the bias point according to the first probability (e.g., 0.3), and a random sampling point is selected as the bias point according to the second probability (e.g., 1-0.3=0.7).

[0035] Then, during heuristic motion space sampling starting from the first path point, the corresponding motion towards the deflection point is first calculated using proportional control. This motion is then Gaussian sampled, with the sampling range decreasing as it approaches the deflection point, resulting in multiple motions. Specifically, the distance error between the first path point and the deflection point is first calculated and input into the P controller (i.e., the proportional controller) to obtain a first speed control command. This first speed control command is then used as the sampling center for Gaussian sampling, yielding multiple candidate motion commands.

[0036] The range of the Gaussian sampling distribution decreases as the distance between the current path point and the bias point decreases, and its sampling mean... ;in, The reference action output of the P controller; the variance of the sampling. ;in, is the preset basic variance; k is the proportional control coefficient; d is the distance between the two points.

[0037] Next, based on the acquired candidate actions, kinematic equations are used for expansion. Considering factors such as path loss and energy consumption, the candidate point with the minimum overall cost is the current optimal reachable point. Therefore, the candidate point with the minimum overall cost is selected from all current candidate points and added to the random tree as a new path point. Specifically, based on each candidate action instruction and the pose of the first path point, the pose one time step after departing from the first path point is calculated, thus determining the corresponding candidate point.

[0038] As one embodiment of this application, calculating the comprehensive cost from the first path point to each candidate point includes: Based on the number of points traversed from the starting point to the first path point, and the distance between the first path point and each candidate point, the path length loss is calculated. Based on the number of points traversed from the starting point to the first path point, and the speed difference between the target robot at the first path point and the candidate point, the energy consumption loss is calculated. Calculate the offset distance between every two adjacent points as the target robot moves from the starting point to the first path point; Based on the number of points traversed from the starting point to the first path point and the offset distance, the smoothness loss is calculated; Based on the path length loss, energy consumption loss, and smoothness loss, the comprehensive cost from the first path point to each candidate point is calculated.

[0039] In the above embodiments, when calculating the comprehensive cost corresponding to each candidate point, in addition to considering the path length loss and robot energy consumption loss when navigating to the candidate point, the smoothness of the path is also taken into consideration. This improves the smoothness of the path while balancing the path length and energy consumption, avoids excessive oscillations in the subsequent tracking process and frequent switching of the thrusters, and ensures the smooth navigation of the target robot.

[0040] Specifically, a comprehensive loss function is constructed considering path length, energy consumption, and smoothness. The details are as follows: ; in, This is the path length loss function; Let this be the energy loss function; This is the smoothness loss function.

[0041] Path length loss: ; in , Let represent the position coordinates of the i-th path point and its preceding path point, respectively; N is the number of points traversed by the path from the current starting point to the candidate point. Energy loss: ; in, , Let represent the velocity vectors of the i-th path point and its preceding path point, respectively; N is the number of points traversed by the path from the current starting point to the candidate point. Smoothness loss (used to evaluate path smoothness): ; in, Let i be the displacement vector between the i-th and i+1-th path points. ; Let i be the displacement vector between the (i-1)th and i-th path points. N represents the number of points traversed by the path from the current starting point to the candidate point.

[0042] The comprehensive loss function constructed in this embodiment can ensure that the final generated target path is not only short and smooth, but also has high energy efficiency, making it well-suited for underwater missions that require limited energy and long-term operation.

[0043] As one embodiment of this application, before inputting the tracking error and the current speed of the target robot into the tracking controller, the method further includes: Obtain the training path for training; The tracking error of the target robot is calculated based on the offset distance between the position of the target robot and the corresponding target path point in the training path; Based on the tracking error of the target robot and its current speed, a state space is constructed; Based on the parameters of the PID controller, the action space is constructed; Based on the tracking error of the target robot, the speed at the current target path point and the speed at the previous target path point, a comprehensive reward function is constructed. Based on the state space, the action space, and the comprehensive reward function, reinforcement learning is used to train the agent, and the trained agent is used as a tracking controller.

[0044] In one embodiment, reinforcement learning is used to pre-train the tracking controller to process errors and speed during path tracking, outputting real-time PID parameter adjustments. The PID parameter tuning process is modeled as an MDP (Markov Decision Process), described as a quintuple (S, A, P, R, Y). Here, S represents the current state, A represents the current action, P represents the state transition probability, R represents the reward function, and Y represents the discount factor. The definitions of state, action, and reward are as follows: (1) The state space S is defined as follows: ;in, Indicates the current tracking error; This indicates the tracking error from the previous step; Indicates the current speed.

[0045] ; ; in, The coordinates of the current target path point; The coordinates of the AUV corresponding to the current target path point; The coordinates of the AUV corresponding to the previous target path point are used. This method divides the path tracking task into relative error tracking tasks for each path point, facilitating subsequent reinforcement learning training. Considering the continuity of the tracking task, the tracking error from the previous step is also included in the state space. Furthermore, for the smoothness of the AUV's motion, the current velocity is also added to the state space.

[0046] (2) The action space A of the PID controller is defined as follows: These are the three parameters of a PID controller: proportional, integral, and derivative gain. Although an AUV has only four actual thruster outputs, after transformation, they are treated as six control outputs in the control problem. Each control output contains the three parameters of the PID controller, for a total of 18 parameters.

[0047] (3) Reward function R: The reward function evaluates the tracking error reward based on the tracking error between the target robot and the reference target path during the navigation process, and evaluates the energy consumption reward and smoothness reward of the navigation process based on the real-time speed and the speed difference between two adjacent path points during the navigation process.

[0048] Furthermore, based on the state space, action space, and comprehensive reward function, reinforcement learning is used to train the agent to learn how to control the target robot to smoothly and accurately track the reference path.

[0049] As one embodiment of this application, a comprehensive reward function is constructed based on the tracking error of the target robot, the speed corresponding to the current target path point, and the speed corresponding to the previous target path point, including: Based on the tracking error of the target robot, a tracking error reward is constructed; An energy consumption reward is constructed based on the target robot's current speed; Based on the speed of the target robot at the current target path point and the speed at the previous target path point, a smooth reward is constructed. A comprehensive reward function is constructed by combining the tracking error reward, the energy consumption reward, the smoothing reward, and their respective weight coefficients.

[0050] In the above embodiments, the comprehensive reward function simultaneously considers tracking error, robot energy consumption, and navigation path smoothness. Specifically, it first considers the error during the tracking process, reusing the tracking error in the state space. and Construct a tracking error reward item: ; Then, considering the energy consumption when completing the task, the speed of the AUV is used. To represent, construct an energy consumption reward item: ; Furthermore, the smoothness of completing the tracking task must be considered. Sudden acceleration or deceleration could not only cause control jitter but also damage the thrusters. This part is represented by the difference in velocity between two consecutive measurements, yielding a smoothness bonus: ; in, The speed of the AUV corresponding to the current target path point; The speed of the AUV corresponding to the previous target path point; The three reward components are weighted and summed to construct the comprehensive reward function as follows: ; in, , , These are the weight coefficients corresponding to tracking error, energy consumption, and smoothness, respectively. In subsequent reinforcement training, these weight coefficients are first set to initial values ​​based on experience, and then adjusted according to the training results.

[0051] As one embodiment of this application, based on the state space, the action space, and the comprehensive reward function, a reinforcement learning agent is trained, including: According to a first time interval, a corresponding action is obtained from the policy network based on the current state of the target robot; the action is generated by the policy network based on the action space. The action is input into the PID controller to obtain the corresponding second speed control command; Based on the second speed control command, the target robot is controlled to move forward to obtain a new state, and the corresponding comprehensive reward is calculated based on the new state. Each state and its corresponding comprehensive reward are used as empirical data and stored in the sample set; According to the target number, multiple sample groups are randomly sampled from the sample set, and the policy network is iteratively updated based on the multiple sample groups.

[0052] The parameter tuning process of a PID controller is a complex nonlinear optimization problem, and traditional tuning methods are difficult to adapt to the dynamic changes in the complex underwater environment. This embodiment uses reinforcement learning to train an intelligent agent, thereby achieving adaptive real-time adjustment of the target robot's PID parameters.

[0053] In this embodiment, the training architecture revolves around a closed-loop interaction of a high-fidelity AUV dynamics simulation model. The target robot's dynamics model is constructed based on the Fossen model, as follows: ; in, It is the inertia matrix that includes the added mass; It is acceleration; It consists of the Coriolis and the centripetal matrix; This represents the first-order damping matrix; It is a second-order damping matrix; It is the gravity matrix; It is the buoyancy matrix; The control input is generated by four thrusters; It is the thruster allocation matrix (TAM).

[0054] During training, the target robot's state is collected at fixed first time intervals, and the corresponding actions are determined in the action space through a policy network. A PID controller generates a second speed control command based on the acquired actions to control the thrusters to move forward, resulting in a new state. The corresponding comprehensive reward is calculated based on the new state. The continuously collected state and reward data are stored as empirical data in a sample set. After acquiring a certain number of samples, multiple sample groups are randomly sampled from the sample set to iteratively update the policy network.

[0055] Figure 4 This is a flowchart illustrating the training of an agent using reinforcement learning in one embodiment of this application. For example... Figure 4 As shown, the Soft Actor-Critic (SAC) algorithm is used to maximize the entropy and expectation of the policy, a double Q-network is used to estimate the action value, and the policy network (Actor network) is updated through the objective function. During the target robot's tracking of the training path, the simulation dynamics model interacts with the simulation environment to generate the next new state. The Actor network (policy network) generates actions based on the current state, uses forward inference to obtain the next state and reward value, and uses the collected state, action, reward at the current time t, and state at time t+1 as empirical data. The data is stored in the experience replay buffer. After a certain amount of experience data is collected, multiple batches are randomly sampled from the experience replay buffer. The Critic network is used to evaluate the effectiveness of the current policy, and the Actor network is updated.

[0056] Compared to traditional PID control schemes, this scheme combines classic PID parameter tuning control with reinforcement learning, which can improve the accuracy and stability of tracking control. Compared to traditional end-to-end reinforcement learning schemes, this scheme learns and optimizes the PID controller parameters, thereby affecting the performance of the PID control rate. It can improve the robustness of underactuated AUV tracking control while preserving its control interpretability and ensuring the operability of tracking control.

[0057] As one embodiment of this application, the policy network is iteratively updated based on the plurality of sample groups, including: Initialize the parameters of the two Q networks; For each sample group, perform the following steps to iteratively update the policy network: Based on the latest state of the target robot, obtain the corresponding action from the current policy network and calculate the corresponding entropy; Calculate the target Q value based on the current immediate reward; Based on the error between the target Q value and the predicted Q values ​​of the two Q networks, the parameters of each Q network are updated; The parameters of the policy network are updated with the optimization direction of maximizing the predicted Q value and maximizing the entropy. A loss function is constructed based on the difference between the actual entropy of the policy currently output by the policy network and the preset target entropy; The coefficients of the regularization term of the entropy are updated with the optimization direction of minimizing the difference between the actual entropy and the target entropy; The target Q-network is updated using a soft update method based on the parameters of the two Q-networks.

[0058] In one embodiment, since the vertical motion along the Z-axis is decoupled in the AUV model—that is, trajectory tracking is primarily in the XY plane—and considering that the vehicle must first adjust its heading to align with the target, the target waypoint is aligned with its current z-axis-based yaw angle. Alignment, meaning the lateral velocity v is approximately zero. Therefore, under this assumption, the tracking error of the world fixed coordinate system... It can be simplified to: ; in, The reference pose obtained by the planning algorithm; This represents the actual pose of the AUV.

[0059] Due to heading angle Aligned, therefore there is That is, error. This error only includes errors in three degrees of freedom.

[0060] After training, the target robot is controlled to track the target path based on the real-time PID parameter adjustments output by the tracking controller. Specifically, the PID controller calculates the control vectors for the AUV's six degrees of freedom based on the PID parameter adjustments output by the tracking controller and the target robot's current tracking error. Since the tracking error is calculated in a fixed world coordinate system, while the thruster input is referenced to the vehicle's fixed coordinate system, the tracking error needs to be transformed to the AUV's vehicle coordinate system and specified for each of the six degrees of freedom. Control Vectors The calculation is as follows: ; Among them, control vector The force / torque required for the corresponding degree of freedom ; Tracking error in the fixed coordinate system of the carrier: diagonal matrix , , These correspond to the proportional, integral, and derivative PID control parameters, respectively.

[0061] Subsequently, by combining the thruster allocation matrix T, the control vectors of the 6 degrees of freedom are converted into the actual output thrust of the 4 thrusters.

[0062] In this embodiment, the beneficial effects of the proposed solution were experimentally verified using a simulation platform. The traditional interpolation planner and the fixed-gain PID scheme were compared in the experiment. The kinematic and dynamic parameters of the target robot simulation model used in this solution are defined and sourced as shown in Table 1 below.

[0063] Table 1

[0064] In the experiment, a starting point and an ending point were given, and five intermediate waypoints (path points) were randomly generated to form the test scenario. The path planning method provided in this scheme was used to generate a reference trajectory, with an average planning time of approximately 0.1 seconds, which meets the real-time requirements of control trajectory tracking. For comparison, the traditional interpolation planning scheme was tested in the experiment, and the comparison results are shown in Figure 5(a) and Figure 5(b).

[0065] Figure 5(a) shows the tracking result of the path planned using this method; Figure 5(b) shows the tracking result of the path planned using the traditional interpolation algorithm. In the figures, the dashed line represents the generated reference trajectory, and the solid line represents the actual running trajectory generated by controlling the target robot to track based on the reference trajectory. It can be seen that the target robot can accurately track the kinematically correct reference trajectory generated by this scheme, while the reference path generated by the traditional interpolation planning scheme does not conform to the kinematic model of the underactuated AUV, and the target robot cannot reliably track it.

[0066] In terms of trajectory tracking control, the tracking control module was trained using the reinforcement training method provided in this scheme and compared with the traditional PID controller. The test was conducted on the reference trajectory generated by the path planning method of this scheme in the above experiment, and the results are shown in Table 2 below.

[0067] Table 2

[0068] To quantitatively evaluate performance, this comparative experiment analyzed multiple indicators, including: tracking time, mean, tracking error, energy consumption, and control variance. Tracking accuracy was evaluated using root mean square error (RMSE). This solution improved by 49.4% compared to the traditional PID controller, and the heading RMSE decreased from 2.13 degrees to 0.97 degrees, significantly enhancing directional control capability.

[0069] In terms of energy efficiency, the total control energy consumption is reduced by approximately 31.2% compared to traditional PID controllers. This improvement stems from the smoother control action achieved through reinforcement learning, avoiding excessive oscillations and frequent thruster switching. Furthermore, the adaptive controller exhibits lower control variance (a reduction of 7.9%), signifying improved stability and smoother thrust output. In summary, the path planning and tracking control scheme presented in this paper outperforms traditional schemes in terms of tracking accuracy, energy consumption, and robustness, validating the feasibility of controlling underactuated AUVs in complex underwater environments.

[0070] Based on the same inventive concept, one embodiment of this application provides a tracking and control device for an underactuated underwater robot based on reinforcement learning. Figure 6 This is a schematic diagram of a reinforcement learning-based underactuated underwater robot tracking and control device 100 according to an embodiment of this application. Figure 6 As shown, the device includes: The path planning module 101 is configured to determine the current position of the target robot as the starting point, and select multiple path points in the three-dimensional space between the starting point and the ending point based on the kinematic constraints of the target robot, so as to generate a target path with the minimum comprehensive cost; the comprehensive cost includes at least: path length cost and energy consumption cost. The monitoring module 102 is configured to calculate the tracking error based on the target path, the current pose and speed of the target robot during the process of the target robot tracking the target path; The control module 103 is configured to input the tracking error and the current speed of the target robot into the tracking controller to obtain the corresponding PID parameter adjustment amount; the tracking controller is obtained through reinforcement learning training based on the tracking error and speed of the target robot, with the optimization objective of maximizing the comprehensive reward; the comprehensive reward is related to the tracking error and energy consumption of the target robot; based on the PID parameter adjustment amount, the target robot is controlled to track the target path.

[0071] As one embodiment of this application, the path planning module 101 is specifically configured to perform the following steps: Construct a dynamic model of the target robot; Initialize the random tree and use the starting point as the root node; Perform the following path point search steps: Select a bias point in three-dimensional space, and start from the first path point in the random tree that is currently closest to the bias point, heuristically sample the action space to obtain multiple candidate points; calculate the comprehensive cost from the first path point to each candidate point, determine the candidate point with the smallest comprehensive cost as the new path point, and add it to the random tree; After each new path point is added to the random tree, it is determined whether the distance between the new path point and the destination is greater than a first threshold. If the distance between the new path point and the destination is greater than the first threshold, the path point search step is repeated. If the distance between the new path point and the destination is less than or equal to the first threshold, a target path is generated based on all path points contained in the random tree.

[0072] In one embodiment of this application, the path planning module 101 is configured to select a bias point in three-dimensional space, and starting from the first path point in the random tree that is currently closest to the bias point, heuristically sample the action space to obtain multiple candidate points, including: The endpoint is selected as the bias point according to a first probability, and a point is randomly sampled in three-dimensional space as the bias point according to a second probability; the sum of the first probability and the second probability is 1. Calculate the distance error between the first path point and the deviation point, and input the distance error into the P controller to obtain the first speed control command; The first speed control command is used as the sampling center for Gaussian sampling to obtain multiple candidate action commands; wherein, the range of the Gaussian sampling distribution decreases as the distance between the first path point and the deviation point decreases; Based on each candidate action command and the speed of the target robot at the first path point, the posture and speed of the candidate point corresponding to each candidate action command are determined.

[0073] In one embodiment of this application, the path planning module 101 is configured to calculate the comprehensive cost from the first path point to each candidate point, including: Based on the number of points traversed from the starting point to the first path point, and the distance between the first path point and each candidate point, the path length loss is calculated. Based on the number of points traversed from the starting point to the first path point, and the speed difference between the target robot at the first path point and the candidate point, the energy consumption loss is calculated. Calculate the offset distance between every two adjacent points as the target robot moves from the starting point to the first path point; Based on the number of points traversed from the starting point to the first path point and the offset distance, the smoothness loss is calculated; Based on the path length loss, energy consumption loss, and smoothness loss, the comprehensive cost from the first path point to each candidate point is calculated.

[0074] In one embodiment of this application, the apparatus further includes a training module configured to perform the following steps: Obtain the training path for training; The tracking error of the target robot is calculated based on the offset distance between the position of the target robot and the corresponding target path point in the training path; Based on the tracking error of the target robot and its current speed, a state space is constructed; Based on the parameters of the PID controller, the action space is constructed; Based on the tracking error of the target robot, the speed at the current target path point and the speed at the previous target path point, a comprehensive reward function is constructed. Based on the state space, the action space, and the comprehensive reward function, reinforcement learning is used to train the agent, and the trained agent is used as a tracking controller.

[0075] In one embodiment of this application, the training module is configured to construct a comprehensive reward function based on the tracking error of the target robot, the velocity corresponding to the current target path point, and the velocity corresponding to the previous target path point, including: Based on the tracking error of the target robot, a tracking error reward is constructed; An energy consumption reward is constructed based on the target robot's current speed; Based on the speed of the target robot at the current target path point and the speed at the previous target path point, a smooth reward is constructed. A comprehensive reward function is constructed by combining the tracking error reward, the energy consumption reward, the smoothing reward, and their respective weight coefficients.

[0076] In one embodiment of this application, the training module is configured to train an agent using reinforcement learning based on the state space, the action space, and the comprehensive reward function, including: According to a first time interval, a corresponding action is obtained from the policy network based on the current state of the target robot; the action is generated by the policy network based on the action space. The action is input into the PID controller to obtain the corresponding second speed control command; Based on the second speed control command, the target robot is controlled to move forward to obtain a new state, and the corresponding comprehensive reward is calculated based on the new state. Each state and its corresponding comprehensive reward are used as empirical data and stored in the sample set; According to the target number, multiple sample groups are randomly sampled from the sample set, and the policy network is iteratively updated based on the multiple sample groups.

[0077] Based on the same inventive concept, one embodiment of this application provides a readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method as described in any of the above embodiments of this application.

[0078] Based on the same inventive concept, one embodiment of this application provides an electronic device. The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When executed by the processor, the computer program implements the steps of the method described in any of the above embodiments of this application.

[0079] Regarding the apparatus in the above embodiments, the specific methods by which each module performs its operations have been described in detail in the embodiments related to the method, and will not be elaborated upon here. The tracking and control device in the embodiments of this application can be a device, or a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device; this application does not specifically limit its functionality. The above descriptions are merely preferred embodiments of this application and are not intended to limit the application. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of this application should be included within the protection scope of this application.

[0080] For the sake of simplicity, the method embodiments are described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and components involved are not necessarily essential to this application.

[0081] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus, or computer program products. Therefore, embodiments of this application can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of this application can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0082] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0083] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0084] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0085] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, this application is to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of this application.

[0086] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0087] The above provides a detailed description of the reinforcement learning-based underactuated underwater robot path planning and control method provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and its core ideas. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A path planning and control method for an underactuated underwater robot based on reinforcement learning, characterized in that, include: The current position of the target robot is determined as the starting point. Based on the kinematic constraints of the target robot, multiple path points are selected in the three-dimensional space between the starting point and the ending point to generate the target path with the minimum overall cost. The overall cost includes at least: path length cost and energy consumption cost; During the process of the target robot tracking the target path, the tracking error is calculated based on the target path, the current pose and speed of the target robot; The tracking error and the current speed of the target robot are input into the tracking controller to obtain the corresponding PID parameter adjustment. The tracking controller is trained through reinforcement learning based on the tracking error and speed of the target robot, with the optimization objective of maximizing the comprehensive reward. The comprehensive reward is related to the tracking error and energy consumption of the target robot. Based on the adjustment of the PID parameters, the target robot is controlled to track the target path.

2. The path planning and control method for underactuated underwater robots based on reinforcement learning according to claim 1, characterized in that, The current position of the target robot is determined as the starting point. Based on the kinematic constraints of the target robot, multiple path points are selected in the three-dimensional space between the starting point and the ending point to generate the target path with the minimum overall cost, including: Construct a dynamic model of the target robot; Initialize the random tree and use the starting point as the root node; Perform the following path point search steps: Select a bias point in three-dimensional space, and start from the first path point in the random tree that is currently closest to the bias point, heuristically sample the action space to obtain multiple candidate points; calculate the comprehensive cost from the first path point to each candidate point, determine the candidate point with the smallest comprehensive cost as the new path point, and add it to the random tree; After each new path point is added to the random tree, it is determined whether the distance between the new path point and the destination is greater than a first threshold. If the distance between the new path point and the destination is greater than the first threshold, the path point search step is repeated. If the distance between the new path point and the destination is less than or equal to the first threshold, a target path is generated based on all path points contained in the random tree.

3. The path planning and control method for underactuated underwater robots based on reinforcement learning according to claim 2, characterized in that, In three-dimensional space, a bias point is selected, and starting from the first path point in the random tree that is closest to the bias point, the action space is heuristically sampled to obtain multiple candidate points, including: The endpoint is selected as the bias point according to a first probability, and a point is randomly sampled in three-dimensional space as the bias point according to a second probability; the sum of the first probability and the second probability is 1. Calculate the distance error between the first path point and the deviation point, and input the distance error into the P controller to obtain the first speed control command; The first speed control command is used as the sampling center for Gaussian sampling to obtain multiple candidate action commands; wherein, the range of the Gaussian sampling distribution decreases as the distance between the first path point and the deviation point decreases; Based on each candidate action command and the speed of the target robot at the first path point, the posture and speed of the candidate point corresponding to each candidate action command are determined.

4. The path planning and control method for underactuated underwater robots based on reinforcement learning according to claim 2, characterized in that, Calculate the comprehensive cost from the first path point to each candidate point, including: Based on the number of points traversed from the starting point to the first path point, and the distance between the first path point and each candidate point, the path length loss is calculated. Based on the number of points traversed from the starting point to the first path point, and the speed difference between the target robot at the first path point and the candidate point, the energy consumption loss is calculated. Calculate the offset distance between every two adjacent points as the target robot moves from the starting point to the first path point; Based on the number of points traversed from the starting point to the first path point and the offset distance, the smoothness loss is calculated; Based on the path length loss, energy consumption loss, and smoothness loss, the comprehensive cost from the first path point to each candidate point is calculated.

5. The path planning and control method for underactuated underwater robots based on reinforcement learning according to claim 1, characterized in that, Before inputting the tracking error and the target robot's current speed into the tracking controller, the method further includes: Obtain the training path for training; The tracking error of the target robot is calculated based on the offset distance between the position of the target robot and the corresponding target path point in the training path; Based on the tracking error of the target robot and its current speed, a state space is constructed; Based on the parameters of the PID controller, the action space is constructed; Based on the tracking error of the target robot, the speed at the current target path point and the speed at the previous target path point, a comprehensive reward function is constructed. Based on the state space, the action space, and the comprehensive reward function, reinforcement learning is used to train the agent, and the trained agent is used as a tracking controller.

6. The path planning and control method for underactuated underwater robots based on reinforcement learning according to claim 5, characterized in that, Based on the tracking error of the target robot, the speed at the current target path point, and the speed at the previous target path point, a comprehensive reward function is constructed, including: Based on the tracking error of the target robot, a tracking error reward is constructed; An energy consumption reward is constructed based on the target robot's current speed; Based on the speed of the target robot at the current target path point and the speed at the previous target path point, a smooth reward is constructed. A comprehensive reward function is constructed by combining the tracking error reward, the energy consumption reward, the smoothing reward, and their respective weight coefficients.

7. The path planning and control method for underactuated underwater robots based on reinforcement learning according to claim 5, characterized in that, Based on the state space, the action space, and the comprehensive reward function, a reinforcement learning agent is trained, including: According to a first time interval, a corresponding action is obtained from the policy network based on the current state of the target robot; the action is generated by the policy network based on the action space. The action is input into the PID controller to obtain the corresponding second speed control command; Based on the second speed control command, the target robot is controlled to move forward to obtain a new state, and the corresponding comprehensive reward is calculated based on the new state. Each state and its corresponding comprehensive reward are used as empirical data and stored in the sample set; According to the target number, multiple sample groups are randomly sampled from the sample set, and the policy network is iteratively updated based on the multiple sample groups.

8. A tracking and control device for an underactuated underwater robot based on reinforcement learning, characterized in that, For performing the method as described in any one of claims 1-7, comprising: The path planning module is configured to determine the current position of the target robot as the starting point, and select multiple path points in the three-dimensional space between the starting point and the ending point based on the kinematic constraints of the target robot, so as to generate the target path with the minimum comprehensive cost; the comprehensive cost includes at least: path length cost and energy consumption cost; The monitoring module is configured to calculate the tracking error based on the target path, the current pose and speed of the target robot during the process of the target robot tracking the target path; The control module is configured to input the tracking error and the current speed of the target robot into the tracking controller to obtain the corresponding PID parameter adjustment amount; the tracking controller is obtained through reinforcement learning training based on the tracking error and speed of the target robot, with the optimization objective of maximizing the comprehensive reward; the comprehensive reward is related to the tracking error and energy consumption of the target robot; based on the PID parameter adjustment amount, the target robot is controlled to track the target path.

9. An electronic device, characterized in that, It includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the method as described in any one of claims 1-7.

10. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the method as described in any one of claims 1-7.

Images