Target driving and online avoidance method for ocean underwater robot

Abstract

The application provides a target driving marching and online avoidance method for a marine underwater robot, wherein the underwater robot, a terminal station and an obstacle are taken as nodes, each node is endowed with a state feature to form a node feature matrix; a self-attention mechanism is used to endow neighbor nodes in the node feature matrix with different weights to generate a state feature vector representing the topological relationship of the environment; a candidate action of the underwater robot is generated according to the state feature vector; based on the kinematic constraint of the underwater robot, the candidate action is screened and constrained in a dynamic window determined by the intersection of the maximum speed space, the reachable speed space and the allowed speed space of the underwater robot, the final action without collision and meeting the kinematic constraint is selected from the dynamic window to control the underwater robot to execute the final action. The method of the application enables the underwater robot to adjust the driving direction according to real-time sensing information, effectively avoids and optimizes the operation track, and balances the operation efficiency and resource consumption.

Description

Technical Field

[0001] This invention belongs to the field of underwater robot path planning technology, specifically relating to a target-driven movement and online avoidance method for marine underwater robots. Background Technology

[0002] Against the backdrop of the accelerated penetration of intelligent technologies into the engineering field, autonomous underwater robots have become the core operational platform for deep-sea resource exploration and marine space monitoring, playing a crucial role in improving the efficiency of marine resource utilization and ensuring the stability of marine ecosystems. However, the underwater space is characterized by high unstructuredness, strong disturbance, and limited perception, which severely restricts the driving stability and route reliability of underwater robots in complex task areas. Especially in situations where responsive avoidance occurs frequently and prior spatial information is lacking, traditional path generation and collision avoidance methods based on offline models are difficult to meet the requirements of real-time performance and safety. Therefore, path generation and collision avoidance methods with spatial topology perception capabilities and autonomous decision-making mechanisms have become a key research direction for achieving highly robust autonomous driving of underwater robots, and have attracted widespread attention in the current field of underwater intelligent control.

[0003] During mission execution, underwater robots need to perceive the spatial state in real time and adjust their movement strategy accordingly to achieve efficient transfer from the initial position to the target area. The design of the obstacle avoidance mechanism must consider the robot's motion constraints to ensure a coordinated optimization of trajectory continuity, operational stability, and obstacle avoidance capabilities. Existing mainstream route generation techniques can be broadly categorized into two types: the first is a global spatial modeling-based overview-style navigation strategy. This approach relies on prior knowledge of the sea area to construct a comprehensive spatial map and uses heuristic search mechanisms to derive an optimal, collision-free trajectory. Representative techniques include the traditional A* algorithm and various intelligent evolutionary search algorithms, such as ant colony optimization, artificial bee colony optimization, and genetic mechanisms. The second type is a response strategy oriented towards local real-time perception. This primarily utilizes the sensing devices onboard the underwater robot to acquire local spatial changes and responsively update the local map, thereby adjusting the current movement... The traces are corrected in a timely manner. Among the technical means, the more typical implementation forms include potential adjustment mechanism based on artificial potential energy field, velocity space screening strategy and adaptive learning control framework that integrates deep representation and policy optimization; in particular, the latter uses deep reinforcement learning (DRL) to integrate perception and decision-making processes, which has strong policy generation and dynamic adjustment capabilities and shows certain practicality in uncertain spaces. However, it also has problems such as policy convergence depending on interaction density, high sensitivity of model parameters, and limited generalization ability. Typical technical means include variants such as DQN, DDPG, PPO, SAC and TD3.

[0004] Currently, DRL-based target-driven movement and online avoidance methods are increasingly widely used in underwater robot control. However, several core challenges remain in practical deployment: Firstly, the reward signals obtained by the agent during the exploration phase are generally scarce, resulting in low policy update efficiency and difficulty in forming effective feedback in complex spaces. Secondly, most current research focuses on the basic objective of avoiding conflict events, lacking systematic modeling and integration of three-dimensional spatial structural characteristics and underwater robot kinematic constraints. This limits the policy's adaptability and control robustness in real multi-dimensional task spaces. Consequently, existing path generation and automatic detour methods suffer from insufficient environmental modeling capabilities, lack of adaptability in control strategies, and slow training convergence speed. Summary of the Invention

[0005] This invention proposes a target-driven movement and online avoidance method for marine underwater robots, which solves the problem of the lack of environmental adaptability in the control strategies of existing path generation methods.

[0006] To address the aforementioned technical problems, this invention provides a method for target-driven movement and online avoidance of an underwater robot, comprising the following steps:

[0007] Step S1: Treat the underwater robot, terminal station, and obstacle as nodes, assign state features to each node, and form a node feature matrix;

[0008] Step S2: Assign different weights to neighboring nodes in the node feature matrix through a self-attention mechanism to generate a state feature vector representing the topological relationship of the environment;

[0009] Step S3: Generate candidate actions for the underwater robot based on the state feature vector;

[0010] Step S4: Based on the kinematic constraints of the underwater robot, within the dynamic window determined by the intersection of the underwater robot's maximum velocity space, reachable velocity space, and allowable velocity space, the candidate actions are screened and constrained. The final action that is collision-free and satisfies the kinematic constraints is selected from the dynamic window, and the underwater robot is controlled to execute the final action.

[0011] Preferably, the state characteristics of the underwater robot in step S1 include spatial coordinates. Action information Pitch angle Distance between the station and the terminal station The state characteristics of the terminal station include spatial coordinates. The obstruction includes stationary obstructions and moving obstructions, and the state characteristics of the stationary obstruction include spatial coordinates. The state characteristics of the motion-impeding body include Speed ​​of movement Relative speed with underwater robot Relative motion direction with the underwater robot Distance between the underwater robot .

[0012] Preferably, step S2 involves processing the node feature matrix using a graph attention network to generate the state feature vector, including the following steps:

[0013] Step S21: Use a multilayer perceptron to perform a linear transformation on the node feature matrix, mapping the state features in the node feature matrix to a latent feature space of a unified dimension;

[0014] The expression for linearly transforming the node feature matrix using a multilayer perceptron is as follows:

[0015] ;

[0016] ;

[0017] In the above formula, Let be the feature vector of the i-th node; This represents a multilayer perceptron; The features of the i-th node; Let be the network weight matrix of the MLP for the i-th node; This represents the total number of nodes; The characteristic matrix;

[0018] Step S22: Calculate the weights between nodes using the attention mechanism, and normalize the calculated weights;

[0019] The expression for calculating the weights between nodes using the attention mechanism is as follows:

[0020] ;

[0021] In the formula, Represents a node For nodes The importance of; It is a non-linear activation function; A learnable parameter vector; Indicates a splicing operation; The weight matrix is ​​a learnable matrix; For the first The feature vector of each node; For nodes The set of neighboring nodes;

[0022] Step S23: Use the normalized weights to perform a weighted summation of the feature vectors of all neighboring nodes, and update the node's state feature vector:

[0023] ;

[0024] In the formula, For nodes Updated state characteristics; It is a linear activation function;

[0025] Step S24: Employ a multi-head attention mechanism to concatenate the outputs of all attention heads to obtain a state feature vector representing the topological relationships of the environment;

[0026] The expression for the state feature vector is:

[0027] ;

[0028] ;

[0029] ;

[0030] In the above formula, The output of the graph attention network; , These are the output state features after passing through the first and second graph attention convolutional layers, respectively. , This represents the weight matrix of the first and second graph attention convolutional layers. , These are the first graph attention convolutional layer and the second graph attention convolutional layer, respectively. Let be the adjacency matrix of the nodes.

[0031] Preferably, in step S3, the state feature vector is input into a policy network based on deep reinforcement learning, and the policy network generates the candidate action.

[0032] Preferably, the deep reinforcement learning-based policy network is trained using a composite reward function, which includes an action reward, a terminal reward, and a collision penalty. The action reward includes an orientation reward, a distance reward, and a velocity reward. The expression for the action reward is:

[0033] ;

[0034] ;

[0035] ;

[0036] ;

[0037] In the above formula, Reward for actions; As a reward for orientation; As a distance reward; As a speed reward; To evaluate the angular components in the function; For the distance component in the evaluation function; For the velocity component in the evaluation function; The attenuation coefficient; The pitch angle of the underwater robot; The x and y coordinates of the terminal station; Let x and y be the coordinates of the underwater robot. This is the distance between the underwater robot and the nearest stationary obstacle. The distance between the underwater robot and the nearest obstacle to its movement; The safety zone radius for underwater robots; , , These are the sway speed, heave speed, and yaw speed of the underwater robot, respectively. , , These represent the maximum values ​​of the underwater robot's sway velocity, heave velocity, and yaw velocity, respectively.

[0038] The expression for the terminal reward is:

[0039] ;

[0040] In the formula, As a reward for the final stop; The distance between the underwater robot and the terminal station;

[0041] The expression for the collision penalty is:

[0042] ;

[0043] In the formula, For collision penalties; for objects that impede movement, Let be the Euclidean distance between the centers of the underwater robot and the moving obstacle. For a stationary obstacle, The height difference between the underwater robot and the stationary obstacle; To hinder the radius of the body.

[0044] Preferably, the deep reinforcement learning is a flexible actuator-evaluator algorithm.

[0045] Preferably, the training process of the deep reinforcement learning-based policy network adopts a priority experience replay mechanism.

[0046] Preferably, the priority experience replay mechanism calculates the priority of the experience based on the temporal difference error of the policy network and the Q network:

[0047] ;

[0048] ;

[0049] ;

[0050] In the above formula, The priority of the i-th experience; for; Score the priority of the i-th sample; To prevent a constant with a sampling probability of 0; Adjustment parameters to balance the temporal difference errors of the policy network and the Q network; , , These are the temporal difference errors of the two networks and the policy network, respectively.

[0051] Preferably, the expression for the maximum velocity space in step S4 is:

[0052] ;

[0053] In the formula, For maximum velocity space; The speed of the underwater robot; The sway speed of the underwater robot; ; The heave / sink rate of the underwater robot; ; The yaw speed of the underwater robot; ;

[0054] The expression for the reachable velocity space is:

[0055] ;

[0056] In the formula, For reachable velocity space; , , The current pitch speed, heave speed, and yaw speed of the underwater robot; , , These are the maximum linear and angular accelerations of the underwater robot. The time step of the dynamic window;

[0057] The expression for the allowed velocity space is:

[0058] ;

[0059] ;

[0060] In the formula, To allow for velocity space; The predicted Euclidean distance from the underwater robot's location to the nearest obstacle; This represents the maximum deceleration of the underwater robot.

[0061] Preferably, both the candidate action and the final action are action vectors that include the desired sway speed, heave speed and yaw rate.

[0062] The beneficial effects of the present invention include at least the following:

[0063] 1. By modeling robots, terminals, and obstacles as nodes and dynamically allocating neighbor weights using a self-attention mechanism, the most critical spatial relationships for the current decision can be adaptively highlighted, avoiding the excessive reliance on the global map by traditional grid or potential field methods, and significantly improving the representation accuracy and generalization ability in sparse prior and structurally abrupt scenarios.

[0064] 2. Based on the candidate actions output by the policy network, a dynamic window consisting of the intersection of the maximum, reachable, and allowed velocity spaces is introduced for secondary screening. This ensures that the final action is not only feasible at the topology level, but also executable under hard constraints such as velocity, acceleration, and minimum turning radius. This effectively suppresses collisions and jitter caused by model mismatch or environmental disturbances.

[0065] 3. The three steps of state feature extraction, action generation and constraint checking are tightly coupled and can be optimized end-to-end within a unified framework. The parallel computing characteristics of the self-attention mechanism and the low-dimensional and efficient search of the dynamic window ensure millisecond-level response on the embedded computing platform, meeting the dual requirements of underwater robots for real-time collision avoidance and continuous navigation. Attached Figure Description

[0066] Figure 1 This is a schematic diagram of the method flow according to an embodiment of the present invention;

[0067] Figure 2 This is a schematic diagram of the coordinate system of the underwater robot according to an embodiment of the present invention;

[0068] Figure 3 This is a schematic diagram of a target-driven movement and automatic detour scenario of an underwater robot according to an embodiment of the present invention.

[0069] Figure 4This is a schematic diagram of the policy network based on deep reinforcement learning according to an embodiment of the present invention;

[0070] Figure 5 This is a schematic diagram illustrating the safety aspects of an underwater robot according to an embodiment of the present invention;

[0071] Figure 6 This is a diagram showing the movement of the underwater robot in test scenario 1 according to an embodiment of the present invention.

[0072] Figure 7 This is a diagram showing the movement of the underwater robot in test scenario 2 according to an embodiment of the present invention.

[0073] Figure 8 This is a map showing the movement of the underwater robot in test scenario 3 according to an embodiment of the present invention. Detailed Implementation

[0074] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0075] like Figure 1 As shown, this embodiment of the invention provides a target-driven movement and online avoidance method for an underwater robot, including the following steps:

[0076] Step S1: Treat the underwater robot, terminal station, and obstacle as nodes, assign state features to each node, and form a node feature matrix.

[0077] Specifically, firstly, a motion model of the underwater robot is established in a fixed Earth coordinate system. Below, the position vector of the underwater robot can be determined. and azimuth vector .in, , and This represents the spatial coordinates of the underwater robot, while , and These represent the roll, pitch, and yaw angles, respectively. Based on the main body... In the coordinate system, traditional mechanical modeling methods employ Newton's principles of momentum and angular momentum, corresponding to translation and rotation in three directions, respectively. Therefore, the underwater robot has six degrees of freedom. The velocity vector of the underwater robot in each dimension is represented as... ,in , and These represent the swell, transverse swell, and heave velocities, respectively. , and These represent the rates of roll, pitch, and yaw, respectively. The specific details of these variables are as follows: Figure 2 The coordinate system is shown.

[0078] While 3D radar can be used in simulations, this is not the case for most underwater robots in real-world applications. These robots typically use RGB or RGBD cameras pointing forward with a limited field of view and do not change their direction of movement rapidly. Therefore, path generation and obstacle avoidance systems only consider sway. heave speed and yaw motion These three degrees of freedom are used to perform control, ignoring , and Therefore, in the control of underwater robots, the motion vector... It can be defined as:

[0079] .

[0080] Before processing node features using a Graph Attention Network (GAT), the underwater robot's spatial representation in the ocean needs to be transformed into an adjacency matrix. The nodes of the graph include the underwater robot, its terminal station, stationary obstacles, and moving obstacles, while edges represent pairwise interactions between the underwater robot and obstacles or the terminal station. The path generation and automatic detour scenarios for the underwater robot are described below. Figure 3 As shown. In this embodiment of the invention, it is assumed that the underwater robot is not visible to the obstructing object, that is, the underwater robot is invisible to the obstructing object, there is no direct interaction between the underwater robot and the obstructing object, and no edges are set between the obstructing object and the obstructing object node, or between the terminal station and the obstructing object node. These are represented by dashed lines in the scene transition.

[0081] Because the map is too large, adding the entire map to the adjacency matrix would slow down the calculation. Therefore, in this embodiment of the invention, the n closest stationary obstacles to the underwater robot are selected as nodes. Assume there exists a graph... Then the set of nodes Includes an underwater robot node A terminal node m motion-impeding nodes n static obstacle nodes .

[0082] The node set is then defined as:

[0083] ;

[0084] The edge set is defined as:

[0085] ;

[0086] In the research scenario of this invention embodiment, The final node set contains 10 nodes: 1 underwater robot node, 1 terminal station node, 4 moving obstacle nodes, and 4 static obstacle nodes. (Regarding the node set...) Each node contains its own associated feature information, forming a node feature matrix. This feature matrix considers not only the distances between the underwater robot and obstacle nodes and the terminal station, but also the relative velocity and terminal station orientation information.

[0087] Among them, the state characteristics of underwater robots include spatial coordinates Action information Pitch angle Distance between the station and the terminal station ,Right now:

[0088] .

[0089] The expression for calculating the distance between the underwater robot and the terminal station is:

[0090] .

[0091] The state characteristics of the terminal include spatial coordinates ,Right now:

[0092] .

[0093] The state characteristics of a stationary obstruction include spatial coordinates. ,Right now:

[0094] ;

[0095] The state characteristics of a movement-impaired body include Speed ​​of movement Relative speed with underwater robot Relative motion direction with the underwater robot Distance between the underwater robot ,Right now:

[0096] .

[0097] These six pieces of information are obtained directly through the sensing system, while the relative speed magnitude and direction... and the distance to underwater robots It is calculated based on the known information using the following formula:

[0098] ;

[0099] ;

[0100] ;

[0101] ;

[0102] In the above formula, The turning radius of the underwater robot is taken as the hull length in this embodiment of the invention, so as to approximately represent the radius of curvature of the underwater robot during yaw.

[0103] In actual obstacle avoidance situations, it is difficult to accurately collect the instantaneous angular velocity information of moving obstacles, and their angular velocity state has little impact on obstacle avoidance, so their angular velocity is not considered.

[0104] Step S2: Assign different weights to neighboring nodes in the node feature matrix through a self-attention mechanism to generate a state feature vector representing the topological relationship of the environment.

[0105] Specifically, the introduction of the Graph Attention Network (GAT) enables the weights of edges to be variably adjusted according to the collision risk between nodes, overcoming the limitation of traditional techniques that rely solely on spatial distance.

[0106] Because the state spaces of underwater robot nodes differ in dimension from those of other nodes, multi-layer perceptrons (MLPs) are needed to align the dimensions of the state features. Then, these three types of node features are concatenated to form a broader feature space. Simultaneously, a new fixed-length latent state feature vector is obtained. The expression for the latent state extracted by MLPs is:

[0107] ;

[0108] ;

[0109] In the above formula, Let be the feature vector of the i-th node; This represents a multilayer perceptron; The features of the i-th node; Let be the network weight matrix of the MLP for the i-th node; This represents the total number of nodes; The feature matrix is ​​defined as follows: the first row corresponds to the potential state of the underwater robot, and the subsequent rows represent the potential states of the terminal station and the obstacle. This setting is used to process and integrate the differences between the underwater robot and other nodes in the state space, thereby effectively representing their interactions in navigation and online avoidance tasks.

[0110] Next, the weights between nodes are learned through an attention mechanism. The attention coefficient of GAT... Represents the node For nodes The importance of computing nodes. and each of its neighboring nodes The features are weighted and concatenated, and the attention coefficient is calculated through a shared attention mechanism:

[0111] ;

[0112] In the formula, It is a non-linear activation function; The learnable parameter vector is used to calculate the attention score between two node features, which controls how the model measures the correlation or importance between the two node features. Indicates a splicing operation; It is a learnable weight matrix used for linearly transforming node features; For the first The feature vector of each node; For nodes The set of neighboring nodes.

[0113] Using linear activation function It processes linear combinations of features from neighboring nodes to output new features that incorporate neighborhood data. :

[0114] ;

[0115] In the formula, For nodes Updated status characteristics.

[0116] The GAT model used in this embodiment of the invention contains two graph attention convolutional layers. During forward propagation, a high-dimensional representation of the node is learned by multi-head aggregation of node features.

[0117] ;

[0118] ;

[0119] ;

[0120] In the above formula, The output of the graph attention network; , These are the output state features after passing through the first and second graph attention convolutional layers, respectively. , This represents the weight matrix of the first and second graph attention convolutional layers. , These are the first graph attention convolutional layer and the second graph attention convolutional layer, respectively. Let be the adjacency matrix of the nodes, used to represent the connection relationships between nodes in the graph. In this embodiment of the invention, assuming i is an underwater robot and j is an obstacle or terminal station, then... If the value is 1, and assuming that both i and j are obstructions or terminals, then If the value is 0, the node itself is set to 0, and self-connection is not used.

[0121] The structure of a two-layer graph attention convolutional layer helps to further process and refine input features, so as to more effectively handle and understand complex interactive scenarios.

[0122] Step S3: Generate candidate actions for the underwater robot based on the state feature vector.

[0123] Step S4: Based on the kinematic constraints of the underwater robot, within the dynamic window determined by the intersection of the underwater robot's maximum velocity space, reachable velocity space, and allowable velocity space, candidate actions are screened and constrained. The final action that is collision-free and satisfies the kinematic constraints is selected from the dynamic window, and the underwater robot is controlled to execute the final action.

[0124] Specifically, the state feature vector is input into a policy network based on deep reinforcement learning, and the policy network generates candidate actions. In this embodiment of the invention, the following is employed: Figure 4 The Soft Actor-Critic (SAC) algorithm is shown.

[0125] As can be seen from the ship motion model established in step S1, the velocity search space of the dynamic window method is... , and The combination of these elements, therefore, the action space Defined as:

[0126] ;

[0127] in, , and The pitching velocity, heave velocity, and yaw rate of the underwater robot are described respectively. The Actor network in the SAC generates a continuous motion distribution based on the current state, and then samples from it to obtain candidate actions. Without constraints, the sampled actions may exceed the physical limits of the underwater robot. Therefore, it is necessary to impose constraints on the action space of the SAC.

[0128] The velocity search space of Dynamic Window Method (DWA) includes the maximum velocity space. achievable speed space and allowable speed space Three parts. Given the current state of the underwater robot and information about its surrounding environment, the actual velocity search space is defined to satisfy the robot's kinematic constraints. The optimal speed is searched only at the intersection of these three spaces, i.e. By combining the velocity window in DWA and the action selection strategy in SAC, a velocity constraint is imposed on the sampled actions to ensure that the output actions of SAC are within the actual velocity search space. Within this range. This not only ensures that the underwater robot meets kinematic constraints and avoids excessive speed changes, but also improves the performance of the SAC in practical applications, especially in complex and changing environments.

[0129] First, the search space should be limited by the minimum and maximum speeds of the underwater robot. These limitations can be imposed by hardware capabilities or safety considerations. Furthermore, considering the typical forward orientation of underwater robot sensors, the following definitions are provided. To avoid moving backward. This search space This is known as the maximum velocity space.

[0130] ;

[0131] In the formula, The speed of the underwater robot; ; ; ;

[0132] By taking into account the current speed of the underwater robot and its maximum linear acceleration and angular acceleration ( , and This further refines the search space. During the actual movement of the underwater robot, only the time step should be considered. The subsequent reachable speeds. These speeds constitute the reachable dynamic window. :

[0133] ;

[0134] In the formula, , , The current pitch speed, heave speed, and yaw speed of the underwater robot; The time step of the dynamic window;

[0135] Speed ​​is limited to reduce the risk of collisions for underwater robots. (During time step) During this period, by applying candidate velocities The underwater robot's future position is predicted. If the distance between the future position and the nearest obstacle is less than the underwater robot's braking distance, the candidate velocity pair is discarded, and the remaining collision-free velocities form the allowable velocity space. :

[0136] ;

[0137] ;

[0138] In the formula, The predicted Euclidean distance from the underwater robot's location to the nearest obstacle; This represents the maximum deceleration of the underwater robot.

[0139] Because the time step is very short, straight-line distance is used instead of curved distance in this embodiment of the invention. Furthermore, straight-line distance is always smaller, thus lowering the speed limit, which makes navigation and online avoidance safer.

[0140] In this embodiment of the invention, the reward function of the underwater robot is designed in conjunction with the evaluation function in DWA. The reward obtained by the agent at each time step is divided into three parts: normal action reward. Terminal Rewards and collision rewards ,Right now:

[0141] .

[0142] Normal action reward refers to the reward obtained by an underwater robot for moving one time step without collision or reaching the destination. To address the sparse reward problem in SAC (Subsurface Acoustic Coding), this embodiment of the invention improves the reward function based on the evaluation function in DWA (Distributed Winged Assessment). The expression of the improved normal action reward function is:

[0143] .

[0144] In the formula, Reward for actions; As a reward for orientation; As a distance reward; As a speed reward; To evaluate the angular components in the function; For the distance component in the evaluation function; This is for evaluating the velocity component in the function.

[0145] Orientation Reward The angular component corresponds to the evaluation function of DWA. The direction of movement of the underwater robot The angle difference between the destination and the terminal direction is inversely proportional to the reward value; the greater the angle difference, the lower the reward value.

[0146] ;

[0147] In the formula, This is the attenuation coefficient, used to control the sensitivity of orientation error to reward; This refers to the pitch angle of the underwater robot.

[0148] Distance Reward The distance component in the evaluation function corresponding to DWA The reward value is directly proportional to the distance between the underwater robot and the nearest obstacle; the farther away from the obstacle, the higher the reward value.

[0149] ;

[0150] In the formula, This is the distance between the underwater robot and the nearest stationary obstacle. The distance between the underwater robot and the nearest obstacle to its movement; This refers to the safety zone radius of the underwater robot.

[0151] Spatial information during underwater robot navigation is obtained through radar and sensor detection. Among these, the distance between the underwater robot and surrounding obstacles is the most direct and important indicator reflecting current spatial information. The safety domain model design for underwater robots is as follows: Figure 5 As shown, the diagram is represented by a sphere centered on the center point of the underwater robot, with a radius of [missing information]. The radius of the safety domain model is determined by both the sensor's sensing capabilities and the task requirements, and in practical tasks, it depends on the maximum detection range of the sonar, lidar, or vision system used by the underwater robot. This model primarily illustrates the maximum detection range achievable by the underwater robot. The rays emitted from the underwater robot symbolize the detection signals sent by the underwater robot's sensor system, covering a radius of [missing information - likely a radius value]. The complete spherical range. By constructing a safety field for underwater robots, the positional information of multiple obstacles within this safety field can be accurately obtained.

[0152] Speed ​​Reward The velocity component corresponding to the evaluation function of DWA The reward value is directly proportional to the speed of the underwater robot; the faster the speed, the higher the reward value.

[0153] .

[0154] Final Reward It is based on the distance between the underwater robot's current position and the terminal station. The reward given at the terminal increases as the distance between the underwater robot and the terminal decreases.

[0155] ;

[0156] In the formula, This represents the distance between the underwater robot and the terminal station.

[0157] Collision Rewards The reward is based on the distance between the underwater robot's current position and the nearest obstacle. The reward is given when the distance between the underwater robot and the nearest obstacle is less than or equal to the radius of the obstacle. If the terminal station is reached, the reward value is -5; otherwise, the reward value is 0.

[0158] ;

[0159] In this embodiment of the invention, for a motion-impeding body, Let be the Euclidean distance between the underwater robot and the center of the obstacle, and For a stationary obstruction, This can be simplified to the height difference between the underwater robot and the obstacle, and

[0160] like Figure 4 As shown, this embodiment of the invention introduces a priority experience replay mechanism to improve SAC. The idea behind priority experience replay is to replay experiences with higher learning value at a higher frequency. These experience examples are typically either very successful or extremely unsuccessful. After frequently learning these important experiences, the agent can learn positive experiences more quickly and avoid persistent negative behaviors.

[0161] To reduce overestimation and improve training stability, SAC employs two Q-networks. In addition, SAC also includes a policy network. Therefore, the timing difference errors of the three networks need to be considered simultaneously. The timing difference error selection... As an indicator for evaluating empirical values, calculate the priority score for the i-th sample. The expression is:

[0162] ;

[0163] In the formula, Adjustment parameters to balance the temporal difference errors of the policy network and the Q network; , , These are the temporal difference errors of the two networks and the policy network, used to measure the training sensitivity or value contribution of each network to the sample.

[0164] To prevent overfitting of the network, experience is extracted probabilistically to ensure that even experiences with zero temporal difference error can be extracted. The priority value of each experience is defined as follows:

[0165] ;

[0166] ;

[0167] In the above formula, The priority of the i-th experience; for; Score the priority of the i-th sample; To prevent a constant with a sampling probability of 0.

[0168] To verify the effectiveness of the method in unknown spaces, three experimental scenarios were designed. The static obstacle data for these scenarios were derived from the General Bathymetric Chart of the Oceans (GEBCO), and seabed topography data from the latitude and longitude range of (62.93S, 32.29W) to (62.84S, 56.91W) was extracted. The seabed topography data was scaled down, and the starting positions and directions of motion of the underwater robot and the moving obstacle were different in each scenario. Five algorithms—DWA-PPO, IIFDS-DDPG, IIFDS-PPO, IIFDS-SAC, and IIFDS-TD3—were compared with the method GD-SAC of this invention.

[0169] Figure 6 , Figure 7 , Figure 8 Table 1 presents the design results of 3D navigation and collision avoidance using different algorithms in three scenarios. The results include numerical values ​​such as travel distance, average reward, motion threat index, static threat index, average turning angle, and maximum heading change for each scenario. The motion threat index is the sum of the distances between the underwater robot and the nearest moving obstacle when the distance is less than a safe distance threshold. The static threat index is calculated similarly.

[0170] Table 1. Navigation and Collision Avoidance Results in Three Spaces

[0171]

[0172] As shown in Table 1, the method of this embodiment achieves the shortest travel distance and the highest reward value in both scenarios 1 and 2. In scenario 3, the method of this embodiment plans a slightly longer travel distance than IIFDS-SAC, and its reward value is second only to IIFDS-DDPG and IIFDS-TD3. Although IIFDS-DDPG and IIFDS-TD3 plan good tracks in scenario 3, the track quality they generate in scenarios 1 and 2 is lower, leading to unstable driving performance. Overall, the method of this embodiment outperforms other implementations in terms of travel distance and average reward value, demonstrating better stability and robustness.

[0173] Regarding collision avoidance, the paths planned by DWA-PPO and IIFDS-TD3 intersected with stationary obstacles, failing to achieve safe avoidance. Other implementations successfully avoided obstacles and reached the destination safely. However, IIFDS-DDPG and IIFDS-PPO overemphasized avoiding moving obstacles, resulting in longer distances and larger turning angles. The method in this embodiment generates a shorter distance, satisfying both the avoidance requirements and the kinematic constraints of the underwater robot.

[0174] Regarding trajectory smoothness, both DWA-PPO and the method of this embodiment consider the kinematic constraints of the underwater robot when designing the driving path. Based on the results of the average turning angle and the maximum turning angle, the method of this embodiment achieves stable driving and obtains the smoothest planned trajectory.

[0175] Simulation results show that when an underwater robot enters a new scenario with different obstacle configurations, the method of this embodiment can learn to interact with the environment and determine the optimal trajectory from the starting point to the target while avoiding obstacles. Furthermore, the method of this embodiment also imposes kinematic constraints on the underwater robot's movements, shortening the trajectory and reducing turning angles while ensuring safety, significantly reducing obstacle avoidance costs. Compared with the driving path design results of other implementations, the method of this embodiment achieved a 100% success rate in all three scenarios, verifying the superiority of the method of this embodiment in driving path design in three-dimensional space with unknown obstacles.

[0176] This invention, based on a deep reinforcement learning decision-making framework, introduces an adjacency matrix perception module to perform topological representation of key elements within the work area, enhancing the system's ability to identify differences in the importance of surrounding states. Simultaneously, combined with a candidate action selection strategy based on velocity space constraints, it effectively circumvents invalid control commands generated in dynamic space by traditional strategies, improving the physical feasibility and action continuity of trajectory execution. Regarding the strategy evaluation mechanism, a composite activation function integrating spatial proximity, motion direction consistency, and velocity feedback is designed to alleviate the information sparsity problem during strategy convergence. A non-uniform sampling mechanism is employed to weight training experience, focusing on strengthening the learning frequency of key decision segments to accelerate the optimization and iteration of the overall strategy.

[0177] Overall, the experimental results of the embodiments of the present invention are superior to many existing comparative schemes in terms of control stability, track smoothness, task execution efficiency and spatial adaptability.

[0178] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described; only preferred embodiments of the present invention are illustrated. The descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the present invention. As long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0179] It should be noted that those skilled in the art can make various modifications and improvements without departing from the inventive concept, and these all fall within the scope of protection of this invention. Therefore, the scope of protection of this invention should be determined by the appended claims.

Claims

1. A method for target-driven locomotion and online avoidance for an underwater marine robot, characterized in that, The method comprises the following steps: Step S1: taking the underwater robot, the terminal station and the obstacle as nodes, giving each node a state feature to form a node feature matrix; Step S2: giving different weights to the neighbor nodes in the node feature matrix through a self-attention mechanism to generate a state feature vector representing the topological relationship of the environment; Step S3: generating a candidate action of the underwater robot according to the state feature vector, inputting the state feature vector into a policy network based on deep reinforcement learning, generating the candidate action by the policy network, training the policy network based on deep reinforcement learning by using a composite reward function, the composite reward function comprising an action reward, a terminal station reward and a collision penalty, the action reward comprising a heading reward, a distance reward and a speed reward, the expression of the action reward being: ； ； ； ； In the above formula, is the action reward; is the heading reward; is the distance reward; is the velocity reward; is the angle component in the evaluation function; is the distance component in the evaluation function; is the velocity component in the evaluation function; is the decay coefficient; is the pitch angle of the underwater robot; is the horizontal and vertical coordinates of the terminal station; is the horizontal and vertical coordinates of the underwater robot; is the distance between the underwater robot and the nearest stationary obstacle; is the distance between the underwater robot and the nearest moving obstacle; is the safety domain radius of the underwater robot; , , are the surge speed, heave speed and yaw speed of the underwater robot, respectively; , , are the maximum values of the surge speed, heave speed and yaw speed of the underwater robot, respectively. the expression of the terminal station reward being: ； wherein is the terminal reward; is the distance between the underwater vehicle and the terminal. the expression of the collision penalty being: ； wherein is a collision penalty; for moving obstacles, is the Euclidean distance between the underwater vehicle and the center of the moving obstacle; for stationary obstacles, is the height difference between the underwater vehicle and the stationary obstacle; is the obstacle radius; Step S4: based on the kinematic constraints of the underwater robot, screening and constraining the candidate action in a dynamic window determined by the intersection of the maximum speed space, the reachable speed space and the allowable speed space of the underwater robot, selecting a final action without collision and meeting the kinematic constraints from the dynamic window to control the underwater robot to execute the final action.

2. The method of claim 1, wherein: The state features of the underwater robot in step S1 include spatial coordinates , action information , pitch angle , and distance to the terminal station ; The state features of the terminal station include spatial coordinates The obstacles include stationary obstacles and moving obstacles, the state features of the stationary obstacles include spatial coordinates The state features of the moving obstacles include moving speed relative moving speed with the underwater robot relative moving direction with the underwater robot distance between the underwater robot .

3. The method of claim 1, wherein: In step S2, the node feature matrix is processed by a graph attention network to generate the state feature vector, comprising the following steps: Step S21: linearly transforming the node feature matrix by using a multilayer perceptron to map the state features in the node feature matrix to a latent feature space of a unified dimension; the expression of linearly transforming the node feature matrix by using a multilayer perceptron being: ； ； In the above formula, is a feature vector for the i-th node; denotes a multi-layer perceptron; is a feature for the i-th node; is a network weight matrix of the MLP for the i-th node; is the total number of nodes; is a feature matrix; Step S22: calculating the weights between nodes by an attention mechanism and normalizing the calculated weights; the expression of calculating the weights between nodes by an attention mechanism being: ； wherein, represents a node importance of a node ; is a non-linear activation function; is a learnable parameter vector; represents a concatenation operation; is a learnable weight matrix; is a feature vector of the th node; is a set of neighboring nodes of the node ; Step S23: weighting and summing the feature vectors of all neighbor nodes by using the normalized weights to update the state feature vector of the node: ； wherein is a node updated state features; is a linear activation function; Step S24: adopting a multi-head attention mechanism to splice the output results of all attention heads to obtain the state feature vector representing the topological relationship of the environment; the expression of the state feature vector being: ； ； ； In the above formula, is the output of the graph attention network; 、 are the output state features of the first layer graph attention convolution layer and the second layer graph attention convolution layer, respectively; 、 are the weight matrices of the first layer graph attention convolution layer and the second layer graph attention convolution layer; 、 are the first layer graph attention convolution layer and the second layer graph attention convolution layer, respectively; is the adjacency matrix of the nodes.

4. The method of claim 1, wherein: The deep reinforcement learning is a flexible actuator-evaluator algorithm.

5. The method of claim 1, wherein: The training process of the policy network based on deep reinforcement learning adopts a priority experience replay mechanism.

6. A method for target drive travel and online avoidance of an ocean underwater robot according to claim 5, characterized in that: The priority experience replay mechanism calculates the priority of experience according to the time difference error of the policy network and the Q network: ； ； ； In the above formula, is the priority of the ith experience; is the priority of the ith experience; is the priority score of the ith sample; is a constant to prevent sampling probabilities of 0; is a tuning parameter to balance the temporal difference error of the policy network and the Q network; , , are the temporal difference errors of the two networks, respectively, and the policy network.

7. The method of target drive travel and online avoidance for an ocean underwater robot of claim 1, wherein: the expression of the maximum speed space in step S4 being: ； wherein is the maximum velocity space; is the velocity of the underwater robot; is the surge velocity of the underwater robot; ; is the heave velocity of the underwater robot; ; is the yaw velocity of the underwater robot; ; the expression of the reachable speed space being: ； wherein is the reachable velocity space; , , is the current surge, heave and yaw velocity of the underwater vehicle; , , is the maximum linear and angular acceleration of the underwater vehicle; is the time step of the dynamic window; the expression of the allowable speed space being: ； ； wherein is the allowed velocity space; is the predicted Euclidean distance of the underwater robot position to the nearest obstacle; is the maximum deceleration of the underwater robot.

8. The method of target drive travel and online avoidance for an ocean underwater robot of claim 1, wherein: The candidate action and the final action are both action vectors comprising expected surge speed, heave speed and yaw rate.

Images