An autonomous navigation method for an unmanned underwater vehicle

Abstract

This invention relates to the field of autonomous control technology for underwater robots, specifically to an autonomous navigation method for unmanned underwater vehicles (UUVs), comprising the following steps: acquiring environmental perception information and combining it with the UUV's own state information to construct complete state information; establishing a decision network, inputting the complete state information into the decision network, and outputting continuous action commands; designing a composite reward function to optimize the completion rate, safety, and energy efficiency of the navigation task; introducing a lateral vision-assisted decision-making module to establish an auxiliary decision-making model; using experience playback and soft update techniques to perform end-to-end training on the auxiliary decision-making model, saving the trained auxiliary decision-making model, and testing it. This invention significantly improves the autonomous navigation capability of UUVs in complex and unknown underwater environments, effectively balancing task completion, obstacle avoidance safety, and energy economy through multi-constraint rewards and vision-assisted decision-making, and possesses stronger environmental adaptability and engineering practicality.

Description

Technical Field

[0001] This invention relates to the field of autonomous control technology for underwater robots, and more specifically to an autonomous navigation method for unmanned underwater vehicles. Background Technology

[0002] Unmanned underwater vehicles (UUVs) are widely used in seabed resource exploration, search and rescue missions, and military reconnaissance due to their self-propulsion and self-drive capabilities. However, UUVs are vulnerable to underwater obstacles (such as reefs and fishing nets), especially in shallow waters. Existing navigation methods largely rely on model-based path planning and tracking, but traditional methods suffer from poor adaptability due to inaccurate UUV dynamic modeling.

[0003] In recent years, deep reinforcement learning (DRL) methods (such as DDPG, SAC, and TD3) have performed well in continuous control tasks, providing new ideas for the intelligent control of UUVs. However, existing DRL methods generally have the following shortcomings when applied to UUV navigation: they do not fully consider the multiple constraints of UUVs in real-world tasks, especially safety and economy; the perception module usually relies only on forward sensors, lacking an effective and rapid response mechanism to laterally approaching obstacles, resulting in collision risks; and the adaptability, robustness, and generalization ability of the algorithms still need to be improved in complex, dynamic, and unknown environments.

[0004] Therefore, there is an urgent need for an autonomous navigation method that can simultaneously ensure safe distance and economical speed, and effectively deal with lateral threats. Summary of the Invention

[0005] To address the problems existing in the prior art, this invention provides an autonomous navigation method for unmanned underwater vehicles. By using a multimodal perception and hybrid decision-making framework, it effectively avoids lateral collisions and reduces energy consumption while ensuring efficient target arrival, thus achieving intelligent, reliable, and energy-efficient autonomous navigation.

[0006] The technical solution adopted in this invention specifically includes the following steps:

[0007] S1: Environmental perception information is acquired based on the forward-facing 3D sonar and lateral depth camera of the unmanned underwater vehicle. The environmental perception information is combined with the unmanned underwater vehicle's own state information to construct complete state information.

[0008] S11: Simulate multibeam sonar and depth cameras in a simulation environment, and integrate sensor data using a robot operating system framework.

[0009] S12: Collect self-state information, which includes the unmanned underwater vehicle's position coordinates, orientation angle, linear velocity, and angular velocity in the world coordinate system.

[0010] S13: Based on the topic publishing and subscription mechanism of the robot operating system, the environmental perception information and the self-state information are combined to construct complete state information and form a multi-dimensional input vector.

[0011] S2: Establish a decision network based on the dual-delay deep deterministic policy gradient algorithm and attention mechanism, input the complete state information into the decision network, process the sequence information using gated recurrent units and attention mechanism, focus on key features and output continuous action instructions.

[0012] S21: Using the dual-delay deep deterministic policy gradient algorithm as the basic framework, it solves the problem of overestimation of action value function in continuous action space through dual evaluation network, delayed action update and target policy smoothing.

[0013] S22: Introduces an attention mechanism based on gated recurrent units to process sonar sequence data, dynamically focusing on key environmental features through attention scores.

[0014] S3: Design a composite reward function to optimize the completion, safety and energy efficiency of navigation tasks. The composite reward function includes a reward for approaching the target, a reward for reaching the target, a reward for obstacle collision, a reward for safe distance, and a reward for economical speed.

[0015] S31: Approach Reward (AT): Based on the change in distance between the current and target positions and the previous moment, a positive reward is given when the UUV approaches the target, and a negative reward is given when it moves away, encouraging the UUV to move towards the target. Reward Coefficient Setting it to 20 is close to the target reward function. As shown below: .

[0016] S32: Target Reward (RT): A sparse positive reward is triggered when the UUV enters a certain range of the target location, incentivizing the UUV to eventually reach the target. Reward Value Set to 10, distance range Set to 1 meter, reward function for reaching the target. As shown below: .

[0017] S33: Obstacle Collision Reward (OC): A negative reward is given when the UUV collides with an obstacle, severely penalizing the collision. Reward Value Set to -20, obstacle collision reward function As shown below: .

[0018] S34: Safe Distance Bonus (SD): Designed based on minimum sonar range and angular velocity, when a UUV enters the safe distance range, a penalty is applied based on the distance and angular velocity to encourage turning and obstacle avoidance. Bonus coefficient. Set to -1, safe distance Set to 4 meters, safe distance reward function As shown below: .

[0019] S35: Economic Speed ​​Bonus (EV): Encourages UUVs to operate within their optimal speed range to save energy. and As upper and lower limits of economic growth, , Reward value Set to 0.01, Economic Speed ​​Reward Function As shown below: .

[0020] S36: Combining the above five sub-rewards, the total reward function... The linear summation of the various sub-rewards is used to collaboratively optimize navigation task completion, safety, and energy efficiency, and the expression is as follows: .

[0021] S4: Introduce a lateral vision-assisted decision-making module. The decision-making module modifies the action instructions output by the decision network based on the execution logic rules to avoid the risk of lateral collisions. Establish an assisted decision-making model based on a dual-delay depth deterministic policy gradient algorithm, attention mechanism, and depth vision.

[0022] S41: Real-time acquisition of left and right distance values ​​based on a lateral depth camera.

[0023] S42: Set a lateral safety distance threshold: If the distance to the left is less than the threshold, increase the output angular velocity to make the unmanned underwater vehicle turn right; if the distance to the right is less than the threshold, decrease the output angular velocity to make the unmanned underwater vehicle turn left.

[0024] S43: The lateral vision-assisted decision-making module and the learning strategy of the decision network operate in parallel, responding instantly to lateral dynamic obstacles.

[0025] S5: The auxiliary decision-making model is trained end-to-end using experience replay and soft update techniques, with the goal of maximizing the cumulative expected value of the composite reward function. The trained auxiliary decision-making model is then saved and tested.

[0026] S51: Initialize the parameters of the decision network and evaluation network and set the target network. Use two independent evaluation networks and take the minimum value of the output as the target action value function value. S52: Use the experience replay buffer to store state transition tuples during training; S53: The evaluation network is updated based on minimizing the Bellman error, and the decision network is updated periodically using a deterministic policy gradient. S54: Update the target network parameters using a soft update method, setting the discount factor and exploring noise parameters.

[0027] Compared with the prior art, the present invention has the following beneficial effects: This invention provides an autonomous navigation method for unmanned underwater vehicles (UUVs) based on deep reinforcement learning and multimodal perception, which effectively overcomes the shortcomings of traditional methods and existing DRL methods in terms of safety, energy efficiency, and reliability. This invention systematically optimizes navigation behavior through a multi-constraint reward function, provides safety redundancy through an auxiliary decision-making module, and improves perception efficiency through a GRU-Attention mechanism, ultimately achieving safe, energy-efficient, and reliable autonomous navigation for UUVs in complex and unknown underwater environments. The proposed algorithm has a reasonable and efficient structure, operates fully automatically, and can provide powerful autonomous navigation support for various underwater operations. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 This is a flowchart of an autonomous navigation method for an unmanned underwater vehicle according to the present invention.

[0030] Figure 2 This is a schematic diagram of the jacket platform for the simulation training environment in this invention.

[0031] Figure 3 This is a schematic diagram of the TD3-Attention network structure in this invention.

[0032] Figure 4 This is a schematic diagram of the GRU-Attention mechanism in this invention.

[0033] Figure 5 This is a logical diagram of the lateral vision-assisted decision-making module in this invention.

[0034] Figure 6 This is the average reward curve of the present invention during the training process.

[0035] Figure 7 This is a schematic diagram of the action decision-making of TD3-Attention when encountering lateral dynamic obstacles in this invention.

[0036] Figure 8 This is a schematic diagram of the action decision-making of TD3-Attention-Depth when encountering lateral dynamic obstacles in this invention. Detailed Implementation

[0037] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0038] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0039] like Figure 1 As shown, this invention discloses an autonomous navigation method for an unmanned underwater vehicle, which mainly includes the following steps: S1: Environmental perception information is acquired based on the forward-facing 3D sonar and lateral depth camera of the unmanned underwater vehicle. The environmental perception information is combined with the unmanned underwater vehicle's own state information to construct complete state information.

[0040] S11: Build an underwater jacket platform in the Gazebo simulation environment, such as... Figure 2As shown, the simulation depicts a real underwater environment with structures and dynamic obstacles. Forward sensing utilizes multi-beam 3D sonar with the following parameters: maximum detection range of 30 meters, minimum detection range of 0.1 meters, horizontal aperture of 130°, vertical aperture of 20°, update frequency of 40Hz, and a total of 360×20 beams to acquire 3D point cloud data in front. Lateral sensing employs depth cameras on both sides to acquire real-time distance information to obstacles on the left and right sides. and The ROS (Robot Operating System) framework is used for sensor data integration and communication, and the real-time transmission of data streams is achieved through the ROS topic publish / subscribe mechanism.

[0041] S12: UUV's own status information, including the UUV's current position coordinates in the world coordinate system, orientation angle, linear velocity, and angular velocity, and also calculating the Euclidean distance between the UUV and the target point. And the angular deviation toward the target, as part of the network input.

[0042] S13: State Vector Construction: Integrate forward sonar data (30-dimensional distance vector, corresponding to the range of -65° to +65°), left and right depth camera data, and self-state information (position, velocity, angle, etc.) into a multi-dimensional state vector through ROS topics. The data is then fed into the TD3-Attention network for real-time processing.

[0043] S2: Establish a decision network based on the dual-delay deep deterministic policy gradient algorithm and attention mechanism, input the complete state information into the decision network, process the sequence information using gated recurrent units and attention mechanism, focus on key features and output continuous action instructions.

[0044] S21: As Figure 3 The diagram shown illustrates the TD3-Attention network structure of this invention. It uses the Twin DelayedDDPG (TD3) algorithm as its basic framework to address the Q-value overestimation problem in continuous action spaces. The network includes: an Actor network. : Responsible for outputting continuous motion (linear velocity) and angular velocity ); Two Critic networks , : Used to evaluate action value, taking the minimum value to suppress overestimation; Target network: through soft updates ( During stable training, the Actor network is updated every 2 steps, the Critic network is updated every step, and the target policy is smoothed to remove noise. Noise trimming .

[0045] S22: Introducing the GRU-Attention mechanism, which processes sonar sequence data using a GRU to remember historical states and capture temporal dependencies. An attention mechanism is then added on top of this, with the following structure: Figure 4 As shown, the attention score is calculated. It dynamically focuses on key obstacle features and filters out irrelevant information.

[0046] The specific calculations are as follows:

[0047]

[0048]

[0049]

[0050] in , and These represent the state input at different times, the output of the GRU network, and the output of the attention mechanism, respectively. It is the bias parameter in the linear transformation. This represents the range from 0 to n at different times. and These are learnable matrices. It's the attention score. By... With the output of the GRU network By multiplying them, we can obtain the different importance of the GRU network outputs and determine which data can be ignored.

[0051] S3: Design a composite reward function to optimize the completion, safety and energy efficiency of navigation tasks. The composite reward function includes a reward for approaching the target, a reward for reaching the target, a reward for obstacle collision, a reward for safe distance, and a reward for economical speed.

[0052] S31: Approach Reward (AT): Based on the change in distance between the current and target positions and the previous moment, a positive reward is given when the UUV approaches the target, and a negative reward is given when it moves away, encouraging the UUV to move towards the target. Reward Coefficient Setting it to 20 is close to the target reward function. As shown below: .

[0053] S32: Target Reward (RT): A sparse positive reward is triggered when the UUV enters a certain range of the target location, incentivizing the UUV to eventually reach the target. Reward Value Set to 10, distance range Set to 1 meter, reward function for reaching the target. As shown below: .

[0054] S33: Obstacle Collision Reward (OC): A negative reward is given when the UUV collides with an obstacle, severely penalizing the collision. Reward Value Set to -20, obstacle collision reward function As shown below: .

[0055] S34: Safe Distance Bonus (SD): Designed based on minimum sonar range and angular velocity, when a UUV enters the safe distance range, a penalty is applied based on the distance and angular velocity to encourage turning and obstacle avoidance. Bonus coefficient. Set to -1, safe distance Set to 4 meters, safe distance reward function As shown below: .

[0056] S35: Economic Speed ​​Bonus (EV): Encourages UUVs to operate within their optimal speed range to save energy. and As upper and lower limits of economic growth, , Reward value Set to 0.01, Economic Speed ​​Reward Function As shown below: .

[0057] S36: Combining the above five sub-rewards, the total reward function... The linear summation of the various sub-rewards is used to collaboratively optimize navigation task completion, safety, and energy efficiency, and the expression is as follows: .

[0058] S4: A lateral vision-assisted decision-making module is introduced. This module modifies the action commands output by the decision network based on execution logic rules to avoid lateral collision risks, establishing an assisted decision-making model based on a dual-delay depth deterministic policy gradient algorithm, attention mechanism, and depth vision. Figure 5 The diagram shown is a logical schematic of the lateral vision-assisted decision-making module in the invention.

[0059] S41: Real-time acquisition of left and right distances measured by the depth camera. and .

[0060] S42: Set a safe distance threshold ,like Then increase angular velocity Turn the UUV to the right; if Then decrease angular velocity Make the UUV turn to the left.

[0061] S43: This module runs in parallel with the learning strategy to ensure an immediate response to lateral dynamic obstacles, forming a hybrid decision architecture (TD3-Attention-Depth).

[0062] S5: The auxiliary decision-making model is trained end-to-end using experience replay and soft update techniques, with the goal of maximizing the cumulative expected value of the composite reward function. The trained auxiliary decision-making model is then saved and tested.

[0063] S51: Initialize the actor network and critic network , The target network is set up by using two independent Critic networks and taking their minimum value as the target Q-value to suppress overestimation of the value function. The Actor network updates less frequently than the Critic network, and the Adam optimizer is used with a learning rate of 0.0001. S52: Store transfer tuples using the experience replay buffer ( , , , (done), capacity is 1,000,000, batch size is 256.

[0064] S53: Update the critic network by minimizing the Bellman error, and update the actor network every 2 steps using the deterministic policy gradient.

[0065] S54: Target network soft update parameters Set to 0.005, discount factor Set to 0.99 to explore noise. Set to 0.2, and execute a maximum of 1000 moves per round, or until a collision or the target is reached.

[0066] pass Figure 6 As shown in the training reward curve, the method proposed in this invention converges rapidly and stably during the training process. Figure 7 and Figure 8 This demonstrates how the auxiliary decision-making module effectively corrects angular velocity to avoid collisions when lateral dynamic obstacles appear.

[0067] The quantitative comparison results of this invention and different algorithms are shown in Table 1: Table 1. Quantitative Comparison Experimental Results of Different Algorithms

[0068] Comparative experimental results show that the present invention outperforms the traditional DRL method in both success rate and path efficiency.

[0069] This invention enables unmanned underwater vehicles to achieve highly safe and efficient autonomous navigation in dynamic and unknown environments. Through the synergistic effect of a multi-constraint reward mechanism and vision-assisted decision-making, obstacle avoidance capabilities and energy utilization are significantly enhanced, demonstrating excellent adaptability and engineering practical value.

[0070] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An autonomous navigation method for an unmanned underwater vehicle, characterized in that, Includes the following steps: Environmental perception information is acquired based on the forward-facing 3D sonar and the lateral depth camera of the unmanned underwater vehicle. The environmental perception information is combined with the unmanned underwater vehicle's own state information to construct complete state information. A decision network is established based on a dual-delay deep deterministic policy gradient algorithm and an attention mechanism. The complete state information is input into the decision network, and the sequence information is processed by a gated recurrent unit and an attention mechanism. Key features are focused and continuous action instructions are output. The design incorporates a composite reward function to optimize the completion rate, safety, and energy efficiency of navigation tasks. The composite reward function includes a reward for approaching the target, a reward for reaching the target, a reward for obstacle collision, a reward for safe distance, and a reward for economical speed. A lateral vision-assisted decision-making module is introduced. The decision-making module modifies the action instructions output by the decision network based on the execution logic rules to avoid the risk of lateral collision. An assisted decision-making model based on the dual-delay depth deterministic policy gradient algorithm, attention mechanism and depth vision is established. The auxiliary decision-making model is trained end-to-end using experience replay and soft update techniques, with the goal of maximizing the cumulative expected value of the composite reward function. The trained auxiliary decision-making model is then saved and tested.

2. The autonomous navigation method for an unmanned underwater vehicle according to claim 1, characterized in that, The steps for constructing complete state information include: Simulations of multibeam sonar and depth cameras are performed in a simulation environment, and sensor data is integrated using a robot operating system framework. Collect its own status information, which includes the unmanned underwater vehicle's position coordinates, orientation angle, linear velocity, and angular velocity in the world coordinate system; Based on the topic publishing and subscription mechanism of the robot operating system, the environmental perception information and the self-state information are combined to construct complete state information and form a multi-dimensional input vector.

3. The autonomous navigation method for an unmanned underwater vehicle according to claim 1, characterized in that, The steps for establishing the decision-making network include: Using a dual-delay deep deterministic policy gradient algorithm as the basic framework, this algorithm solves the problem of overestimation of action value function in continuous action space through dual evaluation network, delayed action update and target policy smoothing. An attention mechanism based on gated recurrent units is introduced to process sonar sequence data, and key environmental features are dynamically focused through attention scores.

4. The autonomous navigation method for an unmanned underwater vehicle according to claim 1, characterized in that, The composite reward function includes a reward for approaching the target, a reward for reaching the target, a reward for obstacle collision, a reward for safe distance, and a reward for economic speed. Approaching target reward: Calculated based on the difference between the current distance to the target point and the previous distance to the target point. A positive reward is given when the distance decreases and a negative reward is given when the distance increases. Reward for reaching the target: When the distance between the unmanned underwater vehicle and the target point is less than a set threshold, a one-time positive reward is triggered; Obstacle collision reward: When the unmanned underwater vehicle is detected to have collided with an obstacle, a one-time negative reward is given; Safe distance reward: When the minimum distance between the unmanned underwater vehicle and an obstacle is less than the safe distance threshold, a negative reward is calculated based on the minimum distance and the current angular velocity to encourage obstacle avoidance behavior; Economic speed bonus: When the linear velocity of the unmanned underwater vehicle is within the preset economic speed range, a positive bonus is given.

5. The autonomous navigation method for an unmanned underwater vehicle according to claim 1, characterized in that, The execution logic of the lateral vision-assisted decision-making module is as follows: Real-time acquisition of left and right distance values ​​based on lateral depth cameras; Set a lateral safety distance threshold: if the distance to the left is less than the threshold, increase the output angular velocity to make the unmanned underwater vehicle turn right; if the distance to the right is less than the threshold, decrease the output angular velocity to make the unmanned underwater vehicle turn left. The lateral vision-assisted decision-making module operates in parallel with the learning strategy of the decision network, responding instantly to lateral dynamic obstacles.

6. The autonomous navigation method for an unmanned underwater vehicle according to claim 1, characterized in that, The steps for end-to-end training of the auxiliary decision-making model using experience replay and soft update techniques include: Initialize the parameters of the decision network and evaluation network and set the target network. Use two independent evaluation networks and take the minimum value of the output as the target action value function value. Use the experience replay buffer to store state transition tuples during training; The evaluation network is updated based on minimizing the Bellman error, and the decision network is updated periodically using a deterministic policy gradient. The target network parameters are updated using a soft update method, and discount factors and exploration noise parameters are set.

7. The autonomous navigation method for an unmanned underwater vehicle according to claim 1, characterized in that, The continuous action commands output by the decision network include linear velocity and angular velocity; the lateral vision-assisted decision module only performs rule-based real-time correction on the angular velocity, while the linear velocity is directly output and executed by the decision network.

Images