UUV autonomous collision avoidance and navigation method and system based on improved ICM-DDQN

Abstract

The invention discloses a UUV autonomous collision avoidance and navigation method and system based on an improved ICM-DDQN, and belongs to the technical field of UUV path planning. The method is provided for improving the exploration capability and the optimal navigation collision avoidance strategy learning capability of an autonomous collision avoidance algorithm in a map-free and sparse award environment. According to the technical key points of the method, a state vector formed by observation data preprocessed by a forward-looking sonar and position information of a UUV relative to a target is input to an improved ICM-DDQN, the improved ICM-DDQN guides an under-actuated UUV to avoid collision according to environment information fed back by the forward-looking sonar and the position information of the target relative to the UUV, and a forward-looking sonar sensing model is constructed. The forward-looking sonar sensing model has a 120-degree open-angle vision field, the maximum sensing radius L of the forward-looking sonar sensing model is determined, and the number of integrated adjacent beams is determined according to the number of beams of the open-angle vision field; designing a network structure of the improved ICM-DDQN method; the control quantities u and omega t output by the improved ICM-DDQN are input into a motion model of the UUV, the position and posture of the UUV at the next moment and the state information of the underwater environment are obtained, the UUV learns a collision avoidance strategy through the improved ICM-DDQN, and the learned collision avoidance strategy is tested in a similar task environment.

Description

Technical Field

[0001] The present invention relates to an autonomous collision avoidance and navigation method for an unmanned underwater vehicle (UUV), and belongs to the technical field of UUV path planning. Background Art

[0002] In recent years, due to the wide application of unmanned underwater vehicles (UUVs) in marine scientific research, marine exploration, and marine engineering operations, it has become a research hotspot in the field of underwater robots. Autonomous navigation and obstacle avoidance are the core technologies to achieve the autonomy of UUVs, which will determine the application prospects of UUVs. The autonomy of UUVs is the key to their operation in complex environments, which has attracted extensive attention from researchers around the world. In a dynamic and uncertain underwater environment, unknown static and dynamic obstacles will pose huge safety problems to UUVs. Therefore, one of the main requirements for UUVs is to make autonomous decisions to avoid obstacles and find safe paths, which is of great significance for the autonomy and safety of UUVs. The marine environment in which UUVs operate is either static or dynamic. In a static environment, global information such as terrain, obstacles, and interference is known, and offline path planning can be carried out in advance. However, for a dynamic environment, the global environmental information is unknown, and UUVs need to perform real-time path planning based on sensor information. Real-time path planning in a dynamic marine environment is of practical significance and difficulty.

[0003] Traditional collision avoidance algorithms have some non-negligible limitations, such as being prone to falling into local minima, having a long planning time, poor real-time performance, high randomness, and slow optimization. Due to the limitations of traditional methods, the academic community has conducted in-depth research on mapless navigation and obstacle avoidance based on deep reinforcement learning and achieved many results. Introducing deep reinforcement learning into UUV collision avoidance enables it to gradually adapt to the environment through autonomous learning without knowing the complete prior knowledge or without any prior knowledge at all. The application of deep reinforcement learning in UUV autonomous collision avoidance has been confirmed in existing research. Compared with value-based and model-free methods, policy gradient-based methods have great advantages in dealing with continuous actions. However, there are problems of slow convergence speed and low training efficiency. Currently, most methods focus on improving the structure of the reinforcement learning algorithm, and the reward values are mostly sparse. However, designing a reasonable reward function can further improve the effect of reinforcement learning. The main research includes experience replay mechanism, exploration and exploitation, and multi-objective learning, etc. Curiosity-driven learning further studies the key factors that determine its performance. Predicting future states in a high-dimensional raw observation space is a challenging problem. Learning dynamics in an auxiliary feature space will produce improved results. However, how to select such a feature encoding space is a key and open problem. Selecting a good feature space can make the prediction task easy to handle and filter out irrelevant aspects in the environmental space.

[0004] Document number CN115906928A discloses a Transformer UUV three-dimensional autonomous collision avoidance planning method based on dual-channel self-attention. It designs a dual-channel self-attention model to parallelly capture the observation features of dual-modal multi-sensors; constructs a network model based on Transformer to execute end-to-end UUV three-dimensional autonomous collision avoidance decisions; and uses the structures of the encoder and decoder to achieve UUV collision avoidance planning based on historical observations and sequential decisions. It can perform end-to-end UUV autonomous collision avoidance planning based on dual-modal multi-sensor observations, which can greatly improve the rapidity of UUV collision avoidance planning, get rid of the dependence of UUV collision avoidance planning on high-precision and stable observations of sensors, and solve the problem of UUV autonomous collision avoidance planning in case of observation failure or target loss. However, this method is based on deep learning. An essential link is to construct a dataset. The dataset is sampled from other collision avoidance algorithms. If the dataset source is an intelligent search collision avoidance algorithm, for the same observation, there may be actions with large differences. At this time, it is more difficult for the neural network to learn. Even if the neural network can learn well, its final effect still cannot exceed the most primitive collision avoidance algorithm. At the same time, this method does not mention how to improve the exploration ability and the ability to learn the optimal navigation and collision avoidance strategy in the mapless and reward-sparse environment for UUV autonomous collision avoidance and navigation methods.

[0005] The document number CN115906928A discloses a Transformer UUV three-dimensional autonomous collision avoidance planning method based on dual-channel self-attention, which is a deep learning collision avoidance algorithm. However, an essential step in this method is to construct a dataset, and the dataset basically comes from other collision avoidance algorithms. If the dataset comes from a collision avoidance algorithm based on intelligent search, for the same observation, there may be significantly different actions. At this time, it is more difficult for the neural network to learn. Even if the neural network can learn well, its final effect still cannot exceed the most original collision avoidance algorithm, which is insurmountable. Summary of the Invention

[0006] The technical problem to be solved by the present invention is:

[0007] The purpose of the present invention is to propose a UUV autonomous collision avoidance and navigation method and system based on improved ICM-DDQN (Intrinsic Curiosity Module Double Deep Q network, hereinafter referred to as ICM-DDQN) to improve the exploration ability of the autonomous collision avoidance algorithm and the ability to learn the optimal navigation collision avoidance strategy in an environment without a map and with sparse rewards.

[0008] The technical solution adopted by the present invention to solve the above technical problem is:

[0009] A UUV autonomous collision avoidance and navigation method based on improved ICM-DDQN. The method inputs the state vector composed of the observation data preprocessed by the forward sonar and the relative position information of the UUV to the target into the improved ICM-DDQN. The improved ICM-DDQN (deep neural network) relies on the environmental information fed back by the forward sonar and the relative position information of the target to the UUV to guide the underactuated UUV to avoid collisions. The implementation process is as follows:

[0010] Step 1: Construct a forward sonar perception model:

[0011] The forward sonar perception model has an opening angle field of view of 120 degrees. Determine the maximum perception radius L of the sonar perception model, and determine the number of adjacent beams to be integrated according to the number of beams in the opening angle field of view.

[0012] Based on d t There are still sonar beams in which obstacles are not detected. To make the forward sonar detection information sparse, it is further processed to obtain (Compared with d t ), contains a large number of smaller values, reducing the computational burden of the collision avoidance algorithm) Use the coordinate difference Δs between the current UUV position coordinate and the target to describe the position information of the UUV and the target; a complete input sample X t contains the preprocessed observation data and the relative target position information Δs of the UUV;

[0013] Step 2: Design the network structure of the improved ICM-DDQN method: The conversion relationship of the constructed improved ICM-DDQN network structure is represented by ω t = f(x t , p t ), where x t represents the environmental information observed from the sonar sensor at time t, and p t represents the position information related to the UUV and the target; x t and p t are regarded as the immediate state information of the UUV; The improved ICM_DDQN network directly maps the state to the action, that is, the yaw angular velocity ω t adopted at time t; For the underactuated UUV autonomous collision avoidance situation, the control inputs are the constant cruising speed u and the yaw angular velocity ω t ;

[0014] Step 3: Input the control quantities u and ω t output by the improved ICM-DDQN network into the motion model of the UUV to obtain the position and attitude of the UUV at the next moment and the state information of the underwater environment;

[0015] Step 4: After the UUV executes the above control commands, observe the state of the UUV, the environmental state, and the magnitude of the environmental feedback reward value. If the UUV collides or exceeds the maximum time step, then initialize the state of the UUV and the environmental state; When the UUV reaches the target, give a positive reward value and initialize the state of the UUV and the environment; Repeat the above process until the UUV completes learning the collision avoidance strategy using the improved ICM-DDQN,

[0016] Test the learned collision avoidance strategy in a similar task environment, evaluate the generalization and robustness of the improved algorithm. If the requirements are met, the training of the improved ICM-DDQN is completed; Otherwise, jump to Step 1 to fine-tune the forward-looking sonar perception model and change the input samples until the generalization and robustness meet the requirements.

[0017] Furthermore, in Step 1, the forward-looking sonar perception model has 60 beams with a width of 2°, L is taken as 120 meters, and the forward-looking sonar sampling data is where is the distance measured by the i-th beam at time t; (Considering that the distance information of the obstacles detected by adjacent beams of the forward-looking sonar is highly correlated) Integrate the information of three adjacent beams in the forward-looking sonar perception model, and take the smaller distance information as the detection result. The simplified distance vector is A complete input sample X t contains and Δs, with a total of 23-dimensional input variables.

[0018] Furthermore, in step two, the improved network structure of the ICM-DDQN method is composed of an intrinsic curiosity network, a GRU network, and a DDQN network. Specifically: after passing through a fully connected network with 256 nodes in 2 layers and the activation function ReLu, the output layer is the value corresponding to 11 discrete actions a t of Q ; in the curiosity module, s t and s t+1 pass through a GRU network, a fully connected network with 256 nodes and the activation function ReLu, and finally, through a linear output layer, the states are respectively encoded as φ(s t ) and φ(s t+1 ); then the encoded output features are input into the inverse model of ICM, which includes a fully connected network layer with 256 nodes in 2 layers and the activation function ReLu, and finally, the output layer predicts the action through softmax The ICM forward model receives the feature φ(s t ) and the action a t represented by one-hot encoding, inputs them into a fully connected layer with 256 nodes in 2 layers and the activation function ReLu, and finally, through a linear output layer, predicts

[0019] Furthermore, in the curiosity network, s t and s t+1 pass through a GRU network, a fully connected network with 256 nodes and the activation function ReLu, and finally, through a linear output layer, the states are respectively encoded as φ(s t ) and φ(s t+1 ); then the encoded output features are input into the inverse model of ICM, which includes a fully connected network layer with 256 nodes in 2 layers and the activation function ReLu, and finally, the output layer predicts the action through softmax The ICM forward model receives the feature φ(s t ) and the action a t represented by one-hot encoding, inputs them into a fully connected layer with 256 nodes and the activation function ReLu; finally, through a linear output layer, it predicts where, φ(s t+1 ) and the intrinsic reward R formed by the prediction error i is defined as The intrinsic reward encourages the agent to visit new states in the environment, guides the agent to get rid of the premature convergence of local minima or sub-optimal strategies, and enables the agent to better utilize the learned environmental dynamics to complete tasks.

[0020] Furthermore, in Step 2, in the improved ICM-DDQN network (improved ICM-DDQN algorithm), the external reward function is used to measure the quality of actions. The definition of the reward function directly affects the results of DRL. When the action taken can assist the UUV to reach the target, a positive reward is obtained. When the UUV collides or moves away from the target, a negative reward is received. A dynamic composite reward is introduced to ensure the stability and convergence of the improved ICM-DDQN algorithm during training in a complex dynamic environment.

[0021] Furthermore, the designed function includes five dynamic rewards, and the reward function can be expressed as R t =R 1 +R 2 +R 3 +R 4 +R 5 ; R 1 =c 1 (d t -d t-1 ),R 1 represents the distance reward function, d t-1 and d t represent the distances from the current position to the target point at time t - 1 and time t respectively; c 1 is the weight coefficient; if (d t-1 -d t ) is positive, it means the UUV is approaching the target and the reward value will increase; on the contrary, the smaller (d t-1 -d t ), the smaller the reward value; R 2 =c 2 n t represents the current cumulative step reward function, n t is the current cumulative step, one of the important indicators for evaluating the quality of the collision avoidance algorithm, indicating the cost of UUV collision avoidance in one episode; c 2 is a negative weight coefficient, and the larger n t , the smaller the reward value; R 3 =c 3 (d ro -d) represents the distance reward function between the obstacle and the UUV, c 3 is the weight coefficient, d ro is the distance between the current UUV and the nearest obstacle, d represents the minimum safety distance, and the closer the obstacle is to the UUV, the smaller the reward value; R 4 =10 represents the reward for reaching the target point, and the UUV can obtain a relatively large positive value when reaching the target point; R 5 =-10 represents the UUV collision reward, which is set to a fixed value.

[0022] Furthermore, the GRU network structure includes a reset gate and an update gate. The reset gate determines how to combine new input information with memory information to capture short-term dependencies in the time series, and the update gate is used to control the amount of data from the previous moment's state information retained in the current state.

[0023] A UUV autonomous collision avoidance and navigation system based on improved ICM-DDQN, which has program modules corresponding to the steps of the above technical solution, and executes the steps in the UUV autonomous collision avoidance and navigation method based on improved ICM-DDQN when running.

[0024] A computer-readable storage medium stores a computer program configured to implement the steps of the UUV autonomous collision avoidance and navigation method based on improved ICM-DDQN when called by a processor.

[0025] The present invention has the following beneficial technical effects:

[0026] The present invention establishes a motion model and a forward sonar perception model of the UUV, and designs a network structure, a dynamic composite reward function, and a simulation environment for UUV collision avoidance of the improved ICM-DDQN method. The present invention is equivalent to designing an end-to-end path planner with discrete action output. To improve the exploration ability and the ability to learn the optimal navigation and collision avoidance strategy of the autonomous collision avoidance algorithm in an environment without a map and with sparse rewards, the internal reward of the Intrinsic Curiosity Module (ICM) is introduced as a reward enhancement signal. The Gated Recurrent Units (GRUs) are used to capture the correlation of long-distance features and extract dynamic features, so that the prediction error in the feature space of the intrinsic curiosity module provides a good intrinsic reward signal, and the generalization and exploration of the present invention are better. In addition, to improve the stability and efficiency of the ICM-DDQN network training, a dynamic composite reward is designed to guide the UUV to reach the target.

[0027] The present invention combines DDQN and ICM to construct a new ICM_DDQN network architecture, improves the training stability and efficiency, constructs a new reward function, and guides the UUV to reach the target quickly. The state information of deep reinforcement learning represents the environmental information perceived by the agent and the change of its own state caused by action decisions. The state information is the basis for the agent to make decisions and evaluate long-term benefits, and the quality of the state design and the ability to extract effective state features directly determine whether the deep reinforcement learning algorithm converges, the convergence speed, and the model performance. By designing a reasonable network structure and reward function to screen out good state information and help the network accurately establish decision relevance, a better strategy can be obtained within a controllable time.

[0028] The technical concept of the present invention is as follows: An improved ICM_DDQN obstacle avoidance algorithm is proposed for complex unknown static obstacles and unpredictable dynamic obstacle scenarios. Aiming at the problem of sparse rewards in complex environments, an intrinsic curiosity module is introduced to enhance the reward source, avoiding the situation where an unmanned underwater vehicle gets stuck due to lack of feedback signals, and there is a certain improvement in the training process of the obstacle avoidance strategy network of the unmanned underwater vehicle. A GRU network is introduced into the feature extraction layer of the intrinsic curiosity module to capture long-distance feature correlations, extract useful dynamic features, and make more full use of state information. Considering the stability and efficiency of the ICM_DDQN algorithm training, a composite dynamic reward is designed. The key points of the present invention are as follows: improving the ICM-DDQN network design, UUV state space and action space design, improving the reward function design of the ICM-DDQN method, and correcting the UUV motion model and preprocessing the forward sonar data. Description of the Drawings

[0029] Figure 1 It is a schematic diagram of the principle of the UUV reference coordinate system (north-east coordinate system and vehicle coordinate system);

[0030] Figure 2 It is a schematic diagram of the principle of the forward sonar perception model (2D sonar model);

[0031] Figure 3 It is a framework diagram of the intrinsic curiosity network, Figure 4 It is a GRU network diagram, Figure 5 It is a DDQN network diagram;

[0032] Figure 6 It is a structural diagram of the UUV autonomous collision avoidance network based on the improved ICM-DDQN (improved ICM_DDQN architecture); Figure 7 It is a convergence curve graph of the reward function of the improved ICM-DDQN network (average return of 200 episodes);

[0033] Figure 8 It is a collision avoidance trajectory graph (corresponding to the collision avoidance trajectory graph of the improved ICM-DDQN algorithm in one environment, which is a verification of the improved method);

[0034] Figure 9 It is the first bow direction change curve of the unmanned underwater vehicle in Case 1; Figure 10 It is the first angular velocity change curve of the unmanned underwater vehicle in Case 1;

[0035] Figure 11 It is a collision avoidance trajectory graph (corresponding to the collision avoidance trajectory graph of the improved ICM-DDQN algorithm in another environment, which is a verification of the improved method);

[0036] Figure 12 It is the second bow direction change curve of the unmanned underwater vehicle in Case 2, Figure 13For the angular velocity change curve 2 of the unmanned underwater vehicle in Case 2,

[0037] Figure 14 It shows the simulation results of the ICM_DDQN algorithm for different moving obstacles in a dynamic environment;

[0038] Figure 15 For the heading change curve 1 of the unmanned underwater vehicle in Case 3, Figure 16 For the obstacle avoidance distance change curve 1 of the unmanned underwater vehicle in Case 3, Figure 17 For the heading change curve 2 of the unmanned underwater vehicle in Case 3, Figure 18 For the obstacle avoidance distance change curve 2 of the unmanned underwater vehicle in Case 3;

[0039] Figure 19 For the trajectory comparison diagram of the unmanned underwater vehicle in Case 4, Figure 20 For the comparison diagram of the heading change curves of the unmanned underwater vehicle in Case 4;

[0040] Figure 21 For the trajectory comparison diagram of the unmanned underwater vehicle in Case 5 (the collision avoidance trajectory comparison diagram of multiple different algorithms compared with the improved ICM-DDQN), Figure 22 For the comparison curve diagram of the heading change of the unmanned underwater vehicle in Case 5. Specific implementation mode

[0041] Combined with the attached Figure 1-21 The implementation of the present invention is described as follows:

[0042] For a UUV autonomous collision avoidance and navigation method based on improved ICM-DDQN of the present invention, the state vector composed of the observation data preprocessed by the forward sonar and the UUV relative target position information is input to the improved ICM-DDQN. The improved ICM-DDQN (deep neural network) relies on the environmental information fed back by the forward sonar and the target relative UUV position information to guide the underactuated UUV to avoid collisions. The implementation process is as follows:

[0043] Step 1: Construct a forward sonar perception model:

[0044] The forward sonar perception model has a 120-degree open angle field of view. Determine the maximum perception radius L of the sonar perception model, and determine the number of adjacent beams to be integrated according to the number of beams in the open angle field of view.

[0045] Based on d t There are still sonar beams that have not detected obstacles in it. To make the forward sonar detection information sparse, it is further processed to obtain (Compared with d t Compared, contains a large number of smaller values, reducing the computational burden of the collision avoidance algorithm) uses the coordinate difference Δs between the current UUV position coordinates and the target to describe the position information of the UUV and the target; a complete input sample X t contains the preprocessed observation data and the relative position information Δs of the UUV to the target;

[0046] Step 2. Design the network structure of the improved ICM-DDQN method: The improved ICM-DDQN network structure constructs the transformation relationship using ω t = f(x t , p t ). Among them, x t represents the environmental information observed from the sonar sensor at time t, and p t represents the position information related to the UUV and the target; x t and p t are regarded as the immediate state information of the UUV; the improved ICM_DDQN network directly maps the state to the action, that is, the yaw angular velocity ω t adopted at time t; for the underactuated UUV autonomous collision avoidance situation, the control inputs are the constant cruising speed u and the yaw angular velocity ω t ;

[0047] Step 3. Input the control quantities u and ω t output by the improved ICM-DDQN network into the motion model of the UUV to obtain the position and attitude of the UUV at the next moment and the state information of the underwater environment;

[0048] Step 4. After the UUV executes the above control commands, observe the state of the UUV, the environmental state, and the magnitude of the environmental feedback reward value. If the UUV collides or exceeds the maximum time step, then initialize the state of the UUV and the environmental state; when the UUV reaches the target, give a positive reward value and initialize the state of the UUV and the environment; repeat the above process until the UUV completes learning the collision avoidance strategy by the improved ICM-DDQN,

[0049] Test the learned collision avoidance strategy in a similar task environment, evaluate the generalization and robustness of the improved algorithm. If the requirements are met, the improved ICM-DDQN training is completed; otherwise, jump to Step 1 to fine-tune the forward sonar perception model and change the input sample until the generalization and robustness meet the requirements.

[0050] The present invention aims to propose a mapless autonomous navigation and collision avoidance system for the UUV. Therefore, an attempt is made to construct the following transformation relationship: ω t = f(x t , p t ). Among them, x t represents the environmental information observed from the sonar sensor at time t, and p tIndicates the position information related to the target. The above information is used as the immediate state information of the UUV. The improved ICM-DDQN network directly maps the state to the yaw angular velocity ω of the UUV at time t t .

[0051] The goal of reinforcement learning is to learn the optimal policy for sequential decision-making problems by optimizing the cumulative future reward signal. To solve sequential decision-making problems, we can learn the optimal value estimate for each action, which is defined as the expected sum of future rewards after taking that action and following the optimal policy. Under a given policy π, the true value of action a in state s is

[0052] Q π ≡ Ε π [R 1 + γR 2 + … | S 0 = s, A 0 = a]

[0053] where γ ∈ [0, is a discount factor that weighs the importance of immediate and future rewards. The optimal value Q * (s, a) = max π (s, a). The optimal policy can be easily derived from the optimal value by choosing the highest-value action in each state. The estimation of the optimal action value can be learned using Q-Learning, which is a type of temporal difference learning. However, for most problems, the state space is too large to learn all action values in all states. In this case, we can learn the parameterized value function Q(s, a; θ t ). After taking action A t in state S t and observing the immediate reward R t+1 and the next state S t+1 , the standard Q-Learning parameter update is as follows:

[0054]

[0055] where α is the step size scale and the target is defined as follows

[0056]

[0057] This update method is similar to stochastic gradient descent, updating the current value Q(S t , A t ; θ t ) towards the target value

[0058] Deep Q network (DQN) is a multi-layer neural network. For a given state st , which outputs an action value Q(s t , ·; θ t ), where θ t are network parameters. Two important components of the DQN algorithm are using a target network and using experience replay. The parameter of the target network is the same as that of the online network, except that its parameters are copied from the online network every τ steps and remain fixed at other steps, that is

[0059]

[0060] Both the target network and experience replay significantly improve the algorithm performance.

[0061] In traditional Q-Learning and DQN, the max operator uses the same value to select and evaluate an action. However, this increases the possibility of selecting overestimated values, resulting in over-optimistic value estimates. As a potential solution to the overestimated value problem, the idea of Double Q-Learning is to reduce overestimation by decomposing the maximization operation in the target into action selection and action evaluation. The greedy policy is evaluated according to the online network, but its value is estimated using the target network. According to Double Q-Learning and DQN, in Double DQN, its target is expressed as

[0062]

[0063] where the online network weights θ t , and the target network weights The update of the target network remains the same as that of DQN, and the online policy network is copied regularly.

[0064] In the intrinsic curiosity module, the inverse dynamics task is to predict the action t given the current state s t+1 and the next state s First, the neural network encodes s t and s t+1 to learn features. Selecting a good feature space can make the prediction task easier to handle. The features should contain important information and filter out irrelevant aspects in the observation space. The sources affecting the agent's observation can be divided into: 1) those that the agent can control; 2) those that the agent cannot control but can affect the agent. A good feature space encoding should consider the above effects. Therefore, the present invention introduces a GRU network to encode the environmental information perceived by the intelligence. Compared with LSTM, the GRU network has fewer parameters, and the GRU network structure has the advantages of extracting dynamic features and capturing long-distance feature correlations. The network structure description of GRU is asFigure 4 As shown, the GRU includes a reset gate and an update gate. The reset gate determines how to combine new input information with memory information, which helps capture short-term dependencies in the time series. The update gate is used to control the amount of previous state information retained in the current state. At time t, the hidden state h passed to the next node t The update rule is as follows:

[0065] z t = δ(W hz h t-1 + W xz x t + b z )

[0066] r t = δ(W hr h t-1 + W xr x t + b r )

[0067]

[0068]

[0069] Among them, the states of the update gating control and the reset gating control at time t are z t and r t ; and h t represent the hidden state and output of the GRU at time t, W hz , W hr , W hh , W xz , W xr , W xh represent weight matrices, and b z , b r , b h are biases.

[0070] The DDQN network module uses a fully connected neural network as an approximation. According to the defined mapping equation, it takes the processed forward sonar detection information and the relative target position information combined together as input. After passing through a fully connected network with 256 nodes in 2 layers and the activation function ReLu, it outputs the Q(s, a) value corresponding to 11 discrete actions a t .

[0071] In the curiosity network module, s t and s t+1 pass through a GRU network layer, a fully connected network layer with 256 nodes and the activation function ReLu, and finally pass through a linear output layer to encode the states as φ(st ) and φ(s t+1 ). Then, the features output from the encoding are input into the inverse model of ICM, which consists of two fully connected layer networks with 256 nodes each and a ReLu activation function. Finally, the output layer predicts the action through softmax The forward model of ICM receives the feature φ(s t ) and the action a represented by one-hot encoding t , and inputs them into two fully connected layers with 256 nodes each and a ReLu activation function. Finally, after a linear output layer, it predicts where φ(s t+1 ) and the intrinsic reward R formed by the prediction error i is defined as The intrinsic reward encourages the agent to visit new states in the environment, which is crucial for guiding the agent to escape from local minima or premature convergence of sub-optimal policies. This exploration strategy enables the agent to better utilize the learned environmental dynamics to complete tasks

[0072] In the improved ICM-DDQN algorithm, the external reward function is used to measure the quality of actions, and the definition of the reward function directly affects the results of DRL. When the action taken can assist the UUV to reach the target, a positive reward is obtained. When the UUV collides or moves away from the target, a negative reward is received. Therefore, to ensure the stability and convergence of the improved ICM-DDQN algorithm during training in a complex dynamic environment, a dynamic composite reward is introduced. The designed function includes five dynamic rewards, and the reward function can be expressed as R t = R 1 + R 2 + R 3 + R 4 + R 5 . R 1 = c 1 (d t - d t-1 ), R 1 represents the distance reward function, d t-1 and d t represent the distances from the current position to the target point at time t-1 and time t respectively. c 1 is the weight coefficient. If (d t-1 - d t ) is positive, it means the UUV is approaching the target and the reward value will increase. On the contrary, the smaller (d t-1 - d t ), the smaller the reward value. R 2 = c 2 n t represents the current cumulative step reward function, n tThe current cumulative step length, one of the important indicators for evaluating the pros and cons of the collision avoidance algorithm, represents the cost used for UUV collision avoidance in one round. c 2 is the negative weight coefficient, n t The larger it is, the smaller the reward value. R 3 = c 3 (d ro - d) represents the distance reward function between the obstacle and the UUV, c 3 is the weight coefficient, d ro is the distance between the current UUV and the nearest obstacle, d represents the minimum safety distance, and the closer the obstacle is to the UUV, the smaller the reward value. R 4 = 10 represents the reward for reaching the target point. The UUV can obtain a relatively large positive value when reaching the target point. R 5 = - 10 represents the UUV collision reward and is set to a fixed value.

[0073] Embodiment:

[0074] The present invention is an obstacle avoidance method for an underactuated unmanned underwater vehicle. Obstacle avoidance on the vertical plane is usually achieved through depth adjustment, and depth adjustment strategies often bring about large pitch adjustments, affecting the attitude control of the unmanned underwater vehicle. Therefore, a horizontal three - degree - of - freedom model of the unmanned underwater vehicle is adopted, which can not only ensure the safety of the unmanned underwater vehicle's obstacle avoidance but also facilitate the motion control of the unmanned underwater vehicle. The carrier coordinate system and the north - east coordinate system are shown as Figure 1 shown. The 3 - degree - of - freedom control model of the unmanned underwater vehicle is described as follows:

[0075]

[0076]

[0077] Define η = [x, y, ψ] T as the pose vector of the unmanned underwater vehicle, corresponding to the position and heading of the unmanned underwater vehicle in the north - east coordinate system respectively. R(ψ) is the transformation matrix from the north - east coordinate system to the carrier coordinate system. V = [u, v, r] T represents the velocity vector of the unmanned underwater vehicle in the carrier coordinate system including surge, sway, and yaw; τ = [τ u , 0, τ r T is the input of the actuator; where M = M RB + M A represents the inertia matrix of the system, C(V) = C RB (V)+C A (V) and D(V) = D l + D nl (V) are the Coriolis - centripetal matrix and the damping matrix respectively. ​

[0078] Specifically,

[0079]

[0080]

[0081]

[0082]

[0083]

[0084]

[0085]

[0086] The kinematic and dynamic equations of the unmanned underwater vehicle can be described as follows:

[0087]

[0088]

[0089]

[0090]

[0091]

[0092]

[0093] Among them, the assumed constant ocean current in the carrier reference frame is represented by the vector [u c , v c T , A = -d 22 v r +(d 23 -u r c 23 -mu c )r, B = (d 32 -u r c 32 )v r -d 33 r + τ r

[0094]

[0095]

[0096]

[0097] ​

[0098]

[0099]

[0100]

[0101]

[0102] d 23 = Y r

[0103]

[0104] d 33 = N r + N |r|r |r|

[0105] For the problem of autonomous obstacle avoidance of an underactuated unmanned underwater vehicle, the control inputs of this paper are respectively the surge velocity u and the yaw angular velocity r, and the kinematics of this unmanned underwater vehicle should satisfy the following constraint conditions:

[0106] u = 2.4 kn

[0107] -10° / s ≤ r ≤ 10° / s (9)

[0108] The input information of the obstacle avoidance algorithm proposed in this invention is measured by a multibeam sonar. The deep neural network adopted in the obstacle avoidance algorithm has good learning ability and feature extraction ability. The deep neural network can rely on the environmental information feedback by the multibeam forward-looking sonar and the position information of the target relative to the unmanned underwater vehicle to guide the unmanned underwater vehicle to avoid obstacles. The simulation sonar model is as Figure 2 shown, which has 60 beams with a width of 2°, and the sonar sampling data is where is the distance measured by the i-th beam at time t. Considering that the beam angle of the adopted simulation sonar model is 2° and the maximum detection distance is 120 m, the distance information of the obstacles detected by its adjacent beams has a high degree of correlation. In order to reduce redundant information, the information of three adjacent beams in the sonar model is integrated, and the smaller distance information is taken as the detection result. The simplified distance vector is

[0109] Since there are beams in d t that do not detect obstacles, in order to make the sonar detection information sparse, the following processing is performed on it

[0110]

[0111] and dt In contrast, contains a large number of smaller values, reducing the computational burden of the obstacle avoidance algorithm. The position information between the unmanned underwater vehicle and the target is described by the coordinate difference Δs between the current position coordinates and the target coordinates. Therefore, a complete input sample X t contains and Δs, with a total of 23 input variables.

[0112] The goal of reinforcement learning is to learn the optimal policy for sequential decision-making problems by optimizing the cumulative future reward signal. To solve sequential decision-making problems, the optimal value estimate for each action can be learned, which is defined as the expected sum of future rewards after taking that action and following the optimal policy. Under a given policy π, the true value of action a in state s is

[0113] Qπ≡Επ[R 1 +γR 2 +…|S 0 = s,A 0 = a] (10)

[0114] where γ ∈ [0,1] is a discount factor that weighs the importance of immediate and future rewards. The optimal value Q * (s,a) = max π (s,a). The optimal policy can be easily derived from the optimal value by choosing the highest-value action in each state. The estimation of the optimal action value can be learned using Q-Learning, which is a type of temporal difference learning. However, for most interesting problems, the state space is too large to learn all action values in all states. In this case, we can learn the parameterized value function Q(s,a;θ t ). After taking action A t in state S t and observing the immediate reward R t+1 and the next state S t+1 , the standard Q-Learning parameter update is

[0115]

[0116] where α is the step size scale and the target is defined as follows

[0117]

[0118] This update method is similar to stochastic gradient descent, updating the current value Q(S t ,A t ;θ t ) towards the target value

[0119] The Deep Q Network (DQN) is a multi-layer neural network that outputs an action value Q(s t , ·; θ t ) for a given state s t , where θ t are the network parameters. Two important components of the DQN algorithm are the use of a target network and experience replay. The target network with parameters is the same as the online network, except that its parameters are copied from the online network every τ steps and remain fixed at other steps, i.e.,

[0120]

[0121] Both the target network and experience replay significantly improve the algorithm performance.

[0122] In traditional Q-Learning and DQN, the max operator uses the same value to select and evaluate an action. However, this increases the likelihood of selecting overestimated values, leading to overly optimistic value estimates. As a potential solution to the overestimated value problem, the idea of Double Q-Learning is to reduce overestimation by decomposing the maximization operation in the target into action selection and action evaluation. The greedy policy is evaluated according to the online network, but its value is estimated using the target network. Based on Double Q-Learning and DQN, Hasselt et al. proposed a new algorithm called Double DQN. In Double DQN, its target is expressed as

[0123]

[0124] where the online network weights are θ t , and the target network weights For the update of the target network, it remains the same as DQN, and the online policy network is copied periodically.

[0125] In the case of normal external reward R e , the present invention uses intrinsic motivation measured by curiosity, and rewards new states to encourage exploration, thereby improving the learning ability of the agent in more environments that require guided exploration. As Figure 3 shown, the intrinsic curiosity module includes a forward model and a reverse model, which predict states and actions respectively. The goal of this module is to provide a mechanism for learning feature representations, so that the prediction error in the learned feature space provides a good intrinsic reward signal. The ICM encodes the states s t , s t+1 into feature vectors φ(s t ), φ(s t+1 ) for predicting at (i.e. inverse dynamics model). The forward model converts φ(s t ) and a t As input prediction s t+1 The feature representation The prediction error in feature space is used as a curiosity-based intrinsic reward signal. t+1 )and Intrinsic reward R formed by prediction error i Defined as

[0126]

[0127] Intrinsic rewards encourage the agent to visit new states in the environment, which is crucial for guiding the agent out of local minima or premature convergence to suboptimal policies. This exploration strategy enables the agent to better utilize the learned environmental dynamics to complete the task.

[0128] In the improved ICM-DDQN algorithm, the external reward function is used to measure the quality of the action. The quality of the reward function definition directly affects the result of DRL. When the action taken can help the UUV reach the target, a positive reward is obtained. When the UUV collides or moves away from the target, a negative reward is obtained. Therefore, in order to ensure the stability and convergence of the improved ICM-DDQN algorithm in complex dynamic environments, a dynamic compound reward is introduced. The designed function includes five dynamic rewards, and the reward function can be expressed as:

[0129] R t =R 1 +R 2 +R 3 +R 4 +R 5 (16)

[0130] in,

[0131] R 1 =c 1 (d t -d t-1 ) (17)

[0132] R 1 represents the distance reward function, d t-1 and d t Respectively represent the distance from the current position to the target point at time t-1 and time t. 1 is the weight coefficient. If (d t-1 -d t ) is a positive number, which means that the reward value will increase as the unmanned underwater vehicle approaches the target. On the contrary, (d t-1 -d t ) is smaller, the smaller the reward value is.

[0133] R 2 = c 2 n t (18)

[0134] R 2 represents the current cumulative step reward function, n t the current cumulative step, one of the important indicators for evaluating the quality of the obstacle avoidance algorithm, representing the cost used for obstacle avoidance by an underwater vehicle in a round. c 2 is the negative weight coefficient, n t The larger it is, the smaller the reward value.

[0135] R 3 = c 3 (d ro - d) (19)

[0136] R 3 represents the distance reward function between the obstacle and the underwater vehicle, c 3 is the weight coefficient, d ro is the distance between the current underwater vehicle and the nearest obstacle, d represents the minimum safety distance, the closer the obstacle is to the underwater vehicle, the smaller the reward value.

[0137] R 4 = 10 (20)

[0138] R 4 represents the reward for reaching the target point. The underwater vehicle can obtain a relatively large positive value when reaching the target point.

[0139] R 5 = - 10 (21)

[0140] R 5 represents the collision reward of the underwater vehicle, which is set to a fixed value.

[0141] In the intrinsic curiosity module, the inverse dynamics task is to predict the action by predicting the given current state s t and the next state s t+1 to predict the action First, use a neural network to encode s t and s t+1Learn features. Selecting a good feature space can make the prediction task easier to handle. The features should contain important information and filter out irrelevant aspects in the observation space. The sources affecting the agent's observations can be divided into: 1) those that the agent can control; 2) those that the agent cannot control but can affect the agent. A good feature space encoding should consider the above influences. Therefore, this paper introduces a GRU network to encode the environmental information of intelligent perception. Compared with LSTM, the GRU network has fewer parameters, and the GRU network structure has the advantages of extracting dynamic features and capturing long-distance feature correlations. The network structure of GRU is described as Figure 4 shown. GRU includes a reset gate and an update gate. The reset gate determines how to combine the new input information with the memory information, which helps to capture short-term dependencies in the time series. The update gate is used to control the amount of data from the previous moment's state information retained in the current state. At time t, the hidden state h passed to the next node t The update rule is

[0142] z t = δ(W hz h t-1 + W xz x t + b z )

[0143] r t = δ(W hr h t-1 + W xr x t + b r )

[0144]

[0145]

[0146] where the states of the update gating control and the reset gating control at time t are z t and r t . and h t represent the hidden state and output of GRU at time t. W hz , W hr , W hh , W xz , W xr , W xh represent weight matrices. b z , b r , b h are biases.

[0147] The improved ICM-DDQN network structure (Structure of ICM_DDQN) is as Figure 5As shown, it combines Double DQN and ICM. The present invention aims to propose a mapless autonomous navigation and obstacle avoidance system for UUVs, so an attempt is made to construct the following transformation relationship:

[0148] ω t = f(x t , p t ) (23)

[0149] Where x t represents the environmental information observed from the sonar sensor at time t, and p t represents the position information related to the target. They are regarded as the immediate state information of the unmanned underwater vehicle. The improved ICM_DDQN network directly maps the state to the action, that is, the yaw angular velocity ω t at time t. The DDQN algorithm uses a fully connected neural network as shown in Figure 6 as an approximation. According to the defined mapping equation, it takes the processed sonar detection information and the position information relative to the target combined together as the input, and the action value Q(s,a) as the output. As shown in Figure 6 , after passing through a fully connected network with 256 nodes in 2 layers and the activation function ReLu, the output layer is the t value corresponding to 11 discrete actions a Q . In the curiosity module, s t and s t+1 pass through a GRU network, a fully connected network with 256 nodes and the activation function ReLu, and finally pass through a linear output layer to encode the states into φ(s t ) and φ(s t+1 ) respectively. Then the encoded output features are input into the inverse model of ICM, which contains a fully connected layer network with 256 nodes in 2 layers and the activation function ReLu, and finally the output layer predicts the action through softmax The ICM forward model receives the feature φ(s t ) and the action a t represented by one-hot encoding, and inputs them into a fully connected layer with 256 nodes in 2 layers and the activation function ReLu. Finally, after passing through a linear output layer, it predicts

[0150] Algorithm 1 describes the pseudocode of the improved ICM-DDQN algorithm. The proposed obstacle avoidance algorithm not only considers the obstacle avoidance decisions for unknown static obstacles and unpredictable dynamic obstacles, but also considers the sparse reward. A reasonable external reward is designed to ensure the training stability and efficiency of the improved ICM-DDQN network, and a GRU is introduced inside the curiosity module to capture the long-distance spatial feature correlation and forget unimportant historical information.

[0151] Algorithm 1:

[0152]

[0153]

[0154] Verification of the effects of the present invention:

[0155] The improved ICM-DDQN algorithm is compared with a series of obstacle avoidance algorithms such as DQN, DDQN, RRT, and PRM in terms of path length, action decision time, and path smoothness. In this experiment, a total of 200 rounds of training are carried out, and the maximum step size for each round is 1200. The performance of the obstacle avoidance algorithm is verified by setting 3 different test scenarios. The size of the simulated map is 600×600 and 800×600, and all obstacle information in the environment is unknown. Obstacles are randomly distributed between the starting point and the target point. The cruising speed of the unmanned underwater vehicle is 1.2 m / s, the maximum angular rate is 10 o / s, and the minimum safety distance is 20 m. The hyperparameters of the algorithm proposed in this invention during training are shown in Table 1. The average return curve of ICM-DDQN during the improvement process is as Figure 7 shown.

[0156] Table 1

[0157]

[0158] Case 1: (Scenario with multiple irregular static obstacles) Figure 8 shows the entire trajectory of the unmanned underwater vehicle from the starting point (60, 550) to the target (480, 60). The unmanned underwater vehicle adopts the learned optimal navigation and obstacle avoidance strategy and moves towards the target point by interacting with the environment. When the unmanned underwater vehicle observes an irregular obstacle ahead, it adjusts the angular velocity to avoid collision and crosses the obstacle area to find a feasible path. According to Figure 9 and Figure 10 analysis, it can be concluded that the unmanned underwater vehicle can plan a safe and collision-free trajectory under the kinematic constraint conditions.

[0159] Case 2: (Maze scenario with multiple irregular static obstacles) The starting position of the unmanned underwater vehicle is (100, 300), and the target position is (580, 435). As Figure 11 shown, when the unmanned underwater vehicle uses the learned strategy to interact with the environment and detects an obstacle, it will adopt an appropriate angular rate and maintain a certain safety distance from the obstacle. By increasing the complexity of the test environment, it is shown that introducing ICM increases the exploratory nature of the unmanned underwater vehicle's obstacle avoidance strategy. According to Figure 12 and Figure 13Analysis shows that under the condition of meeting the motion constraints, the unmanned underwater vehicle can plan a safe collision-free path in an environment with sparse rewards.

[0160] Case 3: (Scenario with multiple dynamic obstacles) Figure 14 Shows the simulation results of the ICM-DDQN algorithm for different moving obstacles in a dynamic environment. In this experiment, it is assumed that the dynamic obstacles move in a uniform straight line at different speeds. The starting position of the unmanned underwater vehicle is (100, 420), and the target position is (430, 80). The ICM-DDQN drives the unmanned underwater vehicle to navigate towards the target direction and takes appropriate angular velocities to avoid obstacles when a collision threat is detected. From Figure 15 , 16 , 17, 18, it can be seen that the underwater unmanned vehicle can effectively avoid dynamic obstacles when meeting the motion constraints and the minimum safety distance.

[0161] Case 4: To test the generalization ability and exploration ability of various methods, the test environment of Case 1 is adopted in this experiment. The obstacle avoidance trajectory and heading change of the unmanned underwater vehicle are respectively as Figure 19 and Figure 20 shown. The evaluation indexes of the obstacle avoidance trajectory are shown in Table 2, including the trajectory length L e , the number of steps S per episode e , and the single-step action decision time T d . It can be seen from the simulation results that all methods can meet the obstacle avoidance requirements. Among them, as global path planning methods, the planning times of RRT and PRM are 620.03 ms and 1100.93 ms respectively. Although the RRT algorithm obtains the shortest trajectory length of 649 m, the RRT and PRM algorithms have problems such as too long planning time, large randomness, and non-smooth generated paths, which require secondary smoothing processing. Compared with the local path planning algorithms DQN and DDQN, the ICM-DDQN plans the shortest path and the fewest steps. According to Figure 20 analysis, the ICM-DDQN can plan a smooth path with a small adjustment of the heading under the condition of meeting the constraints of the unmanned underwater vehicle.

[0162] Table 2

[0163]

[0164] Case 5: To further evaluate the autonomous navigation and obstacle avoidance ability of the ICM_DDQN algorithm in a complex environment. The test environment of Case 2 is adopted in this experiment. In this experiment, the evaluation indexes of the obstacle avoidance trajectory are shown in Table 3, including the trajectory length L e , the number of steps S per episode e , and the single-step action decision time T d . From Figure 21It can be seen that the trajectory planned by the RRT method has the highest collision risk, followed by the PRM method. Moreover, the paths generated by RRT and PRM are less smooth and have poor passability in narrow areas. DDQN, DQN, RRT, PRM, and ICM-DDQN can all guide the unmanned underwater vehicle to reach the target safely in this maze environment. As global path planning algorithms, RRT and PRM have the longest planning times, which are 960.83 ms and 841.86 ms respectively. Compared with the local path planning algorithms DDQN and DQN, ICM-DDQN has the shortest path and the fewest planning steps. According to Figure 22 analysis, under the condition of meeting the motion constraints, ICM-DDQN can obtain a safe and collision-free smooth path through the learned optimal strategy.

[0165] Table 3

[0166]

[0167] The present invention combines DDQN and ICM to construct a new ICM-DDQN network architecture. To improve the training stability and efficiency, a new reward function is constructed to guide the UUV to reach the target quickly. The state information of deep reinforcement learning represents the environmental information perceived by the agent and the change in its own state caused by action decisions. The state information is the basis for the agent to make decisions and evaluate long-term rewards. The quality of the state design and the ability to extract effective state features directly determine whether the deep reinforcement learning algorithm converges, the convergence speed, and the model performance. By designing a reasonable network structure and reward function to screen out good state information and help the network accurately establish decision relevance, a better strategy can be obtained within a controllable time.

Claims

1. A UUV autonomous collision avoidance and navigation method based on improved ICM-DDQN, characterized in that: The method inputs the state vector composed of the observation data preprocessed by the forward-looking sonar and the relative position information of the UUV to the target into the improved ICM-DDQN. The improved ICM-DDQN (deep neural network) guides the underactuated UUV to avoid collision based on the environmental information fed back by the forward-looking sonar and the relative position information of the target to the UUV. The implementation process is as follows: Step 1: Build a forward-looking sonar perception model: The forward-looking sonar perception model has an open-angle field of view of 120 degrees. The maximum perception radius L of the forward-looking sonar perception model is determined. The number of adjacent beams to be integrated is determined according to the number of beams in the open-angle field of view. Based on d t There are still sonar beams that have not detected obstacles. In order to make the forward-looking sonar detection information sparse, it is further processed to obtain The coordinate difference Δs between the current UUV position coordinate and the target is used to describe the position information of the UUV and the target. t Contains preprocessed observation data and the relative target position information Δs of the UUV; Step 2: Design the network structure of the improved ICM-DDQN method: The improved ICM-DDQN network structure constructs the conversion relationship with ω t =f(x t ,p t ) means, where x t represents the environmental information observed from the sonar sensor at time t, p t Indicates the location information of UUV and target; x t and p t is regarded as the real-time state information of the UUV; the improved ICM_DDQN network directly maps the state to the action, that is, the bow angular velocity ω taken at time t t ; For the autonomous collision avoidance of underactuated UUV, the control inputs are constant cruising speed u, bow angular velocity ω t ; Step 3: Improve the control quantities u and ω output by the ICM-DDQN network t Input the UUV motion model to obtain the UUV's next position and attitude as well as the underwater environment status information; Step 4: After the UUV executes the above control command, observe the UUV state, environment state, and the size of the environment feedback reward value. If the UUV collides or exceeds the maximum time step, initialize the UUV state and environment state; when the UUV reaches the target, give a positive reward value and initialize the UUV and environment state; repeat the above process until the UUV improves the ICM-DDQN learning collision avoidance strategy. The learned collision avoidance strategy is tested in a similar task environment to evaluate the generalization and robustness of the improved algorithm. If the requirements are met, the improved ICM-DDQN training is completed; otherwise, jump to step 1 to fine-tune the forward-looking sonar perception model and change the input samples until the generalization and robustness meet the requirements.

2. The UUV autonomous collision avoidance and navigation method based on improved ICM-DDQN according to claim 1 is characterized in that: In step one, The forward-looking sonar perception model has 60 beams with a width of 2°, L is 120 meters, and the forward-looking sonar sampling data is in is the distance measured by the i-th beam at time t; (considering that the obstacle distance information detected by adjacent beams of the forward-looking sonar is highly correlated) the information of the three adjacent beams in the forward-looking sonar perception model is integrated, and the smaller distance information is taken as the detection result. The simplified distance vector is A complete input sample X t Include and Δs, a total of 23 input variables.

3. The UUV autonomous collision avoidance and navigation method based on improved ICM-DDQN according to claim 1 or 2, characterized in that: In step 2, the network structure of the improved ICM-DDQN method is formed by integrating the intrinsic curiosity network, the GRU network, and the DDQN network. Specifically, after passing through the 2-layer fully connected network with 256 nodes and the activation function of ReLu in the DDQN network, the output layer corresponds to 11 discrete actions a t Q value; in the curiosity module, s t and t+1 After a layer of GRU network, a layer has 256 nodes, the activation function is a fully connected network of ReLu, and finally the state is encoded into φ(s t ) and φ(s t+1 ); then the encoded output features are input into the inverse model of ICM, which contains 2 layers with 256 nodes and a fully connected layer network with ReLu activation function. Finally, the output layer predicts the action through softmax The ICM forward model receives the feature φ(s t ) and one-hot encoding of action a t , input it into the 2-layer fully connected layer with 256 nodes and the activation function is ReLu, and finally, it is predicted by the linear output layer 4. The UUV autonomous collision avoidance and navigation method based on improved ICM-DDQN according to claim 3 is characterized in that: In the curiosity network, s t and t+1 After a layer of GRU network, the GRU network has 256 nodes, the activation function is a fully connected network of ReLu, and finally the state is encoded as φ(s t ) and φ(s t+1 ); then the encoded output features are input into the inverse model of ICM, which contains 2 layers with 256 nodes and a fully connected layer network with ReLu activation function. Finally, the output layer predicts the action through softmax The ICM forward model receives the feature φ(s t ) and one-hot encoding of action a t , input it into the 2-layer fully connected layer with 256 nodes and the activation function is ReLu; finally, it is predicted by the linear output layer Among them, φ(s t+1 )and Intrinsic reward R formed by prediction error i Defined as Intrinsic rewards encourage the agent to visit new states in the environment, guide the agent to escape from local minima or premature convergence of suboptimal strategies, and enable the agent to better utilize the learned environmental dynamics to complete tasks.

5. The UUV autonomous collision avoidance and navigation method based on improved ICM-DDQN according to claim 4 is characterized in that: In step 2, In the improved ICM-DDQN network (improved ICM-DDQN algorithm), the external reward function is used to measure the quality of the action. The quality of the reward function definition directly affects the result of DRL. When the action taken can assist the UUV to reach the target, a positive reward is obtained. When the UUV collides or moves away from the target, a negative reward is obtained. A dynamic compound reward is introduced to ensure the stability and convergence of the improved ICM-DDQN algorithm in complex dynamic environments.

6. The UUV autonomous collision avoidance and navigation method based on improved ICM-DDQN according to claim 5 is characterized in that: The designed function includes five dynamic rewards. The reward function can be expressed as R t =R1+R2+R3+R4+R5; R1=c1(d t -d t-1 ), R1 represents the distance reward function, d t-1 and d t Respectively represent the distance from the current position to the target point at time t-1 and time t; c1 is the weight coefficient; if (d t-1 -d t ) is a positive number, which means that the UUV is close to the target and the reward value will become larger; on the contrary, (d t-1 -d t ) is smaller, the smaller the reward value is; R2=c2n t Represents the current cumulative step reward function, n t The current cumulative step length is one of the important indicators for evaluating the quality of the collision avoidance algorithm, which indicates the cost of the UUV to avoid collision in one round; c2 is the negative weight coefficient, n t The larger the value, the smaller the reward value; R3 = c3 (d ro -d) represents the distance reward function between the obstacle and the UUV, c3 is the weight coefficient, d ro is the distance between the current UUV and the nearest obstacle, d represents the minimum safety distance, the closer the obstacle is to the UUV, the smaller the reward value; R4=10 represents the reward for reaching the target point, and the UUV can obtain a larger positive value when reaching the target point; R5=-10 represents the UUV collision reward, which is set to a fixed value.

7. The UUV autonomous collision avoidance and navigation method based on improved ICM-DDQN according to claim 6 is characterized in that: The GRU network structure includes a reset gate and an update gate. The reset gate determines how to combine new input information with memory information to capture short-term dependencies in time series. The update gate is used to control the amount of data retained in the current state from the previous state information.

8. A UUV autonomous collision avoidance and navigation system based on improved ICM-DDQN, characterized by: The system has a program module corresponding to the steps of any one of claims 1 to 7, and executes the steps in the UUV autonomous collision avoidance and navigation method based on improved ICM-DDQN when running.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is configured to implement the steps of the UUV autonomous collision avoidance and navigation method based on improved ICM-DDQN according to any one of claims 1 to 7 when called by a processor.

Images