Self-adaptive track control method of unmanned submersible vehicle

Abstract

The invention relates to the technical field of marine intelligent systems, in particular to a self-adaptive track control method of an unmanned submersible vehicle, which comprises the following steps of: acquiring operating parameters of the unmanned submersible vehicle in a complex marine environment in real time, adaptively extracting key features of the parameters based on a DUAE algorithm, and taking the key features as a state space of an IDDPG algorithm. And continuous Gaussian noise is introduced into the IDDPG algorithm, so that the explorability of the control strategy is effectively improved, and falling into a local optimal solution is avoided. Samples are collected from a replay buffer area in a small-batch random sampling mode, the correlation between the samples is reduced, and a target network is prevented from being over-fitted through a soft update method. An adaptive track control strategy is generated in real time based on an IDDPG algorithm, and it is ensured that the unmanned submersible vehicle completes a track tracking task with high precision. Compared with the prior art, the method has the advantages that the flight path tracking precision and the task execution efficiency of the unmanned submersible vehicle are improved, and the like.

Description

Technical Field

[0001] The present invention relates to the technical field of marine intelligent systems, and in particular to an adaptive track control method for an unmanned underwater vehicle. Background Art

[0002] With the continuous development of marine intelligent systems, unmanned submersibles are increasingly widely used in underwater monitoring, sea area monitoring, deep-sea exploration and other fields. However, how to achieve precise track control of unmanned submersibles has become the research frontier in the field of intelligent control at home and abroad. Existing track control algorithms are difficult to cope with the uncertainty of complex marine environments, especially when facing dynamic factors such as ocean disturbances and underwater obstacles, the stability and accuracy of the control algorithm are seriously challenged. A large number of time-varying navigation dynamics make it difficult for unmanned submersibles to process nonlinear data in real time, aggravating the hysteresis of the control strategy. In addition, the control strategy is insufficient and cannot respond quickly to the changing underwater environment, which greatly reduces the track control accuracy of the submersible and increases the potential risks of unmanned submersible navigation. Therefore, there is an urgent need for high-precision adaptive track control methods for unmanned submersibles to effectively improve the track tracking accuracy and mission execution efficiency of unmanned submersibles. Summary of the invention

[0003] In view of this, the purpose of the present invention is to propose an adaptive track control method for an unmanned submersible to solve the problem that the existing method has insufficient control strategy capability and cannot quickly respond to the changing underwater environment, resulting in insufficient track control accuracy.

[0004] Based on the above purpose, the present invention provides an adaptive track control method for an unmanned underwater vehicle, comprising the following steps:

[0005] S1. Transform the navigation data of the unmanned submersible into i Input into the multi-layer hidden layer of the DUAE (Deep Undercomplete Autoencoder) algorithm, and use nonlinear activation functions to adaptively extract the most important data features of the parameters;

[0006] S2, output the adaptive track control strategy of the unmanned submersible in real time through the IDDPG (Improved Deep Deterministic Policy Gradient) algorithm;

[0007] The IDDPG algorithm increases the exploratory power of the action space by introducing continuous Gaussian noise. The IDDPG algorithm includes a prediction network and a target network. The prediction network includes an Actor network and a Critic network. The Critic network is used to evaluate the action value a t The advantages and disadvantages of the current state s t and action a tCalculate the value function of the Critic network.

[0008] Preferably, the DUAE algorithm uses multiple hidden layers to analyze the complex structure and correlation of the data.

[0009] Preferably, the IDDPG algorithm is implemented by taking action μ(s t ) introduces Gaussian noise to obtain action a t , increasing the exploration of the action space.

[0010] Preferably, the Critic network of the IDDPG algorithm is used to evaluate the action value a t The pros and cons of the current moment are obtained by calculating the reward function r(s t ,a t ).

[0011] Preferably, the Actor network of the IDDPG algorithm continuously adjusts the action strategy in the direction of maximizing the value function output by the Critic network to maximize the global cumulative reward.

[0012] Preferably, the IDDPG algorithm is based on the experience replay technology, and at each moment, the information generated by the interaction between the unmanned submersible and the marine environment is stored in a replay buffer.

[0013] Preferably, the IDDPG algorithm samples the samples in the replay buffer by random sampling in small batches.

[0014] Preferably, the IDDPG algorithm adopts a soft update method, using the parameters of the prediction network Actor and Critic to update the parameters of the target network respectively to prevent the network from overfitting.

[0015] Beneficial effects of the present invention:

[0016] 1. Based on the DUAE algorithm, the present invention inputs the navigation data of the unmanned submersible into a multi-layer hidden layer, uses a nonlinear activation function to adaptively extract the most important features, analyzes the complex structure and coupling relationship between the data, and effectively removes noise and redundant information, so that the unmanned submersible can maintain accurate track tracking accuracy when facing a complex navigation environment.

[0017] 2. Based on the IDDPG algorithm, the present invention introduces continuous Gaussian noise to enable the unmanned submersible to explore different control strategies more extensively in complex navigation environments and avoid falling into local optimal solutions. According to changes in environmental conditions, its action strategy is adaptively adjusted to maximize the global cumulative reward function, effectively improving the track tracking accuracy of the unmanned submersible. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only for the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 A flow chart of an adaptive track control method for an unmanned submersible according to an embodiment of the present invention;

[0020] Figure 2 The figure is a schematic diagram of an improved flow chart of the IDDPG algorithm according to an embodiment of the present invention. DETAILED DESCRIPTION

[0021] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments.

[0022] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the present invention should be understood by people with ordinary skills in the field to which the present invention belongs. The "first", "second" and similar words used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0023] This embodiment provides an adaptive track control method for an unmanned underwater vehicle. Figure 1 and Figure 2 .

[0024] The following steps are involved:

[0025] Step 100, when facing a complex marine environment, collect the average cruising range, average navigation speed, average navigation time, average positioning accuracy, average fuel consumption, average delay time, average mission completion rate and average navigation accuracy of the unmanned submersible in real time to form the unmanned submersible track control parameter x i ={x1,x2,...,x n}, where n is the number of parameter data.

[0026] The DUAE algorithm uses multiple hidden layers to analyze the complex structure and correlation of data, improving the robustness and adaptability of the algorithm. i Input into the multi-layer hidden layer of the DUAE algorithm, and use the nonlinear activation function to adaptively extract the most important data features of the parameters:

[0027] h i =s(w i x i +b i ),i=1,2,...,j

[0028] Where j is the number of hidden layers, w i and b i They represent the weight and bias of the i-th hidden layer, hi is the output of the i-th hidden layer, and s is the nonlinear activation function.

[0029] In step 200, the traditional IDDPG algorithm adopts a deterministic strategy, which may limit the exploration ability of the unmanned underwater vehicle in the action space. In order to overcome this limitation, an improved IDDPG algorithm, namely the IDDPG algorithm, is proposed, which effectively improves the exploration of the control strategy by introducing continuous Gaussian noise, thereby avoiding falling into the local optimal solution.

[0030] Step 201, the IDDPG algorithm includes a prediction network and a target network. The prediction network includes an Actor network and a Critic network, which are used to predict the state and value function of the unmanned submersible in the marine environment. The target network is a copy of the prediction network and is regularly updated using a soft update strategy. The Actor network outputs an action μ(s t ), by taking action μ(s t ) introduces Gaussian noise to obtain action a t , which increases the exploration of the action space, enables the IDDPG algorithm to better balance the relationship between exploration and utilization, effectively remove noise and redundant information, and improve the robustness and performance of the algorithm.

[0031] a t =μ(s t )+Ν(0,σ)

[0032] Here, N(0,σ) represents the noise sampled from a Gaussian distribution with a mean of 0 and a standard deviation of σ.

[0033] Step 202: Critic network is used to evaluate the action value a t The pros and cons of the current moment are obtained by calculating the reward function r(s t ,a t ). By combining the current state st and action at, the value function Q(s) of the Critic network is calculated.t ,a t ), and changes the state to s t+1 .

[0034]

[0035] in, is the Bellman equation, γ is the discount factor, Q(s t+1 ,a t+1 ) indicates that the unmanned submersible is in state s t+1 Execute action a t+1 The obtained value function.

[0036] Step 203: Based on the value function output by the Critic network, the Actor network continuously adjusts the action strategy in the direction of maximizing the value function to maximize the global cumulative reward. In reinforcement learning, if the agent updates the strategy only based on the latest information, it may fall into the local optimum. Based on the experience replay technology, at each moment, the information generated by the interaction between the unmanned submersible and the ocean environment (state st, action a t , Reward t , the next moment state s t+1 ) is stored in the replay buffer. The replay buffer stores long-term information samples to ensure that information at different stages has a chance to be used, thereby balancing the distribution of samples and preventing certain states or actions from being overemphasized or ignored in the update. By randomly sampling the samples in the replay buffer in small batches, the correlation of the samples is effectively reduced. The loss function L of the Critic network is calculated based on the mean square error:

[0037]

[0038] Among them, N represents the number of samples in the small batch sampling, r t represents the reward value at time t. Q'(s t+1 ,a t ' +1 ) indicates that the target network is in state s t+1 Execute action a′ t+1 The obtained value function.

[0039] Step 300, adopt the soft update method to use the parameters of the prediction network Actor and Critic to update the parameters of the target network respectively to prevent the network from overfitting. Based on the IDDPG algorithm, an adaptive track control strategy is generated in real time to ensure that the unmanned submersible completes the track tracking task with high precision.

[0040] Those skilled in the art should understand that the discussion of any of the above embodiments is only exemplary and is not intended to imply that the scope of the present invention is limited to these examples; under the concept of the present invention, the technical features in the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other changes in different aspects of the present invention as described above, which are not provided in detail for the sake of simplicity. Any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. An adaptive track control method for an unmanned submersible, characterized in that: The following steps are involved: S1, unmanned submersible navigation data x i Input into the multi-layer hidden layer of DUAE algorithm, and use nonlinear activation function to adaptively extract the most important data features of parameters; S2, output the adaptive track control strategy of the unmanned submersible in real time through the IDDPG algorithm; The IDDPG algorithm introduces continuous Gaussian noise to increase the exploratory power of the action space. The IDDPG algorithm includes a prediction network and a target network. The prediction network includes an Actor network and a Critic network. The Critic network is used to evaluate the action value a t The advantages and disadvantages of combining the current state s t and action a t Calculate the value function of the Critic network.

2. The method for adaptive track control of an unmanned underwater vehicle according to claim 1, characterized in that: The DUAE algorithm uses multiple hidden layers to analyze the complex structure and correlation of data.

3. The adaptive track control method of an unmanned underwater vehicle according to claim 2, characterized in that: The IDDPG algorithm is implemented by taking action μ(s t ) introduces Gaussian noise to obtain action a t , increasing the exploration of the action space.

4. The adaptive track control method of an unmanned underwater vehicle according to claim 3, characterized in that: The Critic network of the IDDPG algorithm is used to evaluate the action value a t The pros and cons of the current moment are obtained by calculating the reward function r(s t ,a t ).

5. The adaptive track control method of an unmanned underwater vehicle according to claim 1, characterized in that: The Actor network of the IDDPG algorithm continuously adjusts the action strategy in the direction of maximizing the value function output by the Critic network to maximize the global cumulative reward.

6. The method for adaptive track control of an unmanned underwater vehicle according to claim 5, characterized in that: The IDDPG algorithm is based on the experience replay technology, and at each moment, the information generated by the interaction between the unmanned submersible and the marine environment is stored in a replay buffer.

7. The method for adaptive track control of an unmanned underwater vehicle according to claim 1, characterized in that: The IDDPG algorithm samples the samples in the replay buffer by using small batch random sampling.

8. The method for adaptive track control of an unmanned underwater vehicle according to claim 7, characterized in that: The IDDPG algorithm adopts a soft update method, using the parameters of the prediction network Actor and Critic to update the parameters of the target network respectively to prevent the network from overfitting.

Images