Bistatic sonar underwater target tracking AUV decision-making method based on deep reinforcement learning

Abstract

The invention relates to the field of autonomous underwater vehicle (AUV) decision making, deep reinforcement learning and underwater target tracking, in particular to a bistatic sonar underwater target tracking AUV decision making method based on deep reinforcement learning. According to the method, the end-to-end deep reinforcement learning method is adopted to learn the action decision of the AUV carrying the receiver for optimal target tracking, so that the AUV can adjust the motion of the AUV according to the change of the situation so as to improve the tracking precision and the tracking holding time of the target, and the scheme is short in solving time and meets the real-time requirement of target tracking. When the network is trained, rolling horizon optimization and deep reinforcement learning are combined, and strategy learning is guided by using an existing prior model, so that the safety of the AUV can be ensured, and the stability of the strategy is improved.

Description

Technical Field

[0001] The present invention relates to the fields of autonomous underwater robot (AUV) decision-making, deep reinforcement learning and underwater target tracking, and in particular to a dual-base sonar underwater target tracking AUV decision-making method based on deep reinforcement learning. Background Art

[0002] Marine security is an important part of national defense, and the detection and tracking of underwater targets is the main challenge facing marine security. Affected by the characteristics of the underwater environment, sound waves are currently the only way to transmit long distances underwater, and sonar is the only effective tool for long-distance underwater target detection and tracking. Due to the improvement of underwater target noise reduction technology, passive sonar has been unable to meet the needs of underwater target tracking. Thanks to the high sound source level of the sound source, multi-base sonar, especially dual-base sonar, has become an effective tool for underwater target tracking. In particular, dual-base sonar composed of sonars carried by autonomous underwater robots (AUVs) has the advantages of good maneuverability, high flexibility, and durability, and has received increasing attention at home and abroad.

[0003] Underwater targets are maneuverable, which leads to complex and changeable target tracking situations. AUVs need to adjust their behavior in real time according to situation changes to maximize the tracking accuracy and tracking retention time of targets. To address this problem, the current mainstream approach is to establish an environmental model based on human prior knowledge of the environment, and to establish an objective function based on the model (such as based on signal-to-noise ratio, detection probability, or information gain, etc.), and finally use effective solution tools (such as tree search algorithms) to obtain a solution that maximizes the objective function. However, this model-based approach relies on prior knowledge to establish a model. Insufficient prior knowledge will lead to inaccurate models, which will greatly reduce performance. In addition, due to the large state space, the problem solution is very complex and time-consuming, which makes it difficult to meet the AUV computing load requirements and the real-time requirements of underwater target tracking decisions.

[0004] In recent years, with the development of deep reinforcement learning and its successful application in many fields, using deep reinforcement learning to solve the decision-making problem of AUV underwater target tracking has become a promising way. On the one hand, model-free deep reinforcement learning does not require prior knowledge and models, and AUV can directly learn strategies by interacting with the environment. On the other hand, the trained policy network can be deployed on the AUV, and the calculation is simple and efficient. However, AUV decision-making based on deep reinforcement learning still faces challenges. First, directly placing an inexperienced AUV in the environment to interact threatens the safety of the AUV. Second, the performance of the strategy obtained entirely based on deep reinforcement learning cannot be guaranteed. In addition, the sample utilization rate of model-free deep reinforcement learning is low, and a lot of manpower, material and financial resources are required for training. Summary of the invention

[0005] Aiming at the shortcomings of existing methods in the autonomous decision-making application of dual-base sonar underwater target tracking AUV, the present invention provides a dual-base sonar underwater target tracking AUV decision-making method based on deep reinforcement learning, which avoids excessive reliance on prior models, ensures the safety of AUV while enabling AUV to autonomously learn better strategies, and can greatly reduce the number of samples required for training. The trained strategy has the advantages of high performance, short decision-making time and low computational complexity.

[0006] The technical solution adopted by the present invention to achieve the above-mentioned purpose is: a dual-base sonar underwater target tracking AUV decision method based on deep reinforcement learning, comprising the following steps:

[0007] A dual-base sonar underwater target positioning model is established to resolve the measurements generated by acoustic signal processing into two-dimensional coordinates; based on the resolved two-dimensional coordinate measurements, the state and state covariance matrix of the underwater target are estimated using a multi-target tracking algorithm;

[0008] Establish a dual-base sonar underwater target tracking environment model. Based on the dual-base sonar underwater target tracking environment model, dual-base sonar underwater target positioning model and multi-target tracking algorithm, establish the objective function of the rolling time domain optimization strategy and construct a solution method to obtain action decisions;

[0009] Based on the deep neural network, a deep reinforcement learning strategy network and evaluation network are established to output the AUV's action decision and evaluation state-action value respectively; at the same time, the rolling time domain optimization strategy and deep reinforcement learning strategy are used to make the AUV interact with the environment, and the obtained samples are stored in the experience pool;

[0010] Use the data from the experience pool to train the strategy network and evaluation network and adjust the neural network parameters;

[0011] After the training is completed, the trained strategy network is used to make AUV action decisions.

[0012] The dual-base sonar underwater target positioning model is established to resolve the measurements generated by acoustic signal processing into two-dimensional coordinates, specifically:

[0013] The positions of the sound source, target and receiver are connected to form a triangle. Based on the sound arrival time and arrival angle obtained by acoustic signal processing, the two-dimensional coordinates of the target are calculated by combining the sound source coordinates and the receiver coordinates.

[0014] The multi-target tracking algorithm is used to estimate the underwater target state and state covariance matrix, specifically:

[0015] Based on the belief propagation algorithm, the Gaussian mixture approach is adopted to perform data association and filtering on the solved two-dimensional coordinates of the target, and the estimated target state including position and velocity and the state covariance matrix are obtained.

[0016] The establishment of a dual-base sonar underwater target tracking environment model includes a target intensity model, an acoustic propagation model, a sonar directivity model, and a measurement error model, specifically:

[0017] The butterfly diagram of the target intensity of the monostatic sonar is fitted and extended to the bistatic sonar to obtain the target intensity model of the bistatic sonar.

[0018] A sound propagation model is established based on the BELLHOP model to calculate the sound propagation loss;

[0019] Establish a sonar directivity model for sonar linear arrays applied to AUVs;

[0020] The acoustic signal-to-noise ratio is calculated based on the target intensity model, acoustic propagation model and sonar directivity model, and then a measurement error model is established based on the Woodward relationship to obtain the error of the sound arrival time and the error of the sound arrival angle.

[0021] Based on the dual-base sonar underwater target tracking environment model, the dual-base sonar underwater target positioning model and the multi-target tracking algorithm, the objective function of the rolling time domain optimization strategy is established and the solution method is constructed to obtain the optimal decision, specifically:

[0022] 1) Define the information gain based on the trace of the target state estimation prior covariance matrix and the posterior covariance matrix:

[0023]

[0024] Among them, P j|j-1 is the prior covariance matrix, is the posterior covariance matrix, tr() represents the trace of the matrix, N opt To optimize the step size, is the AUV action decision of the jth step in the i-th decision sequence, Indicates execution of an action Information gain of

[0025] exist During the calculation of , the required target position measurement is generated as follows:

[0026] e. Calculate unbiased sound arrival time and arrival angle based on the predicted target state;

[0027] f. Calculate the measurement error of the sound arrival time and arrival angle based on the dual-base sonar underwater target tracking environment model;

[0028] g. Sampling the final biased arrival time from a Gaussian distribution with the unbiased arrival time as the mean and the arrival time measurement error as the standard deviation; sampling the final biased arrival angle from a Gaussian distribution with the unbiased arrival angle as the mean and the arrival angle measurement error as the standard deviation;

[0029] h. Based on the arrival time and arrival angle with deviation, the target position is calculated and measured by the dual-base sonar underwater target positioning model;

[0030] 2) Establish the objective function with the goal of maximizing information gain:

[0031]

[0032] in, is the i-th decision sequence, w j is the weight of the jth step;

[0033] 3) Establish a search tree, where nodes are used to represent candidate AUV action decisions, and use the Monte Carlo tree search algorithm to solve the objective function to find the optimal action decision.

[0034] The strategy network and evaluation network for reinforcement learning based on the deep neural network are specifically:

[0035] The input of the policy network is the source state, the receiver state, the estimated target state, and the diagonal elements of the estimated target posterior covariance matrix. The output is the mean and standard deviation of the action. The policy network uses 2 hidden layers, with full connections between layers and the activation function is ReLU.

[0036] The input of the evaluation network is the source state, the receiver state, the estimated target state, the diagonal elements of the estimated target posterior covariance matrix, and the action of the AUV. The output is the evaluation value of the state-action pair. The evaluation network uses 2 hidden layers, with full connections between layers and the activation function is ReLU.

[0037] The rolling time domain optimization and deep reinforcement learning strategies are used to make the AUV interact with the environment, and the obtained samples are stored in the experience pool, specifically:

[0038] In the early stage of training, a random action strategy is adopted with probability p, and a rolling time domain optimization strategy is adopted with probability 1-p. The initial value of p is 1 and decreases with the increase of training steps.

[0039] In the middle and late stages of training, deep reinforcement learning strategies are used to take actions;

[0040] The experience collected during training is formed into a 5-tuple {s, a, r, s′, d}; where s is the current state, including the source state, receiver state, estimated target state, and the diagonal elements of the estimated target posterior covariance matrix, a is the action taken by the AUV, s′ is the next state reached after the AUV takes action a, r is the reward obtained by taking an action in the current state and transferring to the next state, and d is used to indicate whether the next state is a terminal state. The collected experience is stored in the experience pool.

[0041] Deep reinforcement learning uses the Soft Actor-Critic algorithm, and the method for adjusting neural network parameters is Adam.

[0042] The AUV action decision is made using the trained strategy network, specifically:

[0043] The source state, receiver state, estimated target state, and the diagonal elements of the estimated target posterior covariance matrix are input to the policy network, and the AUV executes the action output of the policy network to track the underwater target.

[0044] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the dual-base sonar underwater target tracking AUV decision-making method based on deep reinforcement learning.

[0045] The present invention has the following beneficial effects and advantages:

[0046] 1. AUV decision-making based on deep reinforcement learning method combines the fitting ability of deep learning with the learning ability of reinforcement learning, enabling AUV to autonomously explore and learn action strategies and perform end-to-end output.

[0047] 2. Combining the model-based rolling horizon optimization strategy with model-free deep reinforcement learning for network training can use existing models to ensure the safety of AUV learning, reduce training costs, improve decision-making performance, and avoid excessive reliance on prior models.

[0048] 3. In the model-based rolling time domain optimization, the objective function is established based on information gain, which can effectively characterize the AUV underwater target tracking performance.

[0049] 4. The Monte Carlo tree search algorithm is used to solve the rolling time domain optimization problem, which is more efficient than other methods such as brute force search. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 It is the geometric positioning diagram of the dual-base sonar in the present invention;

[0051] Figure 2 is the deep reinforcement learning training flow chart of the present invention;

[0052] Figure 3 It is a network structure diagram of the deep reinforcement learning strategy of the present invention. DETAILED DESCRIPTION

[0053] The present invention is further described in detail below in conjunction with the embodiments. The method steps are described in conjunction with the accompanying drawings.

[0054] The basic idea of ​​the present invention is to combine the model-based rolling time domain optimization strategy with the model-free deep reinforcement learning strategy for network training, use the trained strategy network to make decisions, and complete the action decision of the AUV for optimal target tracking. In view of the problem that the environmental model for underwater target tracking is difficult to establish accurately, the present invention uses a model-free deep reinforcement learning method to allow the AUV to explore and learn strategies autonomously, and take an end-to-end approach to quickly make action decisions. In view of the safety issues and lack of performance guarantee faced by AUVs in the training of model-free deep reinforcement learning methods, the present invention uses a model-based rolling time domain optimization strategy to guide AUV learning, which can ensure the effectiveness of the learned strategy while improving the safety of the AUV.

[0055] like Figure 2 , Figure 3 As shown in FIG. 1 , the decision-making method for dual-base sonar underwater target tracking AUV based on deep reinforcement learning includes the following steps:

[0056] Step 1: Establish a dual-base sonar underwater target positioning model and resolve the measurements generated by acoustic signal processing into two-dimensional coordinates. Connect the positions of the sound source, target, and receiver to form a triangle, such as Figure 1 As shown in the figure, each quantity represents the real value. The time it takes for the measured sound wave to travel from the sound source to the target and then to the receiver is recorded as The angle between the measured sound wave and the receiver is recorded as The measured two-dimensional coordinates of the target are

[0057]

[0058] where [x r ,y r ] T and φ are the position and direction of the receiver, respectively, The distance from the measured target to the receiver is calculated as:

[0059]

[0060] Where c is the speed of sound underwater, d sr and The calculation method is:

[0061]

[0062]

[0063] where [x s ,y s ] T is the location of the sound source.

[0064] Step 2: Based on the solved two-dimensional coordinate measurement, the multi-target tracking algorithm is used to estimate the state of the underwater target including position and velocity and the state covariance matrix. Specifically, based on the belief propagation algorithm, the Gaussian mixture implementation approach is adopted to perform data association and filtering on a large number of inaccurate and noisy target two-dimensional coordinates to obtain the estimated target state and state covariance matrix.

[0065] Step 3: Establish a dual-base sonar underwater target tracking environment model, including target intensity model, sound propagation model, sonar directivity model, and measurement error model.

[0066] (1) For the target intensity model, the butterfly diagram of the single-base sonar target intensity, which is difficult to apply in practice, is first fitted. The calculation formula is:

[0067]

[0068] Among them, TS max is the maximum target strength (broadside direction), TS min is the minimum target intensity (end-fire direction), θ is the azimuth, TS mono (θ) is the target intensity of the fitting.

[0069] Then, the above formula is extended to the dual-base sonar to obtain the target intensity model of the dual-base sonar, and the calculation formula is:

[0070]

[0071] Where θ1 is the angle between the line connecting the target scattering center and the center of the sound source and the target moving direction, θ2 is the angle between the line connecting the target scattering center and the center of the receiver and the target moving direction, TS bi (θ1,θ2) is the target intensity of the fitting.

[0072] (2) For the sound propagation model, a sound propagation model is established based on the BELLHOP model. The input is known quantities such as underwater temperature, salinity, and topography, and the output is the sound propagation loss.

[0073] (3) For the sonar directivity model, a sonar directivity model of a sonar array widely used in AUVs is established: when the angle between the line connecting the target and the center of the receiver and the straight line where the receiver is located is less than θ efWhen , the directivity index is set to 0; otherwise, it is the maximum directivity index.

[0074] (4) Establish a measurement error model based on the Woodward relationship:

[0075]

[0076]

[0077] where σ τ is the error in the sound arrival time, σ θ is the error of the sound arrival angle, κ1 and κ2 are proportional coefficients, δ(τ) and δ(θ) are the sonar time resolution and angle resolution respectively, SNR is the signal-to-noise ratio, and is calculated as follows:

[0078] SNR=SL-TL st -TL tr +TS-NL+DI

[0079] The sound source level SL is a fixed constant, TL st and TL tr are the propagation loss of the sound signal from the sound source to the target and from the target to the receiver, respectively, both of which are calculated through the sound propagation model. The target strength TS is obtained through the target strength model. The noise level NL is a constant. The directivity index DI is obtained through the sonar directivity model.

[0080] Step 4: Based on the above-established dual-base sonar underwater target positioning model, dual-base sonar underwater target tracking environment model and multi-target tracking algorithm, establish the objective function and solution method of rolling time domain optimization.

[0081] (1) The objective function of the rolling time domain optimization is established based on information gain, which specifically includes two steps.

[0082] A. Define the information gain based on the target state estimation prior covariance matrix and posterior covariance matrix given in step 2, and calculate it as:

[0083]

[0084] Among them, P j|j-1 is the prior covariance matrix, is the posterior covariance matrix, tr() represents the trace of the matrix, N opt To optimize the step size, is the AUV action decision of the jth step in the i-th decision sequence, where is the heading angle change, Indicates execution of an action The information gain of

[0085] because is the future posterior covariance matrix, so calculate The desired measurement is actually the predicted target measurement, which is generated as follows:

[0086] a. Calculate unbiased sound arrival time and arrival angle based on the predicted target state;

[0087] b. Calculate the signal-to-noise ratio of the acoustic signal based on the target intensity model, acoustic propagation model, and sonar directivity model, and calculate the measurement errors of the acoustic arrival time and arrival angle based on the measurement error model;

[0088] c. The final biased arrival time is obtained by sampling from a Gaussian distribution with the unbiased arrival time as the mean and the arrival time measurement error as the standard deviation; the final biased arrival angle is obtained by sampling from a Gaussian distribution with the unbiased arrival angle as the mean and the arrival angle measurement error as the standard deviation;

[0089] d. Based on the arrival time and arrival angle with deviation, the measured target position is solved through the dual-base sonar underwater target positioning model.

[0090] B. Establish the objective function with the goal of maximizing information gain:

[0091]

[0092] in is the i-th decision sequence, w j is the weight of the jth step.

[0093] (2) The solution to the above objective function is established as a search tree, and the Monte Carlo tree search algorithm with high search efficiency is used to find the optimal decision sequence and execute the first action decision. The steps of the Monte Carlo tree search algorithm are:

[0094] A. Selection. Starting from the root node of the tree, find the non-terminal node with the highest priority that has not been fully expanded from top to bottom. Each node represents a candidate AUV heading angle change.

[0095] B. Expand. Add child nodes to the node found in the previous step.

[0096] C. Simulation. Starting from the newly added child node, execute the default strategy (here is the random strategy) to select nodes until the termination node is reached, and obtain the reward value of the sequence, that is, the weighted sum of information gain.

[0097] D. Backtracking: Pass the obtained sequence reward value upward from the selected node, that is, add the reward value to the estimated value of each node, and increase the number of expansions of each node by 1.

[0098] When the preset search time is reached, the node with the largest average estimated value (estimated value divided by the number of expansions) is selected, and the AUV heading angle change represented by the node is executed.

[0099] Step 5: Establish a policy network and an evaluation network based on the deep neural network, which are used to output the AUV's action decision and evaluate the state-action value respectively.

[0100] (1) Policy network. The input of the policy network is the source state, the receiver state, the estimated target state, and the diagonal elements of the estimated target posterior covariance matrix, which are 16-dimensional vectors; the output is the mean and standard deviation of the action, both scalars; the network uses 2 hidden layers, the number of hidden layer nodes is 64, the layers are fully connected, and the activation function is ReLU. Among them, the action is the change in the AUV heading angle.

[0101] (2) Evaluation network. The input of the evaluation network is the source state, the receiver state, the estimated target state, the diagonal elements of the estimated target posterior covariance matrix, and the action sampling of the AUV, which is a 17-dimensional vector; the output is the estimated value of the state-action pair; the network uses 2 hidden layers, the number of hidden layer nodes is 64, the layers are fully connected, and the activation function is ReLU. Among them, in the state-action pair, the state includes the source state, the receiver state, the estimated target state, and the diagonal elements of the estimated target posterior covariance matrix; the action is the change in the AUV heading angle.

[0102] Step 6: Use the rolling time domain optimization described in step 4 and the deep reinforcement learning strategy described in step 5 to make the AUV interact with the environment, and store the obtained samples in the experience pool. Specifically, in the early stage of training, a random action strategy is adopted with probability p, and a rolling time domain optimization strategy is adopted with probability 1-p. The initial value of p is 1 and decreases with the increase of training steps. The purpose is to prevent the network from overfitting to the rolling time domain optimization strategy; in the middle and late stages of training, the deep reinforcement learning strategy is used to take actions, and the learning and exploration capabilities of the network surpass the rolling time domain optimization strategy. The collected experience is a 5-tuple {s, a, r, s′, d}. Among them, s is the current state, including the source state, the receiver state, the estimated target state, and the diagonal elements of the estimated target posterior covariance matrix, a is the action taken by the AUV, s′ is the next state reached by the AUV after taking action a, r is the reward obtained by taking an action in the current state and transferring to the next state, and d is used to indicate whether the next state is a terminal state. The collected experience is stored in the experience pool for training the deep reinforcement learning algorithm.

[0103] Step 7: When the sample size of the experience pool reaches the preset value, the Soft Actor-Critic algorithm is used to train the neural network using the data from the experience pool and adjust the neural network parameters. Specifically:

[0104] (1) Randomly select a batch of experiences B = {(s, a, r, s′, d)} from the experience pool.

[0105] (2) Calculate the target value y:

[0106]

[0107] Where γ is the discount factor, α is the trade-off coefficient, is the i-th target evaluation network, φ tar,i is the network parameter, π θ For the strategy network, Sampled from π θ (·|s′).

[0108] (3) Use the Adam method to update the parameters of the evaluation network through gradient descent:

[0109]

[0110] in is the i-th evaluation network, φ i is the network parameter, |B| represents the number of experiences.

[0111] (4) Use the Adam method to update the parameters of the policy network through gradient ascent:

[0112]

[0113] in Sampled from π θ (·∣s).

[0114] (5) Update the target network parameter φ targ,i :

[0115] φ targ,i ←ρφ targ,i +(1-ρ)φ i ,i=1,2

[0116] Where ρ is the update coefficient.

[0117] Repeat steps 6 and 7, that is, continuously let the AUV interact with the environment and train the neural network until the training reaches the preset number of steps.

[0118] Step 8: After training is completed, use the trained strategy network to make AUV action decisions. Specifically:

[0119] The source state, receiver state, estimated target state, and diagonal elements of the estimated target posterior covariance matrix are input to the policy network, which outputs the AUV action, i.e., the AUV heading angle change, which the AUV performs to maintain accurate and persistent tracking of the target.

[0120] In addition to the above embodiments, the present invention may also have other implementation modes. Any technical solutions formed by equivalent replacement or equivalent transformation shall fall within the protection scope required by the present invention.

Claims

1. A dual-base sonar underwater target tracking AUV decision-making method based on deep reinforcement learning, characterized by The following steps are involved: A dual-base sonar underwater target positioning model is established to resolve the measurements generated by acoustic signal processing into two-dimensional coordinates; based on the resolved two-dimensional coordinate measurements, the state and state covariance matrix of the underwater target are estimated using a multi-target tracking algorithm; Establish a dual-base sonar underwater target tracking environment model. Based on the dual-base sonar underwater target tracking environment model, dual-base sonar underwater target positioning model and multi-target tracking algorithm, establish the objective function of the rolling time domain optimization strategy and construct a solution method to obtain action decisions; Based on the deep neural network, a deep reinforcement learning strategy network and evaluation network are established to output the AUV's action decision and evaluation state-action value respectively; at the same time, the rolling time domain optimization strategy and deep reinforcement learning strategy are used to make the AUV interact with the environment, and the obtained samples are stored in the experience pool; Use the data from the experience pool to train the strategy network and evaluation network and adjust the neural network parameters; After the training is completed, the trained strategy network is used to make AUV action decisions.

2. The dual-base sonar underwater target tracking AUV decision method based on deep reinforcement learning according to claim 1 is characterized in that: The dual-base sonar underwater target positioning model is established to resolve the measurements generated by acoustic signal processing into two-dimensional coordinates, specifically: The positions of the sound source, target and receiver are connected to form a triangle. Based on the sound arrival time and arrival angle obtained by acoustic signal processing, the two-dimensional coordinates of the target are calculated by combining the sound source coordinates and the receiver coordinates.

3. The dual-base sonar underwater target tracking AUV decision method based on deep reinforcement learning according to claim 1 is characterized in that: The multi-target tracking algorithm is used to estimate the underwater target state and state covariance matrix, specifically: Based on the belief propagation algorithm, the Gaussian mixture approach is adopted to perform data association and filtering on the solved two-dimensional coordinates of the target, and the estimated target state including position and velocity and the state covariance matrix are obtained.

4. The dual-base sonar underwater target tracking AUV decision method based on deep reinforcement learning according to claim 1 is characterized in that: The establishment of a dual-base sonar underwater target tracking environment model includes a target intensity model, an acoustic propagation model, a sonar directivity model, and a measurement error model, specifically: The butterfly diagram of the target intensity of the monostatic sonar is fitted and extended to the bistatic sonar to obtain the target intensity model of the bistatic sonar. A sound propagation model is established based on the BELLHOP model to calculate the sound propagation loss; Establish a sonar directivity model for sonar linear arrays applied to AUVs; The acoustic signal-to-noise ratio is calculated based on the target intensity model, acoustic propagation model and sonar directivity model, and then a measurement error model is established based on the Woodward relationship to obtain the error of the sound arrival time and the error of the sound arrival angle.

5. The dual-base sonar underwater target tracking AUV decision method based on deep reinforcement learning according to claim 1, characterized in that: Based on the dual-base sonar underwater target tracking environment model, the dual-base sonar underwater target positioning model and the multi-target tracking algorithm, the objective function of the rolling time domain optimization strategy is established and the solution method is constructed to obtain the optimal decision, specifically: 1) Define the information gain based on the trace of the target state estimation prior covariance matrix and the posterior covariance matrix: Among them, P j|j-1 is the prior covariance matrix, is the posterior covariance matrix, tr() represents the trace of the matrix, N opt To optimize the step size, is the AUV action decision of the jth step in the i-th decision sequence, Indicates execution of an action Information gain of exist During the calculation of , the required target position measurement is generated as follows: a. Calculate unbiased sound arrival time and arrival angle based on the predicted target state; b. Calculate the measurement errors of the sound arrival time and arrival angle based on the dual-base sonar underwater target tracking environment model; c. The final biased arrival time is obtained by sampling from a Gaussian distribution with the unbiased arrival time as the mean and the arrival time measurement error as the standard deviation; the final biased arrival angle is obtained by sampling from a Gaussian distribution with the unbiased arrival angle as the mean and the arrival angle measurement error as the standard deviation; d. Based on the arrival time and arrival angle with deviation, the target position is calculated and measured by the dual-base sonar underwater target positioning model; 2) Establish the objective function with the goal of maximizing information gain: in, is the i-th decision sequence, w j is the weight of the jth step; 3) Establish a search tree, where nodes are used to represent candidate AUV action decisions, and use the Monte Carlo tree search algorithm to solve the objective function to find the optimal action decision.

6. The dual-base sonar underwater target tracking AUV decision method based on deep reinforcement learning according to claim 1 is characterized in that: The strategy network and evaluation network for reinforcement learning based on the deep neural network are specifically: The input of the policy network is the source state, the receiver state, the estimated target state, and the diagonal elements of the estimated target posterior covariance matrix. The output is the mean and standard deviation of the action. The policy network uses 2 hidden layers, with full connections between layers and the activation function is ReLU. The input of the evaluation network is the source state, the receiver state, the estimated target state, the diagonal elements of the estimated target posterior covariance matrix, and the action of the AUV. The output is the evaluation value of the state-action pair. The evaluation network uses 2 hidden layers, with full connections between layers and the activation function is ReLU.

7. The dual-base sonar underwater target tracking AUV decision method based on deep reinforcement learning according to claim 1, characterized in that: The rolling time domain optimization and deep reinforcement learning strategies are used to make the AUV interact with the environment, and the obtained samples are stored in the experience pool, specifically: In the early stage of training, a random action strategy is adopted with probability p, and a rolling time domain optimization strategy is adopted with probability 1-p. The initial value of p is 1 and decreases with the increase of training steps. In the middle and late stages of training, deep reinforcement learning strategies are used to take actions; The experience collected during training is composed of a 5-tuple {s,a,r,s ′ ,d}; where s is the current state, including the source state, receiver state, estimated target state, and the diagonal elements of the estimated target posterior covariance matrix, a is the action taken by the AUV, and s ′ is the next state reached by the AUV after taking action a, r is the reward obtained by taking action in the current state and transferring to the next state, d is used to indicate whether the next state is the terminal state, and the collected experience is stored in the experience pool.

8. The dual-base sonar underwater target tracking AUV decision method based on deep reinforcement learning according to claim 1, characterized in that: Deep reinforcement learning uses the Soft Actor-Critic algorithm, and the method for adjusting neural network parameters is Adam.

9. The dual-base sonar underwater target tracking AUV decision method based on deep reinforcement learning according to claim 1, characterized in that: The AUV action decision is made using the trained strategy network, specifically: The source state, receiver state, estimated target state, and the diagonal elements of the estimated target posterior covariance matrix are input to the policy network, and the AUV executes the action output of the policy network to track the underwater target.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, the deep reinforcement learning-based dual-base sonar underwater target tracking AUV decision-making method as described in any one of claims 1 to 9 is implemented.

Images