AUV causal reinforcement learning decision-making method for bistatic sonar underwater target tracking

Abstract

The invention discloses an autonomous underwater vehicle (AUV) causal reinforcement learning decision-making method for bistatic sonar underwater target tracking, and belongs to the field of autonomous underwater vehicle (AUV) intelligent decision-making and underwater target tracking. The method comprises the following steps: firstly, constructing a causal model for describing a target tracking problem, and then establishing a measurement generation model based on a heterovariance neural network and a binary classification network; on the basis, a real-time maneuvering strategy of the AUV is trained by adopting a deep reinforcement learning framework, and high-fidelity training data is generated by utilizing an anti-factual reasoning mechanism so as to optimize a learning process. According to the method, the problems of insufficient tracking precision and low sample efficiency of a traditional decision method are effectively solved, the tracking precision and the tracking holding time of the AUV to the underwater target can be remarkably improved, and meanwhile, the data requirement for a real training sample is greatly reduced.

Description

Technical Field

[0001] This invention relates to the field of intelligent decision-making and underwater target tracking for autonomous underwater vehicles (AUVs), specifically a causal reinforcement learning decision-making method for AUVs using bistatic sonar underwater target tracking. Background Technology

[0002] Underwater target tracking aims to estimate the target's state as accurately as possible and is crucial in marine engineering and applications. In recent years, bistatic sonar systems comprised of autonomous underwater vehicles (AUVs) equipped with sonar have gained increasing attention due to their flexibility, autonomy, and potential to improve target tracking performance. In such systems, the geometry between the sound source, target, and receiver has a decisive impact on tracking accuracy; therefore, the AUV needs to be able to adjust its trajectory in real time based on the target's movement to achieve continuous and stable tracking.

[0003] Traditionally, such problems are often solved using planning-based strategies, with tree search being a common technique. This method maps states to nodes and actions to directed edges, obtaining feasible solutions by expanding nodes and evaluating their future rewards. Although this approach achieves non-shortsight optimization through multi-step look-ahead, it has a high computational burden, making real-time processing difficult on resource-constrained AUV platforms. Furthermore, tree search requires discretizing the action space, which limits the diversity of AUV maneuvers and may reduce the optimality of the final strategy.

[0004] Deep reinforcement learning, combining the powerful representational capabilities of deep neural networks with the sequential decision-making framework of reinforcement learning, has become an important method in the field of robotic intelligent decision-making in recent years. Within this framework, the policy network receives observations as input and outputs actions. After executing actions in the environment, it updates network parameters based on collected experience to maximize cumulative rewards. This end-to-end learning mechanism is computationally efficient during the inference phase and supports continuous action spaces, offering significant advantages over planning-based methods. However, the inefficient sample utilization inherent in deep reinforcement learning still limits its application in real-world scenarios. This problem is particularly prominent in bistatic sonar target tracking tasks where real-world data acquisition is difficult, maritime trials are costly, and high-fidelity simulations are computationally expensive.

[0005] Model-based reinforcement learning is an effective way to alleviate the problem of low sample efficiency, but it is difficult to directly apply to bistatic sonar target tracking scenarios. This is because such scenarios involve complex sound propagation processes, environmental uncertainties, and highly nonlinear coupling between system variables, making accurate dynamic modeling extremely challenging. Meanwhile, recent advances in causal reasoning have shown its potential to build robust models by utilizing the inherent causal structure between variables. In particular, counterfactual reasoning does not require a complete state transition model; instead, it generates high-quality simulation samples by mining the inherent causal independence within the problem, thereby effectively expanding training data and improving learning efficiency. Therefore, there is an urgent need for a deep reinforcement learning method that combines counterfactual reasoning to effectively solve the problem of autonomous decision-making for bistatic sonar underwater target tracking AUVs. Summary of the Invention

[0006] To address the current limitations of existing methods in effectively solving the autonomous decision-making problem of AUVs tracking bistatic sonar underwater targets, this invention provides a causal reinforcement learning decision-making method for AUVs tracking bistatic sonar underwater targets. This method overcomes the performance limitations of traditional planning-based methods and effectively alleviates the problem of low training sample efficiency in deep reinforcement learning methods.

[0007] The technical solution adopted by this invention to achieve the above objectives is: an AUV causal reinforcement learning decision-making method for bistatic sonar underwater target tracking, comprising the following steps:

[0008] The target tracking problem is formally described using a causal graph model; the causal relationships between variables in the target tracking problem are represented using a structural causal model.

[0009] A measurement generation model based on neural networks is established to generate counterfactual data;

[0010] A deep reinforcement learning policy network and evaluation network are established based on neural networks, where the policy network is used to generate the actions of the AUV and the evaluation network is used to estimate the state-action value.

[0011] A deep reinforcement learning policy network is used to generate AUV actions and interact with the environment. At the same time, counterfactual data is generated based on a measurement-generative model. Real interaction data and counterfactual data are stored in the experience pool respectively.

[0012] Update the neural network parameters of the measurement generation model using actual measurement data;

[0013] The parameters of the deep reinforcement learning neural network are updated using real and counterfactual data from the experience pool;

[0014] After deep reinforcement learning training is completed, the output of the trained policy network is used as the action policy of AUV.

[0015] The formal description of the target tracking problem based on the causal graph model is as follows:

[0016] Each variable is represented as a node, and the relationships between variables are represented as directed edges, forming a directed acyclic graph; the variables include source state, AUV state, true target state, AUV action, target measurement, target state estimation, clutter, and reward.

[0017] The structural causal model represents the causal relationships between variables in the target tracking problem, specifically as follows:

[0018] ① The true target state at any given moment The structure assignment is as follows: ;

[0019] in, Indicates location, Represents velocity, function for:

[0020]

[0021] in, The sampling interval;

[0022] ②AUV status The structure assignment is as follows: ;

[0023] in For decision-making actions, The calculation method for each component is as follows:

[0024]

[0025] in For the decision time interval, For AUV speed, For the AUV's heading angle, This refers to the change in the AUV's heading angle, i.e., the decision-making action.

[0026] ③The structural assignment for target measurement is as follows: ;

[0027] in, For independent noise that follows a standard Gaussian distribution, The specific implementation is as follows:

[0028] Given the state of the sound source Real target state and AUV status ideal signal arrival time and the angle of arrival for:

[0029]

[0030]

[0031] in and These represent the distances from the sound source to the target and from the target to the AUV, respectively. The speed of sound underwater; due to environmental interference, the actual signal arrival time is... and the angle of arrival for:

[0032]

[0033]

[0034] in Indicates a Gaussian distribution. and The standard deviation depends on the sampling results. ,Right now , ;

[0035] Finally, target measurement The calculation method is as follows:

[0036]

[0037] in Indicates the measurement of the target location. The distance between the target and the AUV is calculated based on bistatic geometry and the actual signal arrival time and angle. The calculation method is as follows:

[0038]

[0039] in, and The calculation method is as follows:

[0040]

[0041]

[0042] ④ Target state estimation The structure assignment is as follows: ;

[0043] in For clutter, This is achieved through a target tracking algorithm based on belief propagation.

[0044] The measurement generation model based on neural networks is specifically as follows:

[0045] A measurement generation model is established based on heteroscedastic neural networks and binary classification networks;

[0046] The heteroscedastic neural network includes a first hidden layer, a second hidden layer, and a third hidden layer in sequence, wherein the third hidden layer is composed of two fourth hidden layers connected in parallel; the size of the first hidden layer and the second hidden layer is 512, and the size of the fourth hidden layer is 256;

[0047] The binary classification network includes a fifth hidden layer, a sixth hidden layer, and a seventh hidden layer in sequence. The size of the fifth hidden layer is 256, and the sizes of the sixth and seventh hidden layers are 128.

[0048] The heteroscedastic neural network is used to predict the standard deviation of the signal arrival time and angle of arrival, and the binary classification network is used to identify whether the target measurement exists. The output of the heteroscedastic neural network is multiplied by the output of the binary classification network to obtain the output of the measurement generation model, including the standard deviation of the signal arrival time. and the standard deviation of the signal arrival angle .

[0049] The policy network and evaluation network for deep reinforcement learning based on neural networks are specifically as follows:

[0050] A fully connected neural network is used to implement the policy network and the evaluation network; the input of the policy network is the sound source state. AUV status and target state estimation The output is the AUV action; the input to the evaluation network is the sound source state. AUV status Target state estimation The AUV action outputs a state-action estimate.

[0051] The process involves using a deep reinforcement learning policy network to generate AUV actions and interactions with the environment, while simultaneously generating counterfactual data based on a measurement-generated model. The real interaction data and counterfactual data are stored separately in an experience pool. Specifically:

[0052] (1) In the environment, the deep reinforcement learning policy network responds to observations Generate AUV action AUV performs actions To obtain the observation at the next moment and rewards ,Will( The data is stored in the real experience pool, where observations are used to represent the source state, AUV state, and target state estimates, and AUV actions are used to represent the AUV heading angle change; the real target state, as well as all measurements of mixed targets and clutter, are stored in the data pool.

[0053] (2) Take a set of experiences from the real experience pool ( );from Extract the target state estimate from and retrieve from the data pool All measurements at any given time are used to find the target tracking algorithm's data correlation. The exogenous variable at time, i.e., clutter; Input policy network to generate counterfactual actions Counterfactual actions Right now Substitute AUV status assignment, and combine AUV status in The counterfactual AUV state is obtained. According to the counterfactual AUV status , sound source status The actual target state in the data pool The arrival time and angle of the target signal are predicted by measurement-generated models, and the target position is calculated based on the target measurement calculation method. Combined with the identified clutter, a counterfactual target state estimate is obtained through a target tracking algorithm. Based on the actual target state and counterfactual target state estimation Calculate counterfactual rewards based on optimal subpattern assignment And splice together the sound source status Counterfactual AUV status Counterfactual target state estimation Constituting observation ,Will( Store in the counterfactual experience pool

[0054] The process of updating the neural network parameters of the measurement generation model using actual measurement data specifically involves:

[0055] Using the sound source state, AUV state, and actual target state as inputs, the measurement generation model predicts the target measurement uncertainty. Based on the actual arrival time and angle, a negative log-likelihood loss function is calculated to update the measurement generation model parameters. The negative log-likelihood loss function is as follows:

[0056]

[0057] in, This represents the measurement generation model input, and the target measurement uncertainty. and Generate model output for measurement; These represent the signal arrival time and angle of arrival under ideal conditions, respectively. and These represent the actual signal arrival time and angle, respectively.

[0058] The method of updating the neural network parameters for deep reinforcement learning using real and counterfactual data from the experience pool specifically involves:

[0059] Simultaneously, the parameters of the deep reinforcement learning neural network are updated using data from both the real experience pool and the counterfactual experience pool, with different weights assigned to each. The weight of the real experience pool data is 1, while the weight of the counterfactual experience pool data is calculated based on the prediction accuracy of the measurement-generative model, using the following formula:

[0060]

[0061] in, and These represent the actual arrival time and angle of arrival, respectively. and These represent the arrival time and angle under ideal conditions. and These are the standard deviations of the predicted time of arrival and the standard deviation of the angle of arrival, respectively. To measure the network input of the generative model.

[0062] The step of using the output of the trained policy network as the action policy for the AUV is as follows:

[0063] The source state, AUV state, and target state estimates are input into the policy network, and the AUV executes the policy network's action output to improve the tracking performance of underwater targets.

[0064] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned AUV causal reinforcement learning decision-making method for bistatic sonar underwater target tracking.

[0065] The present invention has the following beneficial effects and advantages:

[0066] 1. The target tracking process is described based on causal graph models and structural causal models, effectively identifying and quantifying the causal relationships between variables.

[0067] 2. A measurement generation model based on heteroscedastic neural networks and binary classification networks characterizes the uncertainty of target measurement signals and generates effective counterfactual samples.

[0068] 3. The AUV decision execution end-to-end policy output based on causal reinforcement learning method has high computational efficiency and decision performance.

[0069] 4. AUV decision-making based on causal reinforcement learning methods has high sample efficiency during neural network training. Attached Figure Description

[0070] Figure 1 This is a schematic diagram of the scenario of the present invention;

[0071] Figure 2 This is a flowchart of the causal reinforcement learning training process of the present invention;

[0072] Figure 3 This is a network structure diagram of the measurement generation model of the present invention. Detailed Implementation

[0073] The present invention will be further described in detail below with reference to embodiments. The method steps will be described with reference to the accompanying drawings.

[0074] This invention discloses an AUV causal reinforcement learning decision-making method for bistatic sonar underwater target tracking, comprising: firstly, constructing a causal model describing the target tracking problem; and then establishing a measurement generation model based on a heteroscedastic neural network and a binary classification network. Based on this, a deep reinforcement learning framework is used to train the AUV's real-time maneuvering strategy, and a counterfactual reasoning mechanism is used to generate high-fidelity training data to optimize the learning process. This invention effectively solves the problems of insufficient tracking accuracy and low sample efficiency in traditional decision-making methods, significantly improving the tracking accuracy and tracking time of AUVs for underwater targets, while greatly reducing the data requirements for real training samples.

[0075] A cause-effect reinforcement learning decision-making method for AUVs in bistatic sonar underwater target tracking includes the following steps:

[0076] (1) The target tracking problem is formally described based on the causal graph model;

[0077] (2) Representing the causal relationships among variables in the target tracking problem based on a structural causal model;

[0078] (3) Establish a measurement generation model based on neural networks to generate counterfactual data;

[0079] (4) Establish a policy network and an evaluation network for deep reinforcement learning based on neural networks, wherein the policy network generates the actions of the AUV and the evaluation network estimates the state-action values;

[0080] (5) Generate AUV actions and interact with the environment using a policy network based on deep reinforcement learning. At the same time, generate counterfactual data based on a measurement generation model and store the real interaction data and counterfactual data in the experience pool respectively.

[0081] (6) Update the neural network parameters of the measurement generation model using actual measurement data;

[0082] (7) Update the parameters of the deep reinforcement learning neural network using real data and counterfactual data from the experience pool;

[0083] (8) After the deep reinforcement learning training is completed, the trained policy network is used as the action policy of the AUV.

[0084] The formal description of the target tracking problem based on the causal graph model is as follows:

[0085] Each variable is represented as a node, and the relationships between variables are represented as directed edges. Through thorough analysis, a directed acyclic graph is formed.

[0086] The structural causal model represents the causal relationships between variables in the target tracking problem, specifically as follows:

[0087] The variables are divided into endogenous variables and exogenous variables, where endogenous variables are functions of exogenous variables and other endogenous variables, and exogenous variables are random noise.

[0088] The measurement generation model based on neural networks is specifically as follows:

[0089] Considering that the uncertainty of target measurement varies with the geometric relationship of bistatic sonar, a measurement generation model is established based on heteroscedastic neural network and binary classification network.

[0090] The method of establishing a policy network and evaluation network for deep reinforcement learning using neural networks specifically includes:

[0091] A fully connected neural network is used to implement the policy network and the evaluation network. The input of the policy network is the source state, AUV state, and target state estimate, and the output is the AUV action. The input of the evaluation network is the source state, AUV state, target state estimate, and AUV action, and the output is the state-action estimate.

[0092] A deep reinforcement learning policy network is used to generate AUV actions and interactions with the environment. Simultaneously, counterfactual data is generated based on a measurement-generative model. Real interaction data and counterfactual data are stored separately in an experience pool. Specifically:

[0093] (1) In the environment, the deep reinforcement learning policy network responds to observations Generate AUV action AUV performs actions To obtain the observation at the next moment and rewards ,Will( (1) Store the data in the real experience pool; store the real target state, mixed target measurements, and all clutter measurements in the data pool; (2) Take a set of experiences from the real experience pool. ), and retrieve from the data pool All measurements at any given time are used to find the target tracking algorithm's data correlation. Exogenous variables at any given time, i.e., clutter; generating counterfactual actions using a policy network. By combining the identified clutter, counterfactual AUV state and counterfactual target state estimates are calculated based on causal models, measurement-generated models, and target tracking algorithms, and counterfactual rewards are calculated. The splicing of the sound source state, counterfactual AUV state, and counterfactual target state estimates is used for observation. ,Will( Store it in the counterfactual experience pool.

[0094] The process of updating the neural network parameters of the measurement generation model using actual measurement data specifically involves:

[0095] The sound source state, AUV state, and real target state are used as inputs to the neural network. The neural network predicts the target measurement uncertainty, calculates the negative log-likelihood loss function based on the actual measurement, and updates the neural network parameters.

[0096] The method of updating the neural network parameters for deep reinforcement learning using real and counterfactual data from the experience pool specifically involves:

[0097] Simultaneously, the network parameters are updated using data from the real experience pool and the counterfactual experience pool, and different weights are assigned to the data from the real experience pool and the counterfactual experience pool. The weight of the data from the real experience pool is 1, and the weight of the data from the counterfactual experience pool is calculated based on the prediction accuracy of the measurement generative model.

[0098] The process of using the trained policy network as the action policy for the AUV specifically involves:

[0099] The source state, AUV state, and target state estimates are input into the policy network, and the AUV executes the policy network's action output to improve the tracking performance of underwater targets.

[0100] The basic idea of ​​this invention is to carry out AUV decision-making based on deep reinforcement learning methods, and to expand the sample by using counterfactual reasoning to reduce the amount of real samples required for neural network training.

[0101] The scenarios of the present invention are as follows Figure 1As shown, a bistatic sonar system, consisting of a sound source mounted on a fixed buoy and a linear array mounted on a mobile AUV, is used to track underwater targets. The AUV causal reinforcement learning decision-making method for bistatic sonar underwater target tracking of this invention includes the following steps:

[0102] Step 1: Formalize the target tracking problem using a causal graph model. The variables in this problem include source state, AUV state, true target state, AUV action, target measurement, target state estimation, clutter, and reward. Each variable is represented as a node, and the relationships between variables are represented as directed edges, forming a directed acyclic graph.

[0103] Step 2: Represent the causal relationships among variables in the target tracking problem based on a structural causal model. Variables are divided into endogenous and exogenous variables, where endogenous variables are functions of exogenous variables and other endogenous variables, and exogenous variables are random noise. Specifically, clutter is an exogenous variable, and the other variables are endogenous variables. In particular, The true target state at any given moment The structure assignment is as follows:

[0104]

[0105] in Indicates location, Represents velocity, function for:

[0106]

[0107] in The sampling interval is denoted as .

[0108] AUV status The structure assignment is as follows:

[0109]

[0110] in For decision-making actions, The calculation method for each component is as follows:

[0111]

[0112] in For the decision time interval, For AUV speed, For the AUV's heading angle, This refers to the change in the AUV's heading angle, i.e., the decision-making action.

[0113] The target measurement structure is assigned as follows:

[0114]

[0115] in This refers to independent noise that follows a standard Gaussian distribution. A detailed explanation follows. The implementation of.

[0116] Given the state of the sound source Real target state and AUV status ideal signal arrival time and the angle of arrival for:

[0117]

[0118]

[0119] in and The distances from the sound source to the target and from the target to the AUV are the following: The speed of sound underwater. The actual signal arrival time is subject to environmental interference. and the angle of arrival for:

[0120]

[0121]

[0122] in Indicates a Gaussian distribution. and The standard deviation depends on the sampling results. ,Right now , .

[0123] Finally, target measurement The calculation method is as follows:

[0124]

[0125] in Indicates the measurement of the target location. The distance between the target and the AUV is calculated based on bistatic geometry and the actual signal arrival time and angle. The calculation method is as follows:

[0126]

[0127] in and The calculation method is as follows:

[0128]

[0129]

[0130] Target state estimation The structure assignment is as follows:

[0131]

[0132] in This invention addresses clutter. It employs a target tracking algorithm based on belief propagation. .

[0133] Step 3: Establish a measurement generation model based on a neural network. Considering the uncertainty of target measurement varies with the geometric relationship of the bistatic sonar, this invention establishes a measurement generation model based on a heteroscedastic neural network and a binary classification network. The structure of the measurement generation model is as follows: Figure 3 As shown, the binary classification network is responsible for identifying whether the target measurement exists, and the heteroscedastic neural network predicts the standard deviation of the signal arrival time and angle. This indicates element-wise multiplication. The hidden layer size of the heteroscedastic neural network is (512, 512, 256), and the hidden layer size of the binary classification network is (256, 128, 128).

[0134] Step 4: Utilize neural networks to establish a deep reinforcement learning policy network and an evaluation network. The policy network generates the AUV's actions, and the evaluation network estimates the state-action values. Both the policy and evaluation networks are implemented using fully connected neural networks. The policy network takes the source state, AUV state, and target state estimate as input, and outputs the AUV actions. It consists of two hidden layers, each with 128 nodes. The evaluation network takes the source state, AUV state, target state estimate, and AUV actions as input, and outputs the state-action estimates. It also consists of two hidden layers, each with 128 nodes. Both the policy and evaluation networks use the ReLU activation function and the Adam optimizer, with network parameters orthogonally initialized.

[0135] Step 5: Generate AUV actions and interact with the environment using a deep reinforcement learning policy network. At the same time, generate counterfactual data based on the measurement generation model. Store the real interaction data and counterfactual data in the experience database respectively.

[0136] In this embodiment, as Figure 2 As shown, specifically:

[0137] (1) In the environment, the deep reinforcement learning policy network responds to observations Generate AUV action AUV performs actions To obtain the observation at the next moment and rewards ,Will( The data is stored in the real experience pool, where observations are used to represent the source state, AUV state, and target state estimates, and AUV actions are used to represent the AUV heading angle change; the real target state, as well as all measurements of mixed targets and clutter, are stored in the data pool.

[0138] (2) Take a set of experiences from the real experience pool ( );from Extract the target state estimate from and retrieve from the data pool All measurements at any given time are used to find the target tracking algorithm's data correlation. The exogenous variable at time, i.e., clutter; Input policy network to generate counterfactual actions Counterfactual actions Right now Substitute AUV status assignment, and combine AUV status in The counterfactual AUV state is obtained. According to the counterfactual AUV status , sound source status The actual target state in the data pool The arrival time and angle of the target signal are predicted by measurement-generated models, and the target position is calculated based on the target measurement calculation method. Combined with the identified clutter, a counterfactual target state estimate is obtained through a target tracking algorithm. Based on the actual target state and counterfactual target state estimation Calculate counterfactual rewards based on optimal subpattern assignment And splice together the sound source status Counterfactual AUV status Counterfactual target state estimation Constituting observation ,Will( Store in the counterfactual experience pool

[0139] Step 6: Update the neural network parameters of the measurement generation model using actual measurement data. Using the sound source state, AUV state, and true target state as inputs to the neural network, the network predicts the target measurement uncertainty. Based on the actual arrival time and angle of arrival data, a negative log-likelihood loss function is calculated, and the neural network parameters are updated accordingly. The negative log-likelihood loss function is calculated as follows:

[0140]

[0141] in This indicates network input and target measurement uncertainty. and For network output, These represent the signal arrival time and angle of arrival under ideal conditions, respectively. and These represent the actual signal arrival time and angle, respectively.

[0142] Step 7: Update the parameters of the deep reinforcement learning neural network using real and counterfactual data from the experience pool. Specifically, different weights need to be assigned to the real and counterfactual data, where the weight of the real data is 1, and the weight of the counterfactual data is calculated based on the prediction accuracy of the measurement-generative model. The specific calculation method is as follows:

[0143]

[0144] in and Indicates the actual arrival time and angle. and For arrival time and angle under ideal conditions, and The standard deviation of the predicted arrival time and angle of arrival. To measure the network input of the generative model, and to maximize the efficiency of neural network training samples, multiple counterfactual experiences are generated each time, and multi-step counterfactual reasoning is performed. In this invention, 40 counterfactual experiences are generated for each step of environmental interaction, and the counterfactual reasoning step size is 5.

[0145] Step 8: Use the trained policy network as the AUV's action policy. Specifically, the estimated sound source state, AUV state, and target state are input into the policy network, and the AUV executes the policy network's action output to improve its underwater target tracking performance.

[0146] In addition to the above embodiments, the present invention may have other implementation methods. All technical solutions formed by equivalent substitution or equivalent transformation fall within the protection scope claimed by the present invention.

Claims

1. An AUV causal reinforcement learning decision-making method for bistatic sonar underwater target tracking, characterized in that... Includes the following steps: The target tracking problem is formally described based on a causal graph model; Structural causal models are used to represent the causal relationships among variables in target tracking problems. A measurement generation model based on neural networks is established to generate counterfactual data; A deep reinforcement learning policy network and evaluation network are established based on neural networks, where the policy network is used to generate the actions of the AUV and the evaluation network is used to estimate the state-action value. A deep reinforcement learning policy network is used to generate AUV actions and interact with the environment. At the same time, counterfactual data is generated based on a measurement-generative model. Real interaction data and counterfactual data are stored in the experience pool respectively. Update the neural network parameters of the measurement generation model using actual measurement data; The parameters of the deep reinforcement learning neural network are updated using real and counterfactual data from the experience pool; After deep reinforcement learning training is completed, the output of the trained policy network is used as the action policy of AUV.

2. The AUV causal reinforcement learning decision-making method for bistatic sonar underwater target tracking according to claim 1, characterized in that, The formal description of the target tracking problem based on the causal graph model is as follows: Each variable is represented as a node, and the relationships between variables are represented as directed edges, forming a directed acyclic graph; the variables include source state, AUV state, true target state, AUV action, target measurement, target state estimation, clutter, and reward.

3. The AUV causal reinforcement learning decision-making method for bistatic sonar underwater target tracking according to claim 1, characterized in that, The structural causal model represents the causal relationships between variables in the target tracking problem, specifically as follows: ① The true target state at any given moment The structure assignment is as follows: ; in, Indicates location, Represents velocity, function for: ； in, The sampling interval; ②AUV status The structure assignment is as follows: ; in For decision-making actions, The calculation method for each component is as follows: ； in For the decision time interval, For AUV speed, For the AUV's heading angle, This refers to the change in the AUV's heading angle, i.e., the decision-making action. ③The structural assignment for target measurement is as follows: ; in, For independent noise that follows a standard Gaussian distribution, The specific implementation is as follows: Given the state of the sound source Real target state and AUV status ideal signal arrival time and the angle of arrival for: ； ； in and These represent the distances from the sound source to the target and from the target to the AUV, respectively. The speed of sound underwater; due to environmental interference, the actual signal arrival time is... and the angle of arrival for: ； ； in Indicates a Gaussian distribution. and The standard deviation depends on the sampling results. ,Right now , ; Finally, target measurement The calculation method is as follows: ； in Indicates the measurement of the target location. The distance between the target and the AUV is calculated based on bistatic geometry and the actual signal arrival time and angle. The calculation method is as follows: ； in, and The calculation method is as follows: ； ； ④ Target state estimation The structure assignment is as follows: ; in For clutter, This is achieved through a target tracking algorithm based on belief propagation.

4. The AUV causal reinforcement learning decision-making method for bistatic sonar underwater target tracking according to claim 1, characterized in that, The measurement generation model based on neural networks is specifically as follows: A measurement generation model is established based on heteroscedastic neural networks and binary classification networks; The heteroscedastic neural network includes a first hidden layer, a second hidden layer, and a third hidden layer in sequence, wherein the third hidden layer is composed of two fourth hidden layers connected in parallel; the size of the first hidden layer and the second hidden layer is 512, and the size of the fourth hidden layer is 256; The binary classification network includes a fifth hidden layer, a sixth hidden layer, and a seventh hidden layer in sequence. The size of the fifth hidden layer is 256, and the sizes of the sixth and seventh hidden layers are 128. The heteroscedastic neural network is used to predict the standard deviation of the signal arrival time and angle of arrival, and the binary classification network is used to identify whether the target measurement exists. The output of the heteroscedastic neural network is multiplied by the output of the binary classification network to obtain the output of the measurement generation model, including the standard deviation of the signal arrival time. and the standard deviation of the signal arrival angle .

5. The AUV causal reinforcement learning decision-making method for bistatic sonar underwater target tracking according to claim 1, characterized in that, The policy network and evaluation network for deep reinforcement learning based on neural networks are specifically as follows: A fully connected neural network is used to implement the policy network and the evaluation network; the input of the policy network is the sound source state. AUV status and target state estimation The output is the AUV action; the input to the evaluation network is the sound source state. AUV status Target state estimation The AUV action outputs a state-action estimate.

6. The AUV causal reinforcement learning decision-making method for bistatic sonar underwater target tracking according to claim 1, characterized in that, The process involves using a deep reinforcement learning policy network to generate AUV actions and interactions with the environment, while simultaneously generating counterfactual data based on a measurement-generative model. The real interaction data and counterfactual data are then stored separately in an experience pool. Specifically: (1) In the environment, the deep reinforcement learning policy network responds to observations Generate AUV action AUV performs actions To obtain the observation at the next moment and rewards ,Will( The data is stored in the real experience pool, where observations are used to represent the source state, AUV state, and target state estimates, and AUV actions are used to represent the AUV heading angle change; the real target state, as well as all measurements of mixed targets and clutter, are stored in the data pool. (2) Take a set of experiences from the real experience pool ( );from Extract the target state estimate from and retrieve from the data pool All measurements at any given time are used to find the target tracking algorithm's data correlation. The exogenous variable at time, i.e., clutter; Input policy network to generate counterfactual actions Counterfactual actions Right now Substitute AUV status assignment, and combine AUV status in The counterfactual AUV state is obtained. According to the counterfactual AUV status , sound source status The actual target state in the data pool The arrival time and angle of the target signal are predicted by measurement-generated models, and the target position is calculated based on the target measurement calculation method. Combined with the identified clutter, a counterfactual target state estimate is obtained through a target tracking algorithm. Based on the actual target state and counterfactual target state estimation Calculate counterfactual rewards based on optimal subpattern assignment And splice together the sound source status Counterfactual AUV status Counterfactual target state estimation Constituting observation ,Will( Store it in the counterfactual experience pool.

7. The AUV causal reinforcement learning decision-making method for bistatic sonar underwater target tracking according to claim 1, characterized in that, The process of updating the neural network parameters of the measurement generation model using actual measurement data specifically involves: Using the sound source state, AUV state, and actual target state as inputs, the measurement generation model predicts the target measurement uncertainty. Based on the actual arrival time and angle, a negative log-likelihood loss function is calculated to update the measurement generation model parameters. The negative log-likelihood loss function is as follows: ； in, This represents the measurement generation model input, and the target measurement uncertainty. and Generate model output for measurement; These represent the signal arrival time and angle of arrival under ideal conditions, respectively. and These represent the actual signal arrival time and angle, respectively.

8. The AUV causal reinforcement learning decision-making method for bistatic sonar underwater target tracking according to claim 1, characterized in that, The method of updating the neural network parameters for deep reinforcement learning using real and counterfactual data from the experience pool specifically involves: Simultaneously, the parameters of the deep reinforcement learning neural network are updated using data from both the real experience pool and the counterfactual experience pool, with different weights assigned to each. The weight of the real experience pool data is 1, while the weight of the counterfactual experience pool data is calculated based on the prediction accuracy of the measurement-generative model, using the following formula: ； in, and These represent the actual arrival time and angle of arrival, respectively. and These represent the arrival time and angle under ideal conditions. and These are the standard deviations of the predicted time of arrival and the standard deviation of the angle of arrival, respectively. To measure the network input of the generative model.

9. The AUV causal reinforcement learning decision-making method for bistatic sonar underwater target tracking according to claim 1, characterized in that, The step of using the output of the trained policy network as the action policy for the AUV is as follows: The source state, AUV state, and target state estimates are input into the policy network, and the AUV executes the policy network's action output to improve the tracking performance of underwater targets.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the AUV causal reinforcement learning decision-making method for bistatic sonar underwater target tracking as described in any one of claims 1-9.

Images