Underwater target tracking method based on spatial-temporal feature fusion

Abstract

The invention belongs to the technical field of computer vision and target tracking, and particularly relates to an underwater target tracking method based on spatial-temporal feature fusion, which comprises the following steps: S1, constructing a feature extraction network, and performing feature extraction on a template frame and an observation frame to obtain spatial features of the template frame and spatial features of the observation frame; s2, a time sequence modeling module is constructed, a long short-term memory (LSTM) network is used for modeling target spatial features in continuous frames, and time sequence features of a target are extracted; and S3, constructing a spatio-temporal feature fusion module, designing an adaptive gating mechanism, and dynamically fusing the spatial features of the observation frame in the S1 and the time sequence features in the S2 to obtain fused spatio-temporal fusion features. On the premise of not sacrificing real-time performance, an overall solution for improving space perception, time sequence memory and adaptive fusion is provided for typical difficulties such as underwater degradation, changeable appearance and shielding, and robustness, accuracy and deployability of underwater target tracking are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of computer vision and target tracking technology, specifically to an underwater target tracking method based on spatiotemporal feature fusion. Background Technology

[0002] Underwater target tracking is one of the core tasks in computer vision. In real-world marine scenes, it is often affected by factors such as rapid attenuation and color shift of illumination, water scattering and suspended particles, non-rigid deformation of the target, partial and complete occlusion, camera shake, and changes in viewing angle. These factors lead to unstable target appearance, low contrast, weak texture, and high noise, making localization and association prone to drift. Existing technologies can be broadly divided into two categories: traditional methods and deep learning methods.

[0003] Traditional methods include: state estimation based on filtering (such as various Kalman and unscented Kalman variants), which has low computational cost and is stable under noise, but is highly dependent on a pre-defined motion model and is difficult to handle sudden motion and strong deformation; feature matching based (such as KLT, improved TLD, and ASMS), which has good real-time performance, but is prone to failure in low-contrast and occluded scenes; and region detection and association based (such as DPM combined with multi-kernel representation and mean shift), which can handle scale changes and multiple targets, but the detection accuracy decreases in low-contrast and strong scattering backgrounds.

[0004] In terms of deep learning methods, Siamese networks (such as SiamFC, SiamRPN, and their anchorless improvements) achieve high frame rate online matching through template-search similarity learning, but are sensitive to long-term appearance changes and template aging. Reinforcement learning methods model tracking as sequential decision-making, possessing adaptive policy capabilities, but training is complex and inference latency is limited. Meta-learning approaches improve small-sample and fast adaptive capabilities, but rely on high-quality metadata. The attention mechanism Transformer significantly enhances semantic and contextual representations through long-range dependency modeling, but still faces an efficiency-robust tradeoff on complex underwater degradation and resource-constrained platforms. Overall, existing methods generally suffer from an overemphasis on space and a neglect of time: much work focuses on single-frame appearance modeling while weakening the dynamic constraints of the temporal dimension; at the same time, in underwater scenarios with low light and strong scattering, relying solely on appearance templates and local features is prone to drift and false detections. Summary of the Invention

[0005] (a) Technical problems to be solved

[0006] To address the shortcomings of existing technologies, this invention provides an underwater target tracking method based on spatiotemporal feature fusion, which solves the problems mentioned in the background section.

[0007] (II) Technical Solution

[0008] To achieve the above objectives, the present invention specifically adopts the following technical solution:

[0009] An underwater target tracking method based on spatiotemporal feature fusion includes the following steps:

[0010] S1. Construct a feature extraction network to extract features from the template frame and the observed frame, and obtain the spatial features of the template frame. Spatial features of observation frames ;

[0011] S2. Construct a temporal modeling module, using a Long Short-Term Memory (LSTM) network to model the spatial features of the target in consecutive frames, and extract the temporal features of the target. (Historical frame information);

[0012] S3. Construct a spatiotemporal feature fusion module and design an adaptive gating mechanism (AGF) to dynamically fuse the spatial features of the observation frames in S1. Temporal features in S2 The spatiotemporal fusion characteristics after fusion are obtained. ;

[0013] S4. Construct a prediction network, inputting the spatial features of the template frame from S1. And the spatiotemporal fusion features in S3 The SiamRPN is used to classify targets and regress bounding boxes.

[0014] S5. Construct a bi-branch loss function for the prediction network and use the bi-branch loss function to iteratively train and optimize the underwater target tracking model.

[0015] Furthermore, the feature extraction network constructed in S1 is an improved ResNet-50 network with shared parameters in both branches. The improvements specifically include:

[0016] S11. Replace the regular convolutional layers in the ResNet-50 network with depthwise separable convolutional layers;

[0017] S12. Remove the conv5 layer and fully connected layer of the Resnet-50 network, use the conv4 layer as the feature extraction output layer and adjust its stride to 1, and introduce dilated convolution to expand the receptive field.

[0018] S13. Add an information interaction module after the conv4 layer to realize global dependency modeling of enhanced features.

[0019] Furthermore, the information interaction module in S13 generates attention weights by calculating the similarity between the query vector Q, the key vector K, and the value vector V, and performs weighted fusion of the features. Its output serves as enhanced spatial features provided to the subsequent temporal modeling network and prediction network. Specifically, this includes:

[0020] S131. First, the feature extraction output layer in S12 is mapped to the query vector through a convolution operation. Key vector Sum value vector Its formula can be expressed as:

[0021]

[0022] in For the input feature map, , , It is the weight matrix for learning. , , These represent the query vector, key vector, and value vector, respectively.

[0023] S132. Calculate the similarity between the query vector and the key vector. The similarity is measured by the inner product, and the formula can be expressed as:

[0024]

[0025] in It is a location and location Similarity between them This is a standardization operation performed to stabilize the gradient. It is a location The query value, It is a location key-value pairs, Indicates the transpose operation;

[0026] S133. Perform the calculation on the similarity. Normalization yields the weights between each pair of positions, which can be expressed by the formula:

[0027]

[0028] in It is a location and location Attention weights between them It is a location and location Similarity between them This is represented as a normalization operation;

[0029] S134. Use the obtained attention weights to perform a weighted summation on the value vector to obtain a new feature vector for each position. The formula can be expressed as:

[0030]

[0031] in It is a location New feature vectors It is attention weight. It is a location The value vector, It is a summation operation; finally, the weighted feature vectors of all positions are combined to generate the enhanced feature map.

[0032] Furthermore, the temporal modeling module in S2 has a two-layer LSTM structure, where each LSTM module has the same structure, and the processing includes the following steps:

[0033] S21. Spatial features of the observation frame obtained through ROI Align Extract target features, which are derived from the target's spatial location in historical frames;

[0034] S22, The first LSTM layer is used to filter out inter-frame noise and extract primary temporal dynamic features;

[0035] S23. The second LSTM layer is used to abstract high-order motion features, including target velocity changes and occlusion recovery trends, to obtain temporal features with high information density and small fluctuations.

[0036] S24. Output the hidden state h at the end of the second layer. t (1) As a temporal feature .

[0037] Furthermore, the specific structure of the LSTM utilizes three types of gates: a forget gate, an input gate, and an output gate, and is implemented through the following steps:

[0038] S221, The Forgot Gate outputs the previous hidden state. and current input Perform a non-linear mapping to the Sigmoid function and then output a vector. Determines the long-term state of the previous moment. How many will remain in the long-term state at the current moment? Its formula can be expressed as:

[0039]

[0040] in, It is a weight matrix, which corresponds to the input terms. Its dimensions are , It is also a weight matrix, which corresponds to the input terms. Its dimensions are , This means concatenating two vectors into a longer vector. It is the bias term of the forgetting gate. It is the Sigmoid function. For input dimensions, For the dimension of the hidden layer, The dimension of the unit state;

[0041] S222, The input gate outputs by reading the previous hidden state. and current input The input to the network at the current time is determined by the Sigmoid layer and the tanh layer. How many are saved to long-term state? The Sigmoid layer determines which features to update, and its formula can be expressed as:

[0042]

[0043] in, This is the weight matrix. It is the Sigmoid function. It is a bias term;

[0044] The tanh layer will create a new candidate state. Add to long-term state In this context, the formula can be expressed as:

[0045]

[0046] in, This is the weight matrix. It is a bias term. It is the tanh function;

[0047] S223, Update long-term state Through the long-term state of the previous moment Output of the forget gate Determine the information features that need to be discarded, and then use the candidate states. The output of the input gate The formula for selecting the important information features to be retained can be expressed as:

[0048]

[0049] S224, The output gate outputs by reading the previous hidden state. and current input Controlling long-term states How many will be output to the hidden state of the LSTM? Its formula can be expressed as:

[0050]

[0051]

[0052] in, This is the weight matrix. It is a bias term. It is the tanh function. It is the Sigmoid function.

[0053] Furthermore, the specific steps of S3 are as follows:

[0054] S31. First, the input spatiotemporal features are aligned and then concatenated. Then, a 1×1 convolutional layer is used to map the features and generate fusion weights. Finally, dynamic weights are generated using the Sigmoid function. The range is [0, 1], and its formula can be expressed as:

[0055]

[0056] in, The temporal characteristics at the current moment, To observe the spatial characteristics of the current frame, and For learnable parameters, For the Sigmoid function;

[0057] S32. During the fusion process, temporal and spatial features are weighted and summed through an adaptive gating mechanism (AGF) to obtain the final spatiotemporal fused features. Its formula can be expressed as:

[0058]

[0059] in, The dynamic weights are calculated using an adaptive gating mechanism. and These are temporal features and spatial features, The spatiotemporal fusion characteristics of the final fusion.

[0060] Furthermore, the specific steps of S4 are as follows:

[0061] S41, Regression Branch: First, the spatial features of the template frame... Features of spatiotemporal integration Perform cross-correlation operation to generate response feature map, and then perform coordinate regression on the response feature map through 4 layers of 1×1 convolution. Each layer outputs 4 parameters of the corresponding bounding box, namely the center point coordinates and width and height.

[0062] S42, Classification Branch: The input and cross-correlation operation of this branch are the same as those in S41. Then, k anchor points are preset at each position of the response feature map to generate corresponding candidate boxes. Then, the foreground / background confidence score of each anchor point is output through two layers of 1×1 convolution.

[0063] Furthermore, the dual-branch loss function in S5 includes a regression branch loss function and a classification branch loss function, wherein the regression branch loss function... It can be represented as:

[0064]

[0065] in This represents an indicator function, where 1 represents a positive sample and 0 represents a negative sample. The number of positive samples. Represented as the first Predicted bounding boxes at each location, Represented as with the first The ground truth boxes that match a positive sample. Represented as generalized IoU loss, Represented as L1 loss, and Represents the regularization parameter;

[0066] The classification branch loss function It can be represented as:

[0067]

[0068] in Indicates the first The label for each sample indicates whether it is foreground or background; 1 indicates foreground and 0 indicates background. This indicates the probability that the model predicts the sample to be the foreground. The total number of samples.

[0069] (III) Beneficial Effects

[0070] Compared with existing technologies, this invention provides an underwater target tracking method based on spatiotemporal feature fusion, which has the following advantages:

[0071] By structurally improving the backbone network, including conv4 output, stride adjustment and dilated convolution, and depthwise separable convolution to reduce computational power, and introducing an information interaction module to achieve globally dependent attention-based information interaction, background noise is suppressed and target response is enhanced in low-contrast and strong scattering backgrounds. Under the same computational power conditions, resolution and receptive field are balanced, parameters and computational load are reduced, and conditions are created for real-time deployment.

[0072] A two-layer LSTM is used to perform hierarchical dynamic modeling of the short-window history (i.e., the N=5 ROI historical feature sequence): the first layer is used for denoising and fine-grained dynamic extraction, and the second layer abstracts higher-order motion modes such as speed and occlusion recovery. When the target experiences brief mismatch, abrupt appearance changes, or occlusion, the model can rely on temporal memory to perform time-based frame interpolation, significantly reducing drift and loss rates.

[0073] By continuously adjusting the spatial and temporal emphasis within the [0,1] interval using learnable gating weights, the spatial and temporal emphases are adjusted: for stable scenes, spatial details are emphasized, while for scenes with violent motion and occlusion, temporal priors are enhanced. This ensures that the fusion results retain clear localization boundaries while also possessing continuity and consistency constraints, thereby improving robustness and generalization ability in complex environments.

[0074] By cross-correlation and dense prediction of template spatial features and spatiotemporal augmentation features of observation frames in SiamRPN, and by unifying the classification and regression dual-branch loss functions in the end-to-end framework, the model performance can be optimized, which helps to improve both foreground discriminativeness and bounding box localization accuracy. Attached Figure Description

[0075] Figure 1 This is a schematic diagram of the overall process of the present invention;

[0076] Figure 2 This is a schematic diagram of the hierarchical structure of the modified Resnet-50 network in this invention;

[0077] Figure 3 This is a schematic diagram of the information interaction module structure of the feature extraction network in this invention;

[0078] Figure 4 This is a schematic diagram of the two-layer LSTM structure of the timing modeling module in this invention;

[0079] Figure 5 This is a schematic diagram of the three gate structures of the LSTM in the timing modeling module of this invention. Detailed Implementation

[0080] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0081] Example

[0082] like Figure 1-5 As shown in the figure, an underwater target tracking method based on spatiotemporal feature fusion proposed in one embodiment of the present invention generally proceeds as follows: First, ResNet-50 is used as a Siamese backbone network to extract features from template frames and observation frames, and an information interaction module is introduced after the conv4 stage to suppress underwater background noise and enhance the expression of target features; then, in the spatiotemporal feature fusion stage, the spatial features of the extracted observation frames are... The target region features are obtained through ROI Align and input into an LSTM to generate a temporal feature vector. This is to capture the target's dynamic trajectory and appearance evolution. Next, we will... and By combining spatial broadcasting with stitching operations, and utilizing the Adaptive Gating (AGF) mechanism to generate adaptive weights for different regions, a balanced fusion of spatial details and temporal dynamics is achieved, resulting in the final spatiotemporal fusion features. Finally, based on SiamRPN, the target classification and bounding box regression were completed.

[0083] An underwater target tracking method based on spatiotemporal feature fusion includes the following steps:

[0084] S1. Construct a feature extraction network to extract features from the template frame and the observed frame, and obtain the spatial features of the template frame. Spatial features of observation frames The feature extraction network constructed is an improved ResNet-50 network, and its parameter hierarchy information is as follows: Figure 2 As shown, the feature extraction network features parameter sharing in its two branches, and its improvements specifically include:

[0085] S11. Replace the regular convolutional layers in the ResNet-50 network with depthwise separable convolutional layers;

[0086] S12. Remove the conv5 layer and fully connected layer of the Resnet-50 network, use the conv4 layer as the feature extraction output layer and adjust its stride to 1, and introduce dilated convolution to expand the receptive field.

[0087] S13. Add an information interaction module after the conv4 layer to realize global dependency modeling of enhanced features.

[0088] like Figure 3 As shown, the information flow of traditional convolutional neural networks only propagates within a local range, while the information interaction module is based on a global information interaction mechanism. By calculating the similarity between various locations in the image and weighting the sum, the features of each location can interact with the features of other locations, thereby enhancing the model's ability to model long-range dependencies and global context.

[0089] The information interaction module generates attention weights by calculating the similarity between the query vector Q, key vector K, and value vector V, and then performs weighted fusion of features. Its output serves as enhanced spatial features provided to the subsequent temporal modeling and prediction networks. Specific implementations include:

[0090] S131. First, the feature extraction output layer of S12 is mapped to the query vector through a convolution operation. Key vector Sum value vector Its formula can be expressed as:

[0091]

[0092] in For the input feature map, , , It is the weight matrix for learning. , , These represent the query vector, key vector, and value vector, respectively.

[0093] S132. Calculate the similarity between the query vector and the key vector. The similarity is measured by the inner product, and the formula can be expressed as:

[0094]

[0095] in It is a location and location Similarity between them This is a standardization operation performed to stabilize the gradient. It is a location The query value, It is a location key-value pairs, Indicates the transpose operation;

[0096] S133. Perform the calculation on the similarity. Normalization yields the weights between each pair of positions, which can be expressed by the formula:

[0097]

[0098] in It is a location and location Attention weights between them It is a location and location Similarity between them This is represented as a normalization operation;

[0099] S134. Use the obtained attention weights to perform a weighted summation on the value vector to obtain a new feature vector for each position. The formula can be expressed as:

[0100]

[0101] in It is a location New feature vectors It is attention weight. It is a location The value vector, It is a summation operation; finally, the weighted feature vectors of all positions are combined to generate the enhanced feature map.

[0102] S2. Construct a temporal modeling module, using a Long Short-Term Memory (LSTM) network to model the spatial features of the target in consecutive frames, and extract the temporal features of the target. (Historical frame information) compensates for the vulnerability of pure spatial features to rapid target deformation, scale drift and occlusion. Its structure is a two-layer LSTM structure, and the structure of each LSTM module is the same.

[0103] LSTM is a recurrent neural network specifically designed for modeling time-series data, effectively handling long-term dependency problems. Its core mechanism involves selectively retaining and updating information through gating units, thereby maintaining the information flow over long periods. For example... Figure 4 As shown, each blue module represents an LSTM module. Compared to a single-layer LSTM, the stacked design can provide hierarchical temporal abstraction and stronger nonlinear modeling capabilities without increasing the time window. The specific processing steps include:

[0104] S21. Spatial features of the observation frame obtained through ROI Align Extract the target feature vector sequence, which comes from the target's spatial location in historical frames (N=5);

[0105] A three-stage processing strategy is adopted for the input of the time series modeling module:

[0106] Phase 1 is the first frame of the initialization phase. Since there are no historical frames at this time, LSTM is not enabled. Spatial features are used directly to track and predict the target, and the current target features are stored in the feature vector sequence.

[0107] Phase 2 consists of frames 2 to 4. Although there are historical frames at this point, the number is insufficient. The historical frame count is padded to 5 frames by zeros. LSTM is used to process the target feature vector sequence to output temporal features for subsequent spatiotemporal feature fusion.

[0108] Phase 3 is from the 5th frame onwards. At this point, the target feature vector sequence is complete. In subsequent frames, the oldest frame feature is removed and the current frame feature is added, keeping N=5.

[0109] S22, The first LSTM layer is used to filter out inter-frame noise and extract primary temporal dynamic features;

[0110] The specific structure of LSTM involves the use of three gates: a forget gate, an input gate, and an output gate. The structural diagrams of each gate are shown below. Figure 5 As shown, this can be achieved through the following steps:

[0111] S221, The Forgot Gate outputs the previous hidden state. and current input Perform a non-linear mapping to the Sigmoid function and then output a vector. Determines the long-term state of the previous moment. How many will remain in the long-term state at the current moment? Its formula can be expressed as:

[0112]

[0113] in, It is a weight matrix, which corresponds to the input terms. Its dimensions are , It is also a weight matrix, which corresponds to the input terms. Its dimensions are , This means concatenating two vectors into a longer vector. It is the bias term of the forgetting gate. It is the Sigmoid function. For input dimensions, For the dimension of the hidden layer, The dimension of the unit state;

[0114] S222, The input gate outputs by reading the previous hidden state. and current input The input to the network at the current time is determined by the Sigmoid layer and the tanh layer. How many are saved to long-term state? The Sigmoid layer determines which features to update, and its formula can be expressed as:

[0115]

[0116] in, This is the weight matrix. It is the Sigmoid function. It is a bias term;

[0117] The tanh layer will create a new candidate state. Joining the long-term state In this context, the formula can be expressed as:

[0118]

[0119] in, This is the weight matrix. It is a bias term. It is the tanh function;

[0120] S223, Update long-term state Through the long-term state of the previous moment Output of the forget gate Determine the information features that need to be discarded, and then use the candidate states. Output of the input gate The formula for selecting the important information features to be retained can be expressed as:

[0121]

[0122] S224, The output gate outputs by reading the previous hidden state. and current input Controlling long-term states How many will be output to the hidden state of the LSTM? Its formula can be expressed as:

[0123]

[0124]

[0125] in, This is the weight matrix. It is a bias term. It is the tanh function. It is the Sigmoid function.

[0126] S23. The second LSTM layer is used to abstract higher-order motion features, including target velocity changes and occlusion recovery trends, to obtain temporal features with higher information density and less fluctuation.

[0127] S24. Output the hidden state h at the end of the second layer. t (1) As a temporal feature .

[0128] S3. Construct a spatiotemporal feature fusion module and design an adaptive gating mechanism (AGF) to dynamically fuse the spatial features of the observation frames in S1 based on the motion changes of the target. Temporal characteristics in S2 The spatiotemporal fusion characteristics after fusion are obtained. This improves the robustness and accuracy of the model in complex underwater environments.

[0129] The dynamic weights are generated through the Adaptive Gating (AGF) mechanism to achieve dynamic fusion of spatiotemporal features. The specific implementation steps are as follows:

[0130] S31. First, the input spatiotemporal features are aligned and then concatenated. Then, a 1×1 convolutional layer is used to map the features and generate fusion weights. Finally, dynamic weights are generated using the Sigmoid function. The range is [0, 1], and its formula can be expressed as:

[0131]

[0132] in, The temporal characteristics at the current moment, To observe the spatial characteristics of the current frame, and For learnable parameters, For the Sigmoid function;

[0133] S32. During the fusion process, temporal and spatial features are weighted and summed through an adaptive gating mechanism (AGF) to obtain the final spatiotemporal fused features. Its formula can be expressed as:

[0134]

[0135] in, The dynamic weights are calculated using an adaptive gating mechanism. and These are temporal features and spatial features, The spatiotemporal fusion characteristics of the final fusion.

[0136] S4. Construct a prediction network, inputting the spatial features of the template frame from S1. And the spatiotemporal fusion features in S3 The SiamRPN is used to classify targets and regress bounding boxes.

[0137] The specific implementation steps of the regression-classification dual-branch prediction network are as follows:

[0138] S41, Regression Branch: First, the spatial features of the template frame... Features of spatiotemporal integration Perform cross-correlation operation to generate response feature map, and then perform coordinate regression on the response feature map through 4 layers of 1×1 convolution. Each layer outputs 4 parameters of the corresponding bounding box, namely the center point coordinates and width and height.

[0139] S42, Classification Branch: The input and cross-correlation operation of this branch are the same as those in S41. Then, k anchor points are preset at each position of the response feature map to generate corresponding candidate boxes. Then, the foreground / background confidence score of each anchor point is output through two layers of 1×1 convolution.

[0140] S5. Construct a bi-branch loss function for the prediction network and use the bi-branch loss function to iteratively train and optimize the underwater target tracking model.

[0141] The bi-branch loss function includes a regression branch loss function and a classification branch loss function;

[0142] Regression branch loss function It can be represented as:

[0143]

[0144] in This represents an indicator function, where 1 represents a positive sample and 0 represents a negative sample. The number of positive samples. Represented as the first Predicted bounding boxes at each location, Represented as with the first The ground truth boxes that match a positive sample. Represented as generalized IoU loss, Represented as L1 loss, and Represents the regularization parameter;

[0145] Classification branch loss function It can be represented as:

[0146]

[0147] in Indicates the first The label for whether a sample is foreground (1 for foreground, 0 for background). This indicates the probability that the model predicts the sample to be the foreground. The total number of samples.

[0148] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for underwater target tracking based on spatio-temporal feature fusion, characterized in that, The method comprises the following steps: S1, a feature extraction network is constructed to perform feature extraction on the template frame and the observation frame to obtain spatial features of the template frame and the observation frame and the observation frame ; S2, construct a timing modeling module, model the target spatial features in the continuous frames using a long short-term memory network (LSTM), and extract the timing features of the target ; S3, construct a spatio-temporal feature fusion module, design an adaptive gating mechanism AGF, dynamically fuse the spatial features of the observation frame in S1 and the time sequence features in S2 , obtain the fused spatio-temporal fusion features ; S4, constructing a prediction network, inputting the spatial features of the template frame in S1 and the spatio-temporal fusion features in S3 , completing the classification of the target and the regression of the bounding box through SiamRPN; S5, a double-branch loss function of the prediction network is constructed, and the underwater target tracking model is iteratively trained and optimized by using the double-branch loss function. 2.The method of claim 1, wherein: The feature extraction network constructed in S1 is an improved Resnet-50 network, and the double-branch parameters are shared, and the improvement specifically comprises: S11, replacing the conventional convolution layer in the Resnet-50 network with a depth separable convolution layer; S12, removing the conv5 layer and the fully connected layer of the Resnet-50 network, taking the conv4 layer as the feature extraction output layer and adjusting the stride to 1, and introducing dilated convolution to expand the receptive field; S13, adding an information interaction module after the conv4 layer to realize global dependency relationship modeling of enhanced features.

3. The underwater target tracking method based on spatio-temporal feature fusion according to claim 2, characterized in that: The information interaction module in S13 calculates the similarity of the query vector Q, the key vector K and the value vector V, generates attention weights, and performs weighted fusion on the features, and the output is provided to the subsequent time sequence modeling network and the prediction network as the enhanced spatial features, and the specific implementation comprises: S131, first map the feature extraction output layer in S12 to a query vector through a convolution operation , key vector and value vector The formula can be expressed as: wherein is an input feature map, , , is a learned weight matrix, , , query, key, and value vectors, respectively; S132, calculating the similarity between the query vector and the key vector, and measuring the similarity by inner product, and the formula can be expressed as: wherein is a similarity between positions and position is a standardization operation for stabilizing the gradient, is a query value for position is a key value for position denotes a transposition operation;​​​ S133, normalizing the calculated similarity The normalization results in a weight between each pair of positions, which can be expressed by the formula: wherein is an attention weight between positions and position , is a similarity between positions and position , is represented as a normalization operation; S134, using the obtained attention weights to perform weighted summation on the value vector to obtain a new feature vector at each position, and the formula can be expressed as: wherein is a position a new feature vector, is an attention weight, is a position a value vector, is a sum operation; finally, the weighted feature vectors of all positions are combined to generate an enhanced feature map.

4. The underwater target tracking method based on spatio-temporal feature fusion according to claim 1, characterized in that: The structure of the time sequence modeling module in S2 is a double-layer LSTM structure, wherein the module structure of each LSTM is the same, and the processing process comprises: S21, extracting target features from the spatial features of the observation frame extracting target features from the spatial features of the observation frame S22, the first layer LSTM is used to filter out interframe noise and extract primary time sequence dynamic features; S23, the second layer LSTM is used to abstract high-order motion features, including target speed change and occlusion recovery trend, and obtain time sequence features with high information density and small fluctuations; S24, output the hidden state h of the second layer at the end time t (1) as a timing feature .

5. The underwater target tracking method based on spatio-temporal feature fusion according to claim 4, characterized in that: The specific structure of the LSTM is the use of three kinds of gates, namely, the forgetting gate, the input gate and the output gate, which are realized by the following steps: S221, forget gate reads the previous hidden state output and the current input , does a sigmoid non-linearity, and outputs a vector that determines how much of the previous time step's long-term state to keep in the current time step's long-term state , and the formula for that is: where is a weight matrix corresponding to the input items with dimensions , is also a weight matrix corresponding to the input items with dimensions , denotes concatenating two vectors into a longer vector, is a bias term for the forget gate, is a sigmoid function, is the input dimension, is the dimension of the hidden layer, is the dimension of the cell state. S222, the input gate outputs by reading the previous hidden state and the current input , through the Sigmoid layer and the tanh layer to determine the input of the network at the current time how much to save to the long-term state , the Sigmoid layer determines what features to update, and its formula can be expressed as: wherein, is a weight matrix, is a Sigmoid function, is a bias term; The tanh layer creates a new candidate state that is added to the long-term state with the formula: wherein, is a weight matrix, is a bias term, is a tanh function; S223, update long-term state through the long-term state of the last time with the output of the forget gate determine the information features that need to be discarded, and then select the important information features that need to be retained through the candidate state with the output of the input gate The formula can be expressed as: S224, the output gate outputs by reading the previous hidden state and the current input controls the long-term state how much to output to the LSTM's hidden state whose formula can be expressed as: wherein, is a weight matrix, is a bias term, is a tanh function, is a Sigmoid function.

6. The underwater target tracking method based on spatio-temporal feature fusion according to claim 1, characterized in that: The specific steps of S3 are as follows: S31, first, the input spatio-temporal features are aligned and then spliced, and then a 1x1 convolution layer is used to map the features to generate fusion weights , and finally a dynamic weight is generated by a Sigmoid function , the range is [0, 1], and the formula can be expressed as: wherein, is a temporal feature of the current time instant, is a spatial feature of the observation frame at the current time instant, and are learnable parameters, is a sigmoid function; S32、In the fusion process, the timing features and the spatial features are weighted and summed through an adaptive gating mechanism AGF to obtain final spatio-temporal fusion features The formula can be expressed as: wherein, is a dynamic weight calculated by an adaptive gating mechanism, and are a temporal feature and a spatial feature, respectively, is a final fused spatio-temporal fused feature.

7. The underwater target tracking method based on spatio-temporal feature fusion according to claim 1, characterized in that: The specific steps of S4 are as follows: S41, the regression branch: first, the spatial features of the template frame spatiotemporal fusion features correlation operation, generate response feature map, and then through 4 layers of 1*1 convolution, the response feature map is subjected to coordinate regression, and each layer outputs 4 parameters of the corresponding boundary box, namely the center point coordinates and the width and height; S42, classification branch: the input and cross-correlation operation of this branch are the same as S41, then k anchor points are preset at each position of the response feature map to generate corresponding candidate boxes, and then the foreground / background confidence scores of each anchor point are output through 2 layers of 1x1 convolution.

8. The underwater target tracking method based on spatio-temporal feature fusion according to claim 1, characterized in that: The double-branch loss function in S5 includes a regression branch loss function and a classification branch loss function, the regression branch loss function may be represented as: wherein represents an indicator function, 1 for positive samples and 0 for negative samples, is the number of positive samples, represents the predicted box for the th position, represents the ground truth box matched with the th positive sample, represents the generalized IoU loss, represents the L1 loss, and represents a regularization parameter; The classification branch loss function may be represented as: wherein represents the label of whether the i-th sample is foreground or not, 1 for foreground and 0 for background, represents the label of whether the i-th sample is foreground or not, 1 for foreground and 0 for background, represents the probability that the model predicts the i-th sample as foreground, is the total number of samples.

Images