An underwater target tracking method based on twin prototype hybrid network

Abstract

The present invention discloses an underwater target tracking method based on a twin prototype hybrid network. The method first obtains original underwater image data, defines the target and support set images, and performs background separation preprocessing. Secondly, an adaptive cutoff filter and a twin network are designed, a prototype network is established and it is integrated with the twin network. Then, two different prototype expressions under the same support set are introduced, and after mutual convolution, the anchor points near the target are judged and screened through an LSTM network. Finally, a regional candidate network classifier for classification and regression is constructed. Based on the anchor points near the target, the classification channel and the regression channel are output, and multi-dimensional aggregation is performed to obtain multi-category classification results and regression results. The item with the highest probability in the classification result is selected, and its corresponding regression result is output as the target position coordinate to realize category recognition and positioning. The present invention can effectively realize underwater target tracking in the presence of background interference.

Description

Technical Field

[0001] The present invention belongs to the field of underwater target tracking, and in particular relates to an underwater single target tracking method based on a twin prototype hybrid network. Background Art

[0002] As countries increasingly prioritize the development of marine resources, underwater target tracking technology has gained widespread application in deep-sea resource exploration and rare species protection. However, ocean scenarios often present challenges, such as turbulent currents and unstable acquisition platforms, which can lead to background interference in the collected data. These challenges complicate accurate tracking.

[0003] In order to improve the versatility of the model in underwater environments and improve tracking accuracy, many deep learning-based methods combined with image tracking algorithms have been applied to target tracking. As a method based on deep learning, the Siamese network is widely used in single target tracking tasks due to its excellent performance in tracking problems. SiamFC uses different search regions and template features to convolve to obtain similarity, converting the target tracking problem into a similarity matching problem between the template and the search image. On this basis, SiamRPN introduces the residual network (ResNet) and the region proposal network (RPN), using ResNet to improve the accuracy of feature expression. At the same time, the RPN locates the target with candidate boxes of different scales, avoiding the problem of target deformation during tracking and improving the accuracy of system tracking.

[0004] However, since the twin network tracker achieves target tracking by comparing the similarity between the search image and the template image, when the background of the video may change drastically, the change in visual features such as the background color will cause the background of the original template to not match the background of the search image, ultimately resulting in a reduced recognition success rate during similarity judgment. Summary of the Invention

[0005] Based on the above analysis, the present invention proposes an underwater target tracking method based on a twin prototype hybrid network to provide a high-precision and high-success-rate underwater single target tracking solution to solve the problem of poor tracking performance of the twin network tracker in underwater scenes with inconsistent background information.

[0006] The tracking solution consists of two parts: background separation of underwater images using multi-threshold mask segmentation and tracking using a twin prototype hybrid network. The steps are as follows:

[0007] S1 defines and acquires raw underwater image data, dividing it into a training set and a validation set. A twin prototype hybrid network model is then trained based on the training set, and the model is used to track and identify validation set images. First, the underwater image is defined, with the acquired image to be identified defined as the target image and the annotated images defined as the support set images. Next, the training set images are spatially uniformly sampled to expand the sample pairs. Finally, the prototype network's loss function is improved, and a variable class-interval loss function is designed to enhance the model's ability to distinguish between different classes.

[0008] S2 performs background separation preprocessing on the original underwater images. First, grayscale and contrast enhancement are performed on these images. An optimization algorithm is then used to globally optimize the segmentation threshold to obtain the optimal threshold. The image is then segmented using the optimal threshold. The separated background is masked with a weighted mask of 0.7 and then added to the original image to produce a new image without background interference.

[0009] S3 designs an adaptive cutoff filter and a Siamese network. The preprocessed target and support set images are fed into the adaptive cutoff filter to extract high-frequency signals and enhance image texture details. The target and support set images are then fed into two feature extraction networks in the Siamese network, each consisting of residual blocks with shared weights. A maximum pooling layer is added after each block, and a 1×1 convolutional layer is used to ensure that the size and number of channels between features are consistent. Finally, a multidimensional feature map is output, encompassing image features from low to high dimensions.

[0010] S4 builds a prototype network and fuses it with the twin network. The multi-dimensional feature map output by the twin network is projected into the feature space to form the support set prototype expression and the target prototype expression respectively. The minimum uncertainty weighted mapping method is designed to calculate and output the set prototype of each category.

[0011] S5 introduces two different prototype expressions under the same support set from the feature space of S4, convolves them with each other, and uses the LSTM network to determine the approximate position of the target, selecting g×g×k anchor points near the target instead of n×m×k anchor points in the entire feature map.

[0012] S6 builds a region candidate network classifier for classification and regression. The feature map uses g×g×k anchor points near the target selected by S5, and then the anchor points pass through the classification subnet and regression subnet respectively, by convolving the query features with the support features. Finally, 2k channels for classification and 4k channels for regression are output, so that multi-dimensional aggregation can obtain multi-category classification results. And the regression results

[0013] Multi-category classification results of S7 after aggregation Select the one with the highest probability and output its corresponding regression result As the target location coordinates, category recognition and positioning are achieved.

[0014] Furthermore, the specific steps of step S1 are as follows:

[0015] Said S1-1: in the training stage, spatially uniformly sample the training images and expand the sample pairs, generate sample positive pairs by uniformly sampling and cropping the training images according to the set step size, and add semantic negative sample data at the same time. The added negative samples are composed of random samples of different categories to achieve the effect of expanding the training set.

[0016] Said S1-2: Based on the Triplet loss function used in training the prototype network, the distance loss function is improved, and the fixed threshold of the difference between the distance between the query data and the negative sample and the distance between the query data and the positive sample is changed to adaptive. The designed variable category distance loss function is shown in Formula 1.

[0017]

[0018] in are the distances between the i-th label sample and the positive sample and the negative sample, respectively. is a variable class spacing hyperparameter, and its calculation process is j≠k, m is a constant, is the sum of the squares of the distances between the positive and negative samples in the i-th label, It is the sum of the squares of the distances between the positive samples and the query samples in the i-th label.

[0019] Furthermore, in step S2, the background segmentation adopts a multi-threshold segmentation method, and the image is segmented into different categories by setting different grayscale thresholds for the image. The segmentation process is shown in Formula 2.

[0020]

[0021] In formula (2), M={M1,M2,…,M n} are different n categories, t is a pixel of the grayscale image, {th1,th2,…,th n} is the n thresholds set.

[0022] Furthermore, the specific implementation steps of step S3 are as follows:

[0023] The S3-1: Adaptive cutoff filter converts the spatial domain image into a frequency domain signal and then filters the signal according to the adaptive cutoff frequency to filter low-frequency clutter and enhance image texture details. First, the spatial domain image is converted into a frequency domain signal according to Formula 3.

[0024]

[0025] Where M(x,y) is the original spatial domain image, x and y are the coordinates of the image in the spatial domain, H and W are the height and width of the image respectively. is the signal after frequency domain conversion, u and v are the coordinates of the signal in the frequency domain, and i is the imaginary unit. Secondly, the converted signal is filtered out of low-frequency noise using a high-frequency filter, as shown in Equation 4.

[0026]

[0027] Where D(u,v) is the distance from the center of the frequency domain to the point (u,v), and D0 is the adaptive cutoff frequency. D0 is adjusted by the image gradient magnitude G(x,y), which is calculated as shown in Equation 5.

[0028]

[0029] in is the baseline cutoff frequency, and λ is the gain factor. Again, the high-frequency filter is applied to the frequency domain image signal As shown in Equation 6.

[0030]

[0031] Finally, the frequency domain signal is inversely transformed to restore the spatial domain image. The process is shown in Equation 7.

[0032]

[0033] S3-2: Each residual block consists of two 1×1 convolutional regularization layers, an activation function, and a 3×3 convolutional layer. The input is downsampled and added to the convolution result to output the features of that dimension. This invention further eliminates background effects in the target area by combining the features with an attention module after 1×1 convolutional regularization within the residual block.

[0034] Furthermore, in step S4, the minimum uncertainty weighted mapping first calculates the uncertainty of each category prototype expression to form the prototype weight, and finally weights the distance calculation to obtain the set prototype. The specific process is as follows. First, define a prior distribution with Gaussian distribution as the weight μ0 is the mean vector, For the covariance, a variational distribution q(θ) is selected to approximate the true posterior distribution p(θ|D), where D is the observed data. The present invention selects a Gaussian distribution μ is the mean vector, is the covariance, which serves as the variational distribution. Next, we define and calculate the maximum evidence lower bound (ELBO) to adjust the parameters of the variational distribution so that it is as close as possible to the posterior distribution. The calculation formula is shown in Equation 8.

[0035] ELBO=E q(θ) [log p(D|θ]-D KL (q(θ)||p(θ)) (8)

[0036] Among them E q(θ) [log p(D|θ] is the log-likelihood expectation, which represents the average degree of fit of the data under the variational distribution; D KL (q(θ)||p(θ)) is the divergence, which measures the difference between the variational distribution and the prior distribution. The ELBO is optimized by gradient ascent to maximize it, thereby determining the variational distribution q(θ), so the prototype expression output follows x i is the i-th sample, θ s is the sth parameter sample drawn from q(θ). The mean and variance of the predictions are again calculated to estimate the uncertainty, as shown in Equation 9.

[0037]

[0038] Where S is the number of sampling times, is the predicted mean, To account for forecast uncertainty, is the prototype expression of the category. The inverse of the uncertainty value is taken as the weight, as shown in Equation 10.

[0039]

[0040] Where α is a constant close to 0. The weights are normalized as shown in Equation 11.

[0041]

[0042] Where M is the number of prototype expressions. Finally, the prototype expressions are weighted to calculate the set prototype, as shown in Equation 12.

[0043]

[0044] Where S is the category and c is the set prototype of the category.

[0045] Furthermore, in S6, the region candidate network consists of a classification subnet and a regression subnet. If the image has k anchor points, 2k channels are output for classification and 4k channels are output for regression. The support branch and the query branch are convolved to form support classification features. Query classification features Support regression features Query regression features in Keep the number of channels unchanged, The number of channels increases to 2k and 4k times, respectively. Feature correlation is then calculated in each classification and regression subnet, as shown in Equation 13.

[0046]

[0047] In formula 13, A cls 、A reg are the correlations between the classification subnet and the regression subnet, respectively. Conv(x,y) represents the convolution operation. The final multi-dimensional aggregation function is shown in Equation 14.

[0048]

[0049] Where i is the number of residual block layers, α i and β i are the weighting coefficients corresponding to layer i.

[0050] Beneficial effects of the present invention:

[0051] The method proposed in the present invention can effectively track underwater targets. Among them, the multi-threshold mask segmentation preprocessing proposed in the present invention can separate background clutter information, and use the twin prototype hybrid network to perform multi-dimensional similarity comparison between the target and the support set image to obtain the recognition result. At the same time, different images of each category are sent to the LSTM network to capture the contextual relationship, and the approximate anchor point area of ​​the target is obtained, thereby guiding the regional candidate network classifier to screen the anchor point. The tracking network proposed in the present invention can effectively track underwater targets and achieve better accuracy than the conventional tracking network in the presence of background interference. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 The network structure used for underwater single target tracking in the present invention;

[0053] Figure 2 This is the mask segmentation processing during image preprocessing of the present invention;

[0054] Figure 3 The tracking performance of the present invention and other models in underwater scenes. DETAILED DESCRIPTION

[0055] This section further illustrates the present invention in conjunction with specific implementation processes, and lists the following examples and provides detailed explanations with reference to the accompanying drawings.

[0056] like Figure 1 The underwater single target tracking method based on the twin prototype hybrid network and background mask segmentation technology can effectively track underwater targets, including the following implementation steps:

[0057] S1 obtains the original underwater image information and performs preprocessing to separate the background of the image;

[0058] S2 uses an improved twin prototype hybrid network to track single targets in preprocessed images. It first uses an adaptive cutoff filter to remove low-frequency clutter, then uses a ResNet residual block as a feature extraction network. After feature extraction, the prototype network is used to predict the uncertainty expressed by each prototype through minimum uncertainty weighted mapping. This reduces the weight of prototypes with high uncertainty in the calculation of the set prototype, achieving a mapping of multidimensional features to the feature space.

[0059] S3 introduces two different features from different categories in the prototype network, convolves them with each other, and then uses the LSTM network to capture context information to determine the approximate area of ​​the target, thereby determining the anchor point range;

[0060] S4 uses the regional candidate network classifier of classification and regression, and combines it with the anchor region determined by S3 to classify and compare the set features of the target and support sets. At the same time, the classification threshold is set to 0.75. If the threshold is lower than this, no regression calculation is performed.

[0061] S5 performs multi-dimensional aggregation on the classification results, calculates the category with the best classification result to determine the target, and finally outputs the coordinates.

[0062] Furthermore, the pre-processing in the above step S1, such as Figure 2 As shown, it is implemented by the following steps:

[0063] S1-1 Image grayscale conversion: converting color images into grayscale images to prepare for subsequent image feature enhancement;

[0064] S1-2 Image feature enhancement: Perform local histogram averaging on the grayscale image to enhance the grayscale difference of the image, so as to obtain a more accurate segmentation threshold;

[0065] S1-3 Calculate the optimal segmentation threshold: The gray wolf optimization algorithm is used to approximate the global optimal solution, which is divided into three stages: encirclement, pursuit and attack. When building the model, the three wolf packs α, β, and γ closest to the optimal solution are first defined. The remaining wolf packs (candidate solutions) are all updated around these three wolf packs; secondly, in the encirclement stage, the gray wolf pack is Perform position updates to achieve encirclement of the prey; finally, in the pursuit phase, since the location of the optimal solution is uncertain, the present invention uses the three wolf packs α, β, and γ closest to the optimal solution as potential optimal solutions, and uses α, β, and γ to guide other gray wolves to update their positions to achieve global optimization, thereby obtaining the optimal segmentation threshold;

[0066] S1-4 Background segmentation: perform threshold segmentation with the optimal segmentation threshold to obtain separated background information;

[0067] S1-5 Background mask coverage: The obtained background information is covered with a mask with a weight of 0.7, and then the covered mask image is added to the colored original image to finally obtain the preprocessed image with separated background information. The target and support set image sizes are 127×127 and 256×256, respectively.

[0068] Furthermore, the single target tracking in the above step S2 is specifically implemented by the following steps:

[0069] S2-1 Adaptive cutoff filter: After converting the spatial domain image into a frequency domain signal, the signal is filtered according to the high-frequency filter. At the same time, the cutoff frequency of the high-frequency filter is determined by the adaptive cutoff filtering algorithm. It is concluded that, finally, the filtered frequency domain signal is restored to a spatial domain image;

[0070] S2-2 Feature Extraction: The target and support set images are passed through two backbone networks with the same weights, using ResNet residual blocks instead of AlexNet shallow networks to achieve feature extraction in deeper dimensions;

[0071] S2-3 Feature Enhancement: In the backbone network, we use convolutional block attention to connect the channel attention module and the spatial attention module in series, and use convolution operations to fuse the two attention information to obtain more important features in the image and further eliminate background effects;

[0072] S2-4 Multi-dimensional feature space embedding: When processing multi-dimensional features, use the prototype network to project the multi-dimensional feature map into the feature space to form the support prototype expression and the target prototype expression respectively; then define a prior distribution with Gaussian distribution as the weight A variational distribution q(θ) with unknown parameters is selected to approximate the true posterior distribution p(θ|D). The maximum evidence lower bound (ELBO) is calculated to adjust the parameters of the variational distribution so that it is as close as possible to the posterior distribution to determine the variational distribution q(θ). Finally, the uncertainty expressed by each prototype is calculated with the help of the variational distribution q(θ) and then weighted.

[0073] Furthermore, the single target tracking in step S4 is specifically implemented by the following steps:

[0074] Two different features of each category are selected in the multi-dimensional feature space, and the contextual information between them is captured through the LSTM network after being convolved with each other, so as to obtain the approximate area where the target is located, thereby filtering out useless anchor points, reducing subsequent classification calculations, and saving recognition time and cost.

[0075] Evaluation indicators

[0076] In order to evaluate the effectiveness of the method of the present invention, we used the following indicators for evaluation: success rate (Success), precision (Precision) and normalized precision (Norm Precision). The success rate refers to the ratio of the number of frames where the match between the predicted position and the actual position is greater than the threshold to the total number of frames. The precision refers to the ratio of the number of frames where the error between the predicted anchor point and the target anchor point is less than the set threshold to the total number of video frames. The normalized precision normalizes the center point distance between the tracking result and the true value to measure the degree of match between the tracking result and the true value.

[0077] result:

[0078] The proposed method is evaluated on the OTB100 and UTB180 datasets according to the above criteria. Figure 3 The thermal performance of the tracking method of the present invention compared with other conventional tracking models in underwater background difference scenes is demonstrated. It can be observed that the tracking thermal response of the target image of the present invention is more concentrated, while the thermal responses of other tracking models have a certain degree of spillover and offset. To fully demonstrate the effectiveness of the tracking network used in the present invention, the tracking results of this method are compared with the commonly used networks in the field of underwater single target tracking and tested using the above-mentioned evaluation indicators. The comparison results are shown in Table 1. As can be seen from Table 1, the proposed network structure has certain advantages in underwater single target tasks.

[0079] Table 1 Comparison of tracking performance of different networks

[0080]

Claims

1. An underwater target tracking method based on a twin prototype hybrid network, characterized in that: The following steps are involved: S1 obtains the original underwater image data and defines the target and support set images; S2 performs background separation preprocessing on the original underwater image; S3 designs an adaptive cutoff filter and a twin network to output a multidimensional feature map based on the preprocessed target and support set images; S4 builds the prototype network and merges it with the twin network; S5 introduces two different prototype expressions under the same support set, convolves them with each other, and uses an LSTM network to determine the position of the target and select g×g×k anchor points near the target; S6 builds a regional candidate network classifier for classification and regression. Based on the g×g×k anchor points near the target, it outputs 2k channels for classification and 4k channels for regression, and obtains multi-category classification results through multi-dimensional aggregation. And the regression results Select The item with the highest probability among them outputs its corresponding regression result As the target location coordinates, category recognition and positioning are achieved.

2. The underwater target tracking method based on the twin prototype hybrid network according to claim 1 is characterized in that: The specific implementation process of step S1 is as follows: first, the underwater image is defined, the acquired image to be identified is defined as the target image, and the image with known annotations is defined as the support set image; secondly, the training set images are spatially uniformly sampled to expand the sample pairs, and the loss function of the prototype network is improved, and a variable category spacing loss function is designed to enhance the ability to distinguish different categories.

3. The underwater target tracking method based on the twin prototype hybrid network according to claim 2 is characterized in that: The training set images are spatially uniformly sampled to expand the sample pairs and improve the loss function of the prototype network. The specific implementation process is as follows: S1-1: During the training phase, the training images are spatially uniformly sampled and sample pairs are expanded. Positive pairs are generated by uniformly sampling and cropping the training images at a set step size. At the same time, semantic negative sample data is added. The added negative samples are composed of random samples of different categories to achieve the effect of expanding the training set. S1-2: Based on the Triplet loss function used when training the prototype network, the distance loss function is improved. The fixed thresholds of the difference between the distance between the query data and the negative sample and the distance between the query data and the positive sample are changed to adaptive. The designed variable category distance loss function is shown below: in and are the distances between the i-th label sample and the positive sample and the negative sample, respectively. is a variable class spacing hyperparameter, and its calculation process is m is a constant, is the sum of the squares of the distances between the positive and negative samples in the i-th label, It is the sum of the squares of the distances between the positive samples and the query samples in the i-th label.

4. The underwater target tracking method based on the twin prototype hybrid network according to claim 3 is characterized in that: The preprocessing described in step S2 is specifically as follows: first, the grayscale and contrast enhancement processing is performed on the original underwater image; the segmentation threshold is globally optimized again using the optimization algorithm to obtain the optimal segmentation threshold; then the image is segmented with the optimal segmentation threshold, the separated background is covered with a weighted mask and added to the original image to obtain a new image without background interference.

5. The underwater target tracking method based on the twin prototype hybrid network according to claim 4 is characterized in that: Step S3 is implemented as follows: designing an adaptive cutoff filter and a twin network, inputting the preprocessed target and support set images into the adaptive cutoff filter to extract high-frequency signals and enhance image texture details; Then the target and support set images are input into two feature extraction networks composed of residual blocks and shared weights of the twin network, and a maximum pooling layer and a 1×1 convolution layer are added after each block; finally, a multi-dimensional feature map is output; specifically The following processes are included: S3-1: First, the spatial domain image M(x,y) is converted into a frequency domain signal. Second, the converted signal is filtered to remove low-frequency clutter using a high-frequency filter. The filter is as follows: Where D(u,v) is the distance from the center of the frequency domain to the point (u,v), and D0 is the adaptive cutoff frequency; D0 is adjusted by the gradient amplitude G(x,y) of the image: in is the baseline cutoff frequency, λ is the gain factor; again, the high-frequency filter is applied to the frequency domain image signal middle: Finally, the frequency domain signal is inversely transformed to restore the spatial domain image M'(x,y); Said S3-2: Each residual block consists of two 1×1 convolution regularization layers, an activation function and a 3×3 convolution layer. The input is added to the convolution result after downsampling to output the dimension feature, and within the residual block, the feature is regularized by 1×1 convolution and combined with the attention module to eliminate the background effect on the target area.

6. The underwater target tracking method based on twin prototype hybrid network according to claim 5 is characterized in that: The specific implementation of step S4 is as follows: projecting the multidimensional feature map output by the twin network into the feature space to form a support set prototype expression and a target prototype expression respectively, and designing a minimum uncertainty weighted mapping method to calculate and output the set prototypes of each category.

7. The underwater target tracking method based on the twin prototype hybrid network according to claim 6 is characterized in that: The minimum uncertainty weighted mapping is specifically implemented as follows: First, define a prior distribution with Gaussian distribution as weight μ0 is the mean vector, For the covariance, choose a variational distribution q(θ) to approximate the true posterior distribution p(θ|D), where D is the observed data; Next, define and calculate the parameters of the ELBO-adjusted variational distribution that maximizes the evidence lower bound as follows: ELBO=E q(θ) [logp(D|θ]-D KL (q(θ)||p(θ)) Among them E q(θ) [logp(D|θ] is the log-likelihood expectation, which indicates the average degree of fit of the data under the variational distribution; D KL (q(θ)||p(θ)) is the divergence, which measures the difference between the variational distribution and the prior distribution; The ELBO is optimized by gradient ascent to maximize the variational distribution q(θ), so the prototype expression output follows x i is the i-th sample, θ s is the sth parameter sample drawn from q(θ); again, the uncertainty in the mean and variance estimates of the predictions are calculated as follows: Where S is the number of sampling times, is the predicted mean, To account for forecast uncertainty, is the prototype expression of the category; the inverse of the uncertainty value is taken as the weight, as shown below: Where α is a constant close to 0; the weights are normalized to obtain Finally, the prototype expressions are weighted and the set prototype is calculated as follows: Where S is the category and c is the set prototype of the category.

8. The underwater target tracking method based on twin prototype hybrid network according to claim 7 is characterized in that: The classification and regression region candidate network classifier is specifically implemented as follows: the feature map uses g×g×k anchor points near the target screened by S5, and then the anchor points pass through the classification subnet and regression subnet respectively, by convolving the query features with the support features; finally, 2k channels of classification and 4k channels of regression are output, so as to obtain multi-category classification results through multi-dimensional aggregation. And the regression results 9. The underwater target tracking method based on the twin prototype hybrid network according to claim 8, characterized in that: The region candidate network consists of two parts: classification subnet and regression subnet. If the image has k anchor points, 2k channels are output for classification and 4k channels are output for regression. The support branch and the query branch are convolved to form support classification features. Query classification features Support regression features Query regression features in and Keep the number of channels unchanged, and The number of channels is increased to 2k times and 4k times respectively; then the feature correlation is calculated in each classification and regression subnet. The calculation process is as follows: Where A cls 、A reg are the correlations between the classification subnet and the regression subnet, Conv(x,y) represents the convolution operation, and the final multi-dimensional aggregation function is as follows: Where i is the number of residual block layers, α i and β i are the weighting coefficients corresponding to layer i.

Images