Underwater target tracking method based on twin prototype hybrid network

Abstract

The invention discloses an underwater target tracking method based on a twin prototype hybrid network, and the method comprises the steps: firstly obtaining original underwater image data, defining a target and a support set image, and carrying out the background separation preprocessing; secondly, designing an adaptive cut-off filter and a twin network, establishing a prototype network, and fusing the prototype network with the twin network; and then two different prototype expressions under the same support set are introduced, and after mutual convolution, anchor points near the target are judged and screened through an LSTM network. And finally, constructing a classification and regression region candidate network classifier, based on anchor points near the target, outputting a classification channel and a regression channel, performing multi-dimensional aggregation to obtain a multi-category classification result and a regression result, selecting one item with the highest probability in the classification result, outputting the corresponding regression result as a target position coordinate, and realizing category identification and positioning. According to the invention, underwater target tracking can be effectively realized in the presence of background interference.

Description

Technical Field

[0001] The present invention belongs to the field of underwater target tracking, and in particular relates to an underwater single target tracking method based on a twin prototype hybrid network. Background Art

[0002] As countries pay more and more attention to the development of marine resources, underwater target tracking technology has been widely used in deep-sea resource exploration, rare species protection and other fields. However, there are ocean current disturbances in the ocean scene, and there is usually a difficulty that the acquisition platform is unstable, resulting in background information interference in the collected data. These difficulties make it difficult to achieve accurate tracking.

[0003] In order to improve the versatility of the model in underwater environments and improve tracking accuracy, many deep learning-based methods combined with image tracking algorithms have been applied to target tracking. As a method based on deep learning, the twin network is widely used in single target tracking tasks due to its excellent performance in tracking problems. SiamFC uses the features of different search areas and templates to convolve to obtain similarity, and transforms the target tracking problem into a similarity matching problem between the template and the search image. On this basis, SiamRPN introduces the residual network (ResNet) and the region candidate network (RPN), and uses ResNet to improve the accuracy of feature expression. At the same time, the candidate boxes of different proportions of RPN locate the target, avoiding the problem of target deformation during tracking and improving the accuracy of system tracking.

[0004] However, since the twin network tracker achieves target tracking by comparing the similarity between the search image and the template image, when the background of the video may change drastically, the changes in visual features such as the background color will cause the background of the original template to not match the background of the search image, ultimately resulting in a decrease in the recognition success rate when judging the similarity. Summary of the invention

[0005] Based on the above analysis, the present invention proposes an underwater target tracking method based on a twin prototype hybrid network to provide a high-precision and high-success rate underwater single target tracking solution to solve the problem of poor tracking performance of the twin network tracker in underwater scenes with inconsistent background information.

[0006] The tracking scheme includes two parts: background separation of underwater images using multi-threshold mask segmentation and tracking using a twin prototype hybrid network. The steps are as follows:

[0007] S1 defines and obtains the original underwater image data, and divides it into a training set and a validation set. Then, the twin prototype hybrid network model is trained based on the training set, and the model is used to track and identify the validation set images. First, the underwater image is defined, the acquired image to be identified is defined as the target image, and the image with known annotations is defined as the support set image; secondly, the training set images are spatially uniformly sampled to achieve expanded sample pairs; finally, the loss function of the prototype network is improved, and a variable category spacing loss function is designed to enhance the model's ability to distinguish different categories.

[0008] S2 performs background separation preprocessing on the original underwater images. First, grayscale and contrast enhancement are performed on these images. The segmentation threshold is globally optimized using the optimization algorithm to obtain the optimal segmentation threshold. The image is then segmented using the optimal segmentation threshold, and the separated background is covered with a weighted mask with a weight of 0.7 and added to the original image to obtain a new image without background interference.

[0009] S3 designs an adaptive cutoff filter and a twin network. The preprocessed target and support set images are input into the adaptive cutoff filter to extract high-frequency signals and enhance image texture details. The target and support set images are then input into two feature extraction networks of the twin network, which are composed of residual blocks and share weights. A maximum pooling layer is added after each block, and a 1×1 convolutional layer is used to ensure that the size and number of channels between features are the same. Finally, a multi-dimensional feature map is output, which contains image features from low to high dimensions.

[0010] S4 builds a prototype network and merges it with the twin network. The multi-dimensional feature map output by the twin network is projected into the feature space to form the support set prototype expression and the target prototype expression respectively, and the minimum uncertainty weighted mapping method is designed to calculate and output the set prototypes of each category.

[0011] S5 introduces two different prototype expressions under the same support set from the feature space of S4, and uses the LSTM network to determine the approximate position of the target after mutual convolution, and selects g×g×k anchor points near the target instead of n×m×k anchor points in the entire feature map.

[0012] S6 builds a region candidate network classifier for classification and regression. The feature map uses g×g×k anchor points near the target selected by S5, and then the anchor points pass through the classification subnet and regression subnet respectively, and convolve the query features with the support features. Finally, 2k channels for classification and 4k channels for regression are output, so that multi-dimensional aggregation can obtain multi-category classification results. And the regression results

[0013] S7 multi-category classification results after aggregation Select the item with the highest probability and output its corresponding regression result As the target location coordinates, category recognition and positioning are achieved.

[0014] Furthermore, the specific steps of step S1 are as follows:

[0015] S1-1: In the training stage, the training images are spatially uniformly sampled and sample pairs are expanded. Positive sample pairs are generated by uniformly sampling and cropping the training images according to a set step size, and semantic negative sample data is added at the same time. The added negative samples are composed of random samples of different categories to achieve the effect of expanding the training set.

[0016] S1-2: Based on the Triplet loss function used in training the prototype network, the distance loss function is improved, and the fixed threshold of the difference between the distance between the query data and the negative sample and the distance between the query data and the positive sample is changed to adaptive. The designed variable category distance loss function is shown in Formula 1.

[0017]

[0018] in are the distances between the i-th label sample and the positive sample and the negative sample, respectively. is a variable class spacing hyperparameter, and its calculation process is j≠k, m is a constant, is the sum of the squares of the distances between the positive and negative samples in the i-th label, It is the sum of the squares of the distances between the positive samples and the query samples in the i-th label.

[0019] Furthermore, in step S2, the background segmentation adopts a multi-threshold segmentation method, and the image is segmented into different categories by setting different grayscale thresholds for the image. The segmentation process is shown in Formula 2.

[0020]

[0021] In formula (2), M = {M1, M2, ..., M n} are different n categories, t is a pixel of the grayscale image, {th1,th2,…,th n} are the n thresholds set.

[0022] Furthermore, the specific implementation steps of step S3 are as follows:

[0023] S3-1: The adaptive cutoff filter converts the spatial domain image into a frequency domain signal and then filters the signal according to the adaptive cutoff frequency to filter low-frequency clutter and enhance image texture details. First, the spatial domain image is converted into a frequency domain signal according to Formula 3.

[0024]

[0025] Where M(x,y) is the original spatial domain image, x and y are the coordinates of the image in the spatial domain, H and W are the height and width of the image, respectively. is the signal after frequency domain conversion, u and v are the coordinates of the signal in the frequency domain, and i is an imaginary unit. Secondly, the converted signal is filtered out of low-frequency clutter according to a high-frequency filter, and the filter is shown in Equation 4.

[0026]

[0027] Where D(u,v) is the distance from the center of the frequency domain to the point (u,v), and D0 is the adaptive cutoff frequency. D0 is adjusted by the gradient amplitude G(x,y) of the image, and its calculation is shown in Equation 5.

[0028]

[0029] in is the baseline cutoff frequency, and λ is the gain factor. Again, the high-frequency filter is applied to the frequency domain image signal As shown in Equation 6.

[0030]

[0031] Finally, the frequency domain signal is inversely transformed and restored to the spatial domain image. The process is shown in Equation 7.

[0032]

[0033] S3-2: Each residual block consists of two 1×1 convolution regularization layers, an activation function and a 3×3 convolution layer, and the input is added to the convolution result after downsampling to output the feature of this dimension. In the present invention, the feature is combined with the attention module after 1×1 convolution regularization in the residual block to further eliminate the background effect on the target area.

[0034] Furthermore, in step S4, the minimum uncertainty weighted mapping first calculates the uncertainty of each category prototype expression to form the prototype weight, and finally weights the distance calculation to obtain the set prototype. The specific process is as follows. First, define a prior distribution with Gaussian distribution as the weight μ0 is the mean vector, is the covariance, and a variational distribution q(θ) is selected to approximate the true posterior distribution p(θ|D), where D is the observed data. The present invention selects a Gaussian distribution μ is the mean vector, is the covariance, as the variational distribution. Secondly, define and calculate the maximum evidence lower bound (ELBO) to adjust the parameters of the variational distribution to make it as close to the posterior distribution as possible. The calculation formula is shown in Equation 8.

[0035] ELBO=E q(θ) [log p(D|θ]-D KL (q(θ)||p(θ)) (8)

[0036] Where E q(θ) [log p(D|θ] is the log-likelihood expectation, which indicates the average degree of fit of the data under the variational distribution; D KL (q(θ)||p(θ)) is the divergence, which measures the difference between the variational distribution and the prior distribution. The ELBO is optimized by gradient ascent to maximize the variational distribution q(θ), so the prototype expression output follows x i is the i-th sample, θ s is the sth parameter sample drawn from q(θ). The mean and variance of the prediction results are calculated again to estimate the uncertainty, as shown in Equation 9.

[0037]

[0038] Where S is the number of sampling times, is the predicted mean, To account for forecast uncertainty, is the prototype expression of the category. The inverse of the uncertainty value is taken as the weight, as shown in Formula 10.

[0039]

[0040] Where α is a constant approaching 0. The weights are normalized, as shown in Formula 11.

[0041]

[0042] Where M is the number of prototype expressions. Finally, the prototype expressions are weighted to calculate the set prototype, as shown in Formula 12.

[0043]

[0044] Where S is the category and c is the set prototype of the category.

[0045] Furthermore, in S6, the region candidate network consists of a classification subnet and a regression subnet. If the image has k anchor points, 2k channels are output for classification and 4k channels are output for regression. The support branch and the query branch are convolved to form support classification features. Query classification features Support for regression features Query regression features in Keep the number of channels unchanged. The number of channels of is increased to 2k times and 4k times respectively. Then, the feature correlation is calculated in each classification and regression subnet, and the calculation process is shown in Equation 13.

[0046]

[0047] In formula 13, A cls , A reg are the correlations between the classification subnet and the regression subnet, respectively, and Conv(x,y) represents the convolution operation. The final multi-dimensional aggregation function is shown in Formula 14.

[0048]

[0049] Where i is the number of residual blocks, α i With β i are the weighting coefficients corresponding to the i layer respectively.

[0050] Beneficial effects of the present invention:

[0051] The method proposed in the present invention can effectively realize the effective tracking of underwater targets. Among them, the multi-threshold mask segmentation preprocessing proposed in the present invention can separate the background clutter information, and use the twin prototype hybrid network to perform multi-dimensional similarity comparison between the target and the support set image, so as to obtain the recognition result. At the same time, different images of each category are sent to the LSTM network to capture the contextual relationship, and the approximate anchor point area of ​​the target is obtained, thereby guiding the regional candidate network classifier to screen the anchor point. The tracking network proposed in the present invention can effectively realize the tracking of underwater targets and achieve better accuracy than the conventional tracking network in the presence of background interference. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 The network structure used for underwater single target tracking of the present invention;

[0053] Figure 2 It is the mask segmentation processing during the image preprocessing of the present invention;

[0054] Figure 3 The tracking performance of the present invention and other models in underwater scenes. DETAILED DESCRIPTION

[0055] This section further explains the present invention in conjunction with a specific implementation process, and the following examples are listed herewith, and described in detail with reference to the accompanying drawings.

[0056] like Figure 1 The underwater single target tracking method based on the twin prototype hybrid network and background mask segmentation technology shown in the figure can effectively realize the effective tracking of underwater targets, including the following implementation steps:

[0057] S1 obtains the original underwater image information and performs preprocessing to separate the background of the image;

[0058] S2 uses an improved twin prototype hybrid network to track a single target in the preprocessed image. First, the low-frequency clutter is filtered out by an adaptive cutoff filter, and then the ResNet residual block is used as the feature extraction network. After feature extraction, the prototype network is used to predict the uncertainty expressed by each prototype through the minimum uncertainty weighted mapping, reduce the weight of the prototype with high uncertainty in the calculation of the set prototype, and realize the mapping of multi-dimensional features to the feature space.

[0059] S3 introduces two different features of different categories in the prototype network, which are convolved with each other and then captured through the LSTM network to determine the approximate area of ​​the target, thereby determining the range of the anchor point;

[0060] S4 uses the regional candidate network classifier of classification and regression, and combines it with the anchor region determined by S3 to classify and compare the set features of the target and support sets, and sets the classification threshold to 0.75. If the threshold is lower than this, no regression calculation will be performed.

[0061] S5 performs multi-dimensional aggregation on the classification results, calculates the category with the best classification result to determine the target, and finally outputs the coordinates.

[0062] Furthermore, the preprocessing in the above step S1, such as Figure 2 As shown, it is implemented by the following steps:

[0063] S1-1 Image grayscale conversion: convert color images into grayscale images to prepare for subsequent enhancement of image features;

[0064] S1-2 Image feature enhancement: Perform local histogram averaging on the grayscale image to enhance the grayscale difference of the image, so as to obtain a more accurate segmentation threshold;

[0065] S1-3 Calculate the optimal segmentation threshold: The gray wolf optimization algorithm is used to approximate the global optimal solution, which is divided into three stages: encirclement, pursuit and attack. When establishing the model, the three wolf packs α, β, and γ closest to the optimal solution are first defined, and the remaining wolf packs (candidate solutions) are updated around these three wolf packs; secondly, in the encirclement stage, the gray wolf pack is Update the position to achieve the encirclement of the prey; finally, in the hunting stage, since the position of the optimal solution is uncertain, the present invention takes the three wolf packs α, β, and γ closest to the optimal solution as potential optimal solutions, and guides other gray wolves to update their positions through α, β, and γ to achieve global optimization, thereby obtaining the optimal segmentation threshold;

[0066] S1-4 Background segmentation: perform threshold segmentation with the optimal segmentation threshold to obtain separated background information;

[0067] S1-5 Background mask coverage: The obtained background information is covered with a mask with a weight of 0.7, and then the covered mask image is added to the colored original image to finally obtain the preprocessed image with separated background information. The target and support set image sizes are 127×127 and 256×256, respectively.

[0068] Furthermore, the single target tracking in the above step S2 is specifically implemented by the following steps:

[0069] S2-1 Adaptive cutoff filter: After converting the spatial domain image into a frequency domain signal, the signal is filtered using a high-frequency filter. At the same time, the cutoff frequency of the high-frequency filter is determined by the adaptive cutoff filtering algorithm. It is concluded that, finally, the filtered frequency domain signal is restored to a spatial domain image;

[0070] S2-2 Feature extraction: The target and support set images are respectively passed through two backbone networks with the same weights, and the ResNet residual block is used instead of the AlexNet shallow network to achieve feature extraction of deeper dimensions;

[0071] S2-3 Feature enhancement: In the backbone network, the channel attention module and the spatial attention module are connected in series using convolutional block attention, and the two attention information are fused using convolution operations to obtain more important features in the image and further eliminate background effects;

[0072] S2-4 Multi-dimensional feature space embedding: When processing multi-dimensional features, use the prototype network to project the multi-dimensional feature map into the feature space to form the supporting prototype expression and the target prototype expression respectively; then define a prior distribution with Gaussian distribution as the weight A variational distribution q(θ) with unknown parameters is selected to approximate the true posterior distribution p(θ|D), and the maximum evidence lower bound (ELBO) is calculated to adjust the parameters of the variational distribution to make it as close as possible to the posterior distribution to determine the variational distribution q(θ). Finally, the uncertainty expressed by each prototype is calculated with the help of the variational distribution q(θ) and then weighted.

[0073] Furthermore, the single target tracking in the above step S4 is specifically implemented by the following steps:

[0074] Two different features of each category are selected in the multi-dimensional feature space, and the contextual information between them is captured through the LSTM network after being convolved with each other, so as to obtain the approximate area where the target is located, thereby filtering out useless anchor points, reducing subsequent classification calculations, and saving recognition time and cost.

[0075] Evaluation indicators

[0076] In order to evaluate the effectiveness of the method of the present invention, we use the following indicators for evaluation: Success rate, precision and normalized precision. The success rate refers to the ratio of the number of frames where the match rate between the predicted position and the actual position is greater than the threshold to the total number of frames, the precision refers to the ratio of the number of frames where the error between the predicted anchor point and the target anchor point is less than the set threshold to the total number of video frames, and the normalized precision normalizes the center point distance between the tracking result and the true value, which is used to measure the degree of match between the tracking result and the true value.

[0077] result:

[0078] The proposed method is evaluated on the OTB100 and UTB180 datasets according to the above criteria. Figure 3 The thermal performance of the tracking method of the present invention in underwater background difference scenes compared with other conventional tracking models is demonstrated. It can be observed that the tracking thermal performance of the target image of the present invention is more concentrated, while the thermal responses of other tracking models have a certain degree of spillover and offset. In order to fully illustrate the effectiveness of the tracking network used in the present invention, we compared the tracking results of this method with the commonly used networks in the field of underwater single target tracking, and used the above-mentioned evaluation indicators for verification. The comparison results are shown in Table 1. It can be seen from Table 1 that the proposed network structure has certain advantages in underwater single target tasks.

[0079] Table 1 Comparison of tracking performance of different networks

[0080]

Claims

1. An underwater target tracking method based on a twin prototype hybrid network, characterized in that: The following steps are involved: S1 obtains the original underwater image data and defines the target and support set images; S2 performs background separation preprocessing on the original underwater image; S3 designs adaptive cutoff filters and twin networks to output multidimensional feature maps based on the preprocessed target and support set images; S4 builds the prototype network and merges it with the twin network; S5 introduces two different prototype expressions under the same support set, which are convolved with each other and then the position of the target is determined through the LSTM network, and the g×g×k anchor points near the target are selected; S6 builds a regional candidate network classifier for classification and regression. Based on the g×g×k anchor points near the target, it outputs 2k channels for classification and 4k channels for regression, and obtains multi-category classification results through multi-dimensional aggregation. And the regression results Select The item with the highest probability among them, output its corresponding regression result As the target location coordinates, category recognition and positioning are achieved.

2. The underwater target tracking method based on twin prototype hybrid network according to claim 1 is characterized in that: The specific implementation process of step S1 is as follows: first, an underwater image is defined, the acquired image to be identified is defined as a target image, and the image with known annotations is defined as a support set image; secondly, the training set images are spatially uniformly sampled to expand the sample pairs, and the loss function of the prototype network is improved, and a variable category spacing loss function is designed to enhance the ability to distinguish different categories.

3. The underwater target tracking method based on twin prototype hybrid network according to claim 2 is characterized in that: The training set images are spatially uniformly sampled to expand the sample pairs and improve the loss function of the prototype network. The specific implementation process is as follows: S1-1: In the training phase, the training images are spatially uniformly sampled and sample pairs are expanded. Positive pairs of samples are generated by uniformly sampling and cropping the training images according to the set step size, and semantic negative sample data is added at the same time. The added negative samples are composed of random samples of different categories to achieve the effect of expanding the training set; S1-2: Based on the Triplet loss function used in training the prototype network, the distance loss function is improved, and the fixed threshold of the difference between the distance between the query data and the negative sample and the distance between the query data and the positive sample is changed to adaptive. The designed variable category distance loss function is as follows: in and are the distances between the i-th label sample and the positive sample and the negative sample, respectively. is a variable class spacing hyperparameter, and its calculation process is m is a constant, is the sum of the squares of the distances between the positive and negative samples in the i-th label, It is the sum of the squares of the distances between the positive samples and the query samples in the i-th label.

4. The underwater target tracking method based on twin prototype hybrid network according to claim 3 is characterized in that: The preprocessing described in step S2 is specifically as follows: first, grayscale and contrast enhancement processing is performed on the original underwater image; the segmentation threshold is globally optimized again using the optimization algorithm to obtain the optimal segmentation threshold; then the image is segmented with the optimal segmentation threshold, the separated background is covered with a weighted mask and then added to the original image to obtain a new image that eliminates background interference.

5. The underwater target tracking method based on twin prototype hybrid network according to claim 4 is characterized in that: The step S3 is implemented as follows: designing an adaptive cutoff filter and a twin network, inputting the preprocessed target and support set images into the adaptive cutoff filter to extract high-frequency signals and enhance image texture details; Then the target and support set images are respectively input into two feature extraction networks composed of residual blocks and shared weights of the twin network, and a maximum pooling layer and a 1×1 convolution layer are added after each block; finally, a multi-dimensional feature map is output; specifically The process includes: S3-1: First, the spatial domain image M(x,y) is converted into a frequency domain signal; second, the converted signal is filtered out of low-frequency clutter using a high-frequency filter, and the filter is as follows: Where D(u,v) is the distance from the center of the frequency domain to the point (u,v), and D0 is the adaptive cutoff frequency; D0 is adjusted by the gradient amplitude G(x,y) of the image: in is the baseline cutoff frequency, and λ is the gain factor; again, the high-frequency filter is applied to the frequency domain image signal middle: Finally, the frequency domain signal is inversely transformed to restore the spatial domain image M'(x,y); S3-2: Each residual block consists of two 1×1 convolution regularization layers, an activation function and a 3×3 convolution layer. The input is added to the convolution result after downsampling to output the feature of this dimension, and in the residual block, the feature is combined with the attention module after 1×1 convolution regularization to eliminate the background influence on the target area.

6. The underwater target tracking method based on twin prototype hybrid network according to claim 5 is characterized in that: The step S4 is specifically implemented as follows: projecting the multidimensional feature map output by the twin network into the feature space to respectively form a support set prototype expression and a target prototype expression, and designing a minimum uncertainty weighted mapping method to calculate and output the set prototypes of each category.

7. The underwater target tracking method based on twin prototype hybrid network according to claim 6 is characterized in that: The minimum uncertainty weighted mapping is specifically implemented as follows: First, define a prior distribution with Gaussian distribution as weight μ0 is the mean vector, is the covariance, and a variational distribution q(θ) is selected to approximate the true posterior distribution p(θ|D), where D is the observed data; Secondly, define and calculate the parameters of the ELBO adjusted variational distribution that maximizes the evidence lower bound as follows: ELBO=E q(θ) [logp(D|θ]-D KL (q(θ)||p(θ)) Where E q(θ) [logp(D|θ] is the log-likelihood expectation, which indicates the average degree of fit of the data under the variational distribution; D KL (q(θ)||p(θ)) is the divergence, which measures the difference between the variational distribution and the prior distribution; The ELBO is optimized by gradient ascent to maximize the variational distribution q(θ), so the prototype expression output follows x i is the i-th sample, θ s is the sth parameter sample drawn from q(θ); the mean and variance estimation uncertainty of the prediction results are calculated again as follows: Where S is the number of sampling times, is the predicted mean, To account for forecast uncertainty, is the prototype expression of the category; the inverse of the uncertainty value is taken as the weight, as shown below: Where α is a constant close to 0; the weights are normalized to obtain Finally, the prototype expressions are weighted and the set prototype is calculated as follows: Where S is the category and c is the set prototype of the category.

8. The underwater target tracking method based on twin prototype hybrid network according to claim 7 is characterized in that: The classification and regression region candidate network classifier is specifically implemented as follows: the feature map uses g×g×k anchor points near the target selected by S5, and then the anchor points pass through the classification subnet and the regression subnet respectively, by convolving the query feature with the support feature; finally, 2k channels for classification and 4k channels for regression are output, so that multi-dimensional aggregation can obtain multi-category classification results. And the regression results 9. The underwater target tracking method based on twin prototype hybrid network according to claim 8 is characterized in that: The region candidate network consists of two parts: a classification subnet and a regression subnet. If the image has k anchor points, 2k channels are output for classification and 4k channels are output for regression. The support branch and the query branch are convolved to form support classification features. Query classification features Support for regression features Query regression features in and Keep the number of channels unchanged. and The number of channels is increased to 2k times and 4k times respectively; then the feature correlation is calculated in each classification and regression subnet, and the calculation process is as follows: Where A cls , A reg are the correlations between the classification subnet and the regression subnet, Conv(x,y) represents the convolution operation, and the final multi-dimensional aggregation function is as follows: Where i is the number of residual blocks, α i With β i are the weighting coefficients corresponding to the i layer respectively.

Images