Tracking method for task separation based on full convolution

Abstract

The invention relates to a tracking method for task separation based on full convolution, which can better process the relation between classification and positioning tasks and solve the problem of target tracking. The method comprises the following steps: firstly, respectively inputting a template image and a search frame image into a backbone network to extract features; then, constructing a prior code of a target in the template image, combining the prior code with template features, and constructing an efficient fusion module to respectively learn positioning and classification tasks of the target; then a cross task guiding module is constructed, classification and positioning features are associated from the spatial dimension, a series matching operator is constructed in the matching stage, and template and search feature matching is carried out in a global mode; and finally, fusing target positioning and classification information at the head by using a simple full-connection layer to obtain final target classification output. According to the method, target classification and positioning are facilitated, information of specific tasks can be better extracted, the tracking accuracy and success rate are effectively improved, and therefore the method can be widely applied to the civil and military fields.

Description

Technical Field

[0001] The present invention belongs to the field of target tracking, and in particular relates to a tracking method based on full convolution and task separation. Background Art

[0002] With the advancement of computer vision and deep learning, tracking methods have gained widespread application in both civilian and military fields and are becoming increasingly mature in many practical scenarios. Unmanned aerial vehicles (UAVs), a common platform for tracking targets, offer excellent tracking performance, but they also present several challenges. The limited computing resources of UAVs make it difficult to achieve both real-time speed and good performance. Furthermore, targets in a UAV's field of view are often small and exhibit significant viewpoint variations, making UAV target tracking even more challenging and placing new demands on tracking methods.

[0003] In recent years, Siamese networks have become one of the most popular and successful tracking methods for drone target tracking. Most Siamese networks use two branches to learn classification and localization features to achieve better performance. However, many trackers ignore the relationship between these two tasks, resulting in lower accuracy. Patent CN202211120456.2, "A Method and Apparatus for Target Tracking Based on Multiple Prediction Output Alignment Based on Siamese Networks," primarily implements a target tracking method based on multiple prediction output alignment based on Siamese networks. However, the introduction of additional prediction heads requires additional training. This approach strengthens the connection between classification and localization in the results, rather than at the feature level, and can partially address this issue. Another limitation of current Siamese networks is that correlation modules (e.g., cross-correlation, depth correlation, pixel correlation) operate locally, making the network prone to local optima. The Transformer architecture has a global receptive field, making it well-suited for integrating templates and search features. Patent CN202311254115.9, "A Method and System for Target Tracking Based on Spatiotemporal Context Based on Transformers," primarily implements a Transformer-based target tracking method based on spatiotemporal context. However, the downside is that Transformer has high computational complexity, which makes it difficult to deploy in real time with limited computing resources. The common problem mentioned above is that the connection between classification and localization tasks is not properly handled, resulting in low tracking accuracy. Summary of the Invention

[0004] The present invention proposes a tracking method based on full convolution and task separation to solve the problem in the prior art that a large amount of calculation makes it difficult to deploy in real time under limited computing resources.

[0005] To achieve the above object, the technical solution of the present invention is as follows: a tracking method based on task separation of full convolution, comprising the following steps:

[0006] Step 1: Input the template image and search image into the backbone network respectively to extract the template features and search features;

[0007] Step 2: After feature extraction, the center position and bounding box information of the target in the template image are first encoded a priori. Then, the a priori encoding is combined with the template features, and an efficient fusion module is used to learn the classification and positioning features of the target respectively.

[0008] Step 3: Input the learned classification and positioning features into the cross-task guidance module, and use the tandem matching operator to globally integrate the template features and search features;

[0009] Step 4: Send the integrated template features and search features to the classification and positioning head, and finally fuse the target positioning information with the classification information to obtain the final target classification output.

[0010] Furthermore, the above step 2 includes the following two steps:

[0011] Step 2.1 constructs the prior encoding of the target in the template image after sampling and combines it with the template features:

[0012] In step 2.2, an efficient fusion module is used to learn target classification and positioning features respectively.

[0013] Furthermore, the above step 2.1 specifically includes:

[0014] First, the template image is sampled at a certain ratio to contain both target information and background information;

[0015] Then the center position and bounding box information of the target in the template image are encoded a priori;

[0016] Finally, the target center position and bounding box information are fused with the template features to enhance the target features.

[0017] Furthermore, in the last step of step 2.1 above, in order to combine the bounding box information encoding with the template feature, the size of the template feature is set to k, and the bounding box size is b = [c x ,c y ,w,h], where c x and c y Represents the center coordinates of the bounding box, w and h represent the width and length of the bounding box respectively. The constructed bounding box information is as follows:

[0018]

[0019] Subsequently, n = [l, t, r, b] is connected through the channel and mapped to the channel dimension of the feature using a fully connected layer, and then fused to the feature map, as shown in Formula 2:

[0020]

[0021] in, Represents a fully connected layer that merges bounding box information into the feature map, α is a learned parameter, and the template feature z containing the bounding box information loc Input positioning information fusion branch.

[0022] Furthermore, in the last step of step 2.1 above, for the center position of the target, a Gaussian function is used to construct a Gaussian response as the target position encoding; after constructing the position information, a fully connected layer is used to map it to the number of feature channels and integrate it into the network, as shown in the following formula

[0023] z cla =z+β·η(g) (6)

[0024] Among them, η is a fully connected layer, β is a learned parameter containing position information z cla The template features will be input into the classification fusion branch.

[0025] Furthermore, in the above step 2.2, first obtain the global information map

[0026]

[0027] The superscript T represents the transpose operation of the matrix, which obtains the global information and reshapes it into the same dimension as the search box space;

[0028] Then, it is mapped into template information and search information through two convolutional layers. At the same time, the learned information is mapped to 1 / 2 of the number of channels to amplify the global information. The two types of global information after mapping are

[0029] and

[0030] For ω p It is connected with the amplified search frame features by channel, and then a 1×1 convolution layer is used for dimensionality reduction to reduce the number of channels to c

[0031] x p =conv 1×1 (cat(ω p ,conv 1×1 (x))) (8)

[0032] Where, conv 1×1represents a convolution layer with a convolution kernel of 1, and cat represents a connection operation;

[0033] Use ω n To learn the search feature as spatial attention, the global information is converted into a spatial weight map, and the feature x that integrates the global information is p ∈R c×h×w Perform spatial weighting

[0034] x w =mean(ω n )·x p (9)

[0035] For the learned search features, deep convolution with different convolution kernel sizes is used to further learn the features and obtain the target feature information.

[0036] Furthermore, in the above step 3.1, the spatial information of the features learned from the two tasks is selected for learning fusion. First, the classification features and work features Flattened according to the spatial dimension and Then the two features are learned separately and fused using addition operation

[0037]

[0038] Where, Represents the fused features. The fused features are then mapped and learned to be converted back into classification and positioning features.

[0039]

[0040] Furthermore, in the above step 3.2, the fusion expression of template features and search features is

[0041] f=concat(FC(avgpool(z)),x) (15)

[0042] Where avgpool represents the adaptive pooling layer, FC represents the fully connected layer, and the learned features After the fully connected layer, the dimension is reduced and the features are finally learned. The tandem matching operator is used to utilize the entire template features to obtain fusion features.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] (1) The present invention constructs a priori coding of the target, encodes the center position and bounding box information of the target in the template, and combines it with the template features, thereby enhancing the target information and facilitating the classification and positioning of the target.

[0045] (2) The present invention designs an efficient fusion module, which uses two fusion modules to construct an efficient fusion module, respectively learning the classification and positioning information of the target, avoiding the contradictions in the learning process, and better extracting information for specific tasks.

[0046] (3) In the efficient fusion module, classification and positioning information are learned separately, and task guidance interaction is performed on the two features, so that each task introduces content from another task that can help the task learn, thereby constructing a cross-task guidance module and improving the accuracy of task learning.

[0047] (4) A tandem matching operator is used to globally integrate template features and search features. By further learning global features, the learned template and search features are better matched to enhance the connection between classification and positioning tasks, avoiding the problem of large-scale calculations leading to difficulty in real-time deployment under limited computing resources, and effectively improving the accuracy of target tracking.

[0048] (5) The present invention provides a faster and better fusion method, which improves the accuracy and success rate of tracking, and is therefore widely applicable to civil and military fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 This is the overall flow chart of the tracking method based on full convolution task separation;

[0050] Figure 2 Visualization of the results for the fully convolutional based task separation tracking method. DETAILED DESCRIPTION

[0051] The present invention will be described in further detail below with reference to the accompanying drawings and embodiments.

[0052] like Figure 1 The overall flow chart of the present invention is shown below. First, the template image and search image are input into the backbone network to extract template and search features. A priori encoding of the target in the template image is constructed and integrated into the template features. Simultaneously, an efficient fusion module learns target classification and localization features separately. The cross-task guidance module then connects the spatial classification and localization tasks, and globally integrates the template and search features using a cascade matching operator. Finally, this information is fed into the classification and localization head, where the target location information is fused with the classification information to obtain the final target classification output.

[0053] The present invention provides a tracking method based on full convolution task separation, and the specific implementation steps are as follows:

[0054] Step 1: Input the template image and search image into the backbone network to extract template features and search features. The specific operations are as follows:

[0055] This method uses the ImageNet pre-trained model of AlexNet as the backbone, denoted as our_alex. The backbone network is used to extract feature information from the template image and the search image. The template image and the search image are input into the backbone network to extract the template features. and search features A 1×1 convolutional layer is used to map the feature channels to 256.

[0056] Step 2: Priori encode the center position and bounding box information of the target in the template image, then combine the priori encoding with the template features, and use an efficient fusion module to learn the classification and positioning features of the target respectively. The specific operation includes the following two steps that are carried out simultaneously:

[0057] Step 2.1 constructs the prior encoding of the target in the template image after sampling and combines it with the template features:

[0058] First, the template image is sampled at a ratio of 2, so that it contains both target information and background information, thereby learning the context to facilitate target classification and positioning.

[0059] Then the center position and bounding box information of the target in the template image are encoded a priori.

[0060] Finally, the target’s center position and bounding box information are fused with the template features to enhance the target features, which consists of two parts:

[0061] (1) In order to combine the bounding box information encoding with the template feature, the size of the template feature is set to k. In this embodiment, k∈[0,15]. The bounding box size is b=[c x ,c y ,w,h], where c x and c y Represents the center coordinates of the bounding box, i.e., the mean, w and h represent the width and length of the bounding box respectively. The constructed bounding box information is as follows:

[0062]

[0063] Subsequently, n = [l, t, r, b] is connected through the channel and mapped to the channel dimension of the feature using a fully connected layer, and then fused to the feature map, as shown in Formula 2:

[0064]

[0065] in, represents a fully connected layer that merges bounding box information into the feature map. α is a learned parameter. The template feature z containing the bounding box information is loc Input positioning information fusion branch.

[0066] (2) For the center position of the target, a Gaussian function is used to construct a Gaussian response as the target position code. The Gaussian structure is shown in formulas (3)-(5)

[0067]

[0068] g=gauss_x×gauss_y (5)

[0069] The exponential function is exp(), and the variance in the x and y dimensions is σ x and σ y express.

[0070] By using the bounding box information to learn the variance of the Gaussian response, the bounding box map is input to a fully connected layer and the channels are compressed to 1. The channel dimension and the spatial dimension are then transposed and the bias in both directions is learned in the same way.

[0071] After constructing the position information, a fully connected layer is used to map it to the number of feature channels and fuse it into the network, as shown in the following formula.

[0072] z cla =z+β·η(g) (6)

[0073] Among them, η is a fully connected layer, β is a learned parameter containing position information z cla The template features will be input into the classification fusion branch.

[0074] Step 2.2 uses an efficient fusion module to learn target classification and positioning features respectively:

[0075] Two branches are constructed to learn target location and classification information respectively, using a hierarchical learning structure. Two cross-fusion modules are used to construct the entire fusion module. The template branch is first fused with the search branch, allowing the search features to learn the target information in the template features, giving them a certain degree of resolution. The learned features are then fused with the template features, allowing the template features to incorporate the target information in the current frame, adapting to the current target's deformation and other conditions, and better distinguishing the target.

[0076] The Transformer module can obtain global information during tracking, so that it can achieve better performance in tracking. Considering the problem of large computational complexity of the original Transformer module, the present invention adopts point convolution to obtain global information and introduces the global information into the search frame features.

[0077] First, the template features obtained in step 1 and search features Expand according to the spatial dimension, h z ,h s Represents the length of template feature and search feature, w z ,w s Represents the width of the template feature and the search feature, and then flattens the space and Perform multiplication operations to obtain a global information graph.

[0078]

[0079] The superscript T represents the transpose operation of the matrix, which obtains the global information and reshapes it into the same dimension as the search box space.

[0080] After obtaining the global information, it is mapped into template information and search information through two convolutional layers. At the same time, the learned information is mapped to 1 / 2 of the number of channels to amplify the global information. The two types of global information after mapping are and

[0081] For ω p It is concatenated with the amplified search frame features by channel, and then a 1×1 convolutional layer is used for dimensionality reduction to reduce the number of channels to c.

[0082] x p =conv 1×1 (cat(ω p ,conv 1×1 (x))) (8)

[0083] Where, conv 1×1 represents a convolution layer with a convolution kernel of 1, and cat represents a connection operation.

[0084] For ω n , use it to learn the search feature as spatial attention. Since global information is obtained by calculating the similarity of two features in the spatial dimension, it is converted into a spatial weight map, and the other part of the feature x that integrates the global information is used. p ∈R c×h×w Perform spatial weighting.

[0085] x w =mean(ω n )·x p (9)

[0086] After learning the search features, we further learn them using deep convolutions at different resolutions. We also use deep convolutions with different kernel sizes, 3×3 and 5×5, to further learn the features. We use these kernel sizes to learn the correlation between the features and ultimately combine them to obtain target feature information and improve tracking accuracy.

[0087] x f =DWconv 3×3 (x w )+DWconv 5×5 (x w ) (10)

[0088] Finally, the search features after integrating global information are obtained by addition and regularization.

[0089] x'=BN(x f +x) (11)

[0090] In the formula, BN stands for batch normalization. In the process of searching for features to interact with template features, the global information is compressed and converted from Compress to This completes the process of interaction from the target's template information to the search information, enabling it to acquire the ability to distinguish targets.

[0091] Step 3: Input the learned template and search features into the cross-task guidance module, link the classification and positioning tasks in the spatial dimension, and use the tandem matching operator to globally integrate the template features and search features. The specific operations are as follows:

[0092] Step 3.1 The cross-task guidance module connects the classification and localization tasks in the spatial dimension:

[0093] In the efficient fusion module, classification and localization are learned separately, but there is still a certain connection between the two tasks. Therefore, after efficient fusion, the two features are interactively guided by each other, allowing each task to introduce content from the other task that can help the task learn.

[0094] Classification and localization tasks have great differences in spatial dimensions. Therefore, we choose to learn and fuse the spatial information of the features learned from the two tasks. First, we transform the classification features into and work features Flattened according to the spatial dimension and Then the two features are learned separately and fused using addition operation.

[0095]

[0096] Where, Represents the fused features. The fused features are then mapped and learned to be converted back into classification and positioning features.

[0097]

[0098] This completes the fusion of features learned from the classification and localization tasks, completing the interactive guided learning corresponding to the two tasks. The features after guided learning will be directly input into the final fusion structure to learn the final features for localization and classification.

[0099] Step 3.2: Concatenate matching operators to globally integrate template features and search features

[0100] In order to better achieve the matching of templates and search features, a global matching operator is required. Compared with Transformer, the cascade matching operator can effectively integrate features globally.

[0101] In order to better match the template features, the pooling layer is used to compress the template features, and then the compressed features are learned through the fully connected layer. After the learned features are spliced ​​to each feature point in the search feature, the fusion of the template features and the search features is completed.

[0102] f=concat(FC(avgpool(z)),x) (15)

[0103] Where avgpool represents the adaptive pooling layer, FC represents the fully connected layer, and the learned features After the fully connected layer, the dimension is reduced and the features are finally learned. The serial matching operator is used to utilize the entire template features to obtain more effective fusion features.

[0104] Step 4: Send the integrated template features and search features to the classification and localization head. The head uses a simple fully connected layer and finally fuses the target's localization information with the classification information to obtain the final target classification output.

[0105] In this paper, the performance of the final object classification output in step 4 is evaluated using the evaluation (OPE) criterion, namely, accuracy and success rate. The former is calculated by the center position error (CLE) between the ground truth center and the predicted box. The latter is defined by the intersection over union (IoU).

[0106] CLE is defined as the Euclidean distance between the predicted bounding box and the ground truth bounding box. The calculation equation is as follows.

[0107]

[0108] x pre and y preRepresents the x and y coordinates of the predicted bounding box, x gt and y gt Represents ground coordinates.

[0109] The accuracy is calculated as the ratio of the number of frames above a certain threshold to the total number of frames. The threshold is set to 20 pixels following the default setting of the dataset.

[0110] The success rate is calculated by IoU, also known as AUC, which can be calculated as follows.

[0111]

[0112] B pre and B gt denote the predicted bounding box and the true bounding box, respectively, and ∩ and ∪ represent the intersection and union.

[0113] The definition of success rate is similar to accuracy: the ratio of the number of frames exceeding a certain threshold to the total number of frames. The threshold is set to 50, following the default setting of the dataset. The OPE evaluation metrics are shown in Table 1.

[0114] Table 1. Evaluation results of tracking methods based on full convolution task separation on different UAV tracking

[0115]

[0116]

[0117] As shown in Table 1, the overall performance shows that our_alex achieves the best results in terms of success rate and accuracy. On the three datasets DTB70, UAV123, and UAVTrack112, both the success rate and accuracy are higher than TCTrack++, SiamAPN++, and HiFT, achieving the best tracking rate. Some sequences from the DTB70 and UAVTrack112 datasets are selected to show the visualization of the tracking method, as shown in the figure below. Figure 2 As shown, the blue boxes represent our_alex, while the red, pink, and black boxes represent TCTrack++, SiamAPN++, and HiFT. These sequences demonstrate complex backgrounds, varying viewpoints, and scales. The figures show that our tracking method successfully tracks the target, demonstrating greater robustness than other tracking methods.

[0118] The present invention proposes a faster and better feature fusion for drone tracking. Specifically, the separated feature branches learn the classification and positioning tasks respectively, and the prior encoding is proposed to enhance the target information. The efficient fusion module fuses the global response map into the search feature. In order to connect the two tasks, a cross-task guidance module is designed to link the classification and positioning features. In order to efficiently match the template features and the search features in the global view, a tandem matching operator is used in the final matching stage. Experimental results show that the task separation-based tracking method proposed in the present invention performs well on the three datasets of DTB70, UAV123 and UAVTrack112. Compared with similar representative methods, its tracking accuracy reaches a better level.

[0119] The above description is an explanation of the specific implementation of the present invention, rather than a limitation of the present invention. Those skilled in the relevant technical field can also make various equivalent technical solutions without departing from the scope of the present invention, so all equivalent technical solutions should be included in the scope of protection of the present invention.

Claims

1. A task separation tracking method based on full convolution, characterized by: The following steps are involved: Step 1: Input the template image and search image into the backbone network respectively to extract the template features and search features; Step 2: After feature extraction, the center position and bounding box information of the target in the template image are first encoded a priori. Then, the a priori encoding is combined with the template features, and an efficient fusion module is used to learn the classification and positioning features of the target respectively. Step 3: Input the learned classification and positioning features into the cross-task guidance module, and use the tandem matching operator to globally integrate the template features and search features; Step 4: Send the integrated template features and search features to the classification and positioning head, and finally fuse the target positioning information with the classification information to obtain the final target classification output.

2. The tracking method based on full convolution and task separation according to claim 1, characterized in that: The second step includes the following two steps: Step 2.1 constructs the prior encoding of the target in the template image after sampling and combines it with the template features: In step 2.2, an efficient fusion module is used to learn target classification and positioning features respectively.

3. The tracking method based on full convolution and task separation according to claim 2, characterized in that: The step 2.1 specifically includes: First, the template image is sampled at a certain ratio to contain both target information and background information; Then the center position and bounding box information of the target in the template image are encoded a priori; Finally, the target center position and bounding box information are fused with the template features to enhance the target features.

4. The tracking method based on full convolution and task separation according to claim 3, characterized in that: In the last step of step 2.1, in order to combine the bounding box information encoding with the template feature, the size of the template feature is set to k, and the bounding box size is b = [c x ,c y ,w,h], where c x and c y Represents the center coordinates of the bounding box, w and h represent the width and length of the bounding box respectively. The constructed bounding box information is as follows: Subsequently, n = [l, t, r, b] is connected through the channel and mapped to the channel dimension of the feature using a fully connected layer, and then fused to the feature map, as shown in Formula 2: in, Represents a fully connected layer that merges bounding box information into the feature map, α is a learned parameter, and the template feature z containing the bounding box information loc Input positioning information fusion branch.

5. The tracking method based on full convolution and task separation according to claim 3, characterized in that: In the last step of step 2.1, for the center position of the target, a Gaussian function is used to construct a Gaussian response as the target position encoding; after constructing the position information, a fully connected layer is used to map it to the number of feature channels and integrate it into the network, as shown in the following formula With cla =z+β η(g) (6) Among them, η is a fully connected layer, β is a learned parameter containing position information z cla The template features will be input into the classification fusion branch.

6. The tracking method based on full convolution and task separation according to claim 4 or 5, characterized in that: In step 2.2, first obtain the global information map The superscript T represents the transpose operation of the matrix, which obtains the global information and reshapes it into the same dimension as the search box space; Then, it is mapped into template information and search information through two convolutional layers. At the same time, the learned information is mapped to 1 / 2 of the number of channels to amplify the global information. The two types of global information after mapping are and For ω p It is connected with the amplified search frame features by channel, and then a 1×1 convolution layer is used for dimensionality reduction to reduce the number of channels to c x p =conv 1×1 (cat(ω p ,conv 1×1 (x))) (8) Where, conv 1×1 represents a convolution layer with a convolution kernel of 1, and cat represents a connection operation; Use ω n To learn the search feature as spatial attention, the global information is converted into a spatial weight map, and the feature x that integrates the global information is p ∈R c×h×w Perform spatial weighting x w =mean(ω n )·x p (9) For the learned search features, deep convolution with different convolution kernel sizes is used to further learn the features and obtain the target feature information.

7. The tracking method based on task separation based on full convolution according to claim 6, characterized in that: In step 3.1, the spatial information of the features learned from the two tasks is selected for learning fusion. First, the classification features and work features Flattened according to the spatial dimension and Then the two features are learned separately and fused using addition operation Where, Represents the fused features. The fused features are then mapped and learned to be converted back into classification and positioning features.

8. The tracking method based on full convolution and task separation according to claim 7, characterized in that: In step 3.2, the fusion expression of template features and search features is: f=concat(FC(avgpool(z)),x) (15) Where avgpool represents the adaptive pooling layer, FC represents the fully connected layer, and the learned features After the fully connected layer, the dimension is reduced and the features are finally learned. The tandem matching operator is used to utilize the entire template features to obtain fusion features.

Images