A Target Tracking Method and System Based on Cross-Layer Feature Calibration Fusion

Abstract

This invention relates to a target tracking method and system based on cross-layer feature calibration fusion. The method includes: inputting a template region image and a search region image into a Siamese convolutional neural network with shared weights; extracting features from the template branch and search branch using a modified ResNet50, and extracting features from the third and fourth convolutional blocks; fusing the features of the third and fourth convolutional blocks of the template branch and search branch using an attention-based feature fusion module; calibrating the features of the fourth convolutional block of the template branch and search branch using the fused features through a cross-layer calibration module; performing deep cross-correlation on the features of the third convolutional block of the template branch and search branch, the fused features, and the calibrated features to obtain three similarity response maps, which are then concatenated and dimensionality reduced; inputting the dimensionality-reduced similarity response maps into a prediction head for classification regression, and finally obtaining the target's position. This method and system are beneficial for improving target tracking performance.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to a target tracking method and system based on cross-layer feature calibration fusion. Background Technology

[0002] Object tracking is a hot topic in computer vision research. Over the past few decades, object tracking research has made significant progress, and it remains an important research subject. The task of object tracking is to continuously track a target and predict its trajectory based on the target information given in the initial frame. In real life, visual object tracking has wide applications, such as camera tracking and focusing, traffic monitoring, human-computer interaction, robot visual navigation, virtual reality, and automatic target tracking for drones. However, because targets often encounter situations such as lighting changes, rapid deformation, and cluttered backgrounds during tracking, designing more accurate and robust tracking algorithms remains a challenging task.

[0003] Prior to 2016, mainstream target tracking algorithms were primarily based on correlation filtering principles, with classic algorithms including MOSSE, CSK, KCF, CN, and SAMF. With the development of deep learning and its application in computer vision, many researchers began to explore applying deep neural networks to target tracking. Classic convolutional neural networks (CNNs) include AlexNet, VGG-Net, GoogLeNet, and ResNet. Because the computational speed of these algorithms is severely limited by the network depth, while deep neural network-based target tracking algorithms have improved performance, their real-time performance remains significantly affected; for example, VITAL's tracking speed is less than 1 FPS. In 2016, Siamese network trackers, represented by SINT and SiamFC, gained widespread attention due to their astonishing speed and considerable accuracy. Much subsequent work has focused on improving upon these, such as DaSiamRPN, SiamRPN, SiamMask, and SiamRPN++. Notably, in recent years, tracking algorithms based on Siamese networks have achieved excellent results in papers accepted by CVPR, ICCV, and ECCV.

[0004] In the current field of object tracking, Siamese networks are a research hotspot. Siamese networks were initially used for signature verification and face recognition, with excellent results. Siamese networks used for object tracking typically have two branches: a template branch, which uses the target from the first frame as a template; and a search branch, which uses subsequent images as the search domain. The target template image and the search domain image are each processed by two parameter-sharing convolutional neural networks to extract features. Then, a pre-trained offline metric function is used to calculate the similarity between the two images, and the region with the highest similarity is used as the predicted region for the current frame. Since SiamFC demonstrated a significant speed advantage, more and more trackers are based on Siamese networks. Li et al. combined Siamese networks with region proposal extraction networks and trained the network offline on a large-scale dataset, resulting in a tracker that is not only highly accurate but also very fast.

[0005] Existing technologies either simply use shallow or deep features, or they simply fuse shallow and deep features through addition. Therefore, they cannot fully utilize the different layers of the feature extraction network to extract features. However, shallow and deep networks each have their advantages and disadvantages. Shallow networks have smaller receptive fields and smaller overlapping areas, ensuring the network captures more details. The advantage is higher resolution, containing more location and detail information; however, due to fewer convolutions, their semantic meaning is lower and they have more noise. Deep networks have increased receptive fields and greater overlap, compressing image information and acquiring information about the overall image. The advantage is stronger semantic information, but the resolution is very low, and the ability to perceive details is poor. Summary of the Invention

[0006] The purpose of this invention is to provide a target tracking method and system based on cross-layer feature calibration fusion, which is beneficial to improving the performance of target tracking.

[0007] To achieve the above objectives, the technical solution adopted by this invention is: a target tracking method based on cross-layer feature calibration fusion, comprising the following steps:

[0008] A. Construct a twin convolutional neural network, which includes a template branch and a search branch with shared weights; input the template region image and the search region image into the template branch and the search branch of the twin convolutional neural network, respectively;

[0009] B. The modified feature extraction network ResNet50 is used to extract features from the template branch and the search branch respectively, and the extracted features include the features of the third convolutional block and the features of the fourth convolutional block;

[0010] C. Use the attention-based feature fusion module to fuse the features of the third and fourth convolutional blocks of the template branch, and fuse the features of the third and fourth convolutional blocks of the search branch respectively;

[0011] D. The template branch and the search branch respectively use the cross-layer calibration module to calibrate the features of the fourth convolutional block using the fused features;

[0012] E. Perform deep cross-correlation operations on the features of the third convolutional block of the template branch and the search branch, the features fused from the template branch and the search branch, and the features calibrated from the template branch and the search branch, respectively, to obtain three similarity response maps;

[0013] F. Combine the three similarity response maps and then perform dimensionality reduction.

[0014] G. Input the dimensionality-reduced similarity response map into the prediction head for classification and regression, and finally obtain the location of the target.

[0015] Furthermore, in step B, the modified feature extraction network resnet50 is modified by changing the effective stride of the last two layers of the resnet50 network to 8, and changing the convolution mode of the last two layers to dilated convolution, so as to reduce the stride without shrinking the receptive field.

[0016] Furthermore, in step B, the features of the third convolutional block of the template branch are extracted respectively. Features of the fourth convolutional block And the features of the third convolutional block in the search branch. Features of the fourth convolutional block

[0017] In step C, an attention-based feature fusion module is used to fuse the features. and get and integration and get

[0018] Furthermore, in step C, the specific implementation method of the attention-based feature fusion module is as follows:

[0019] Given two input features a and b, we first perform a simple initial feature integration on a and b to obtain feature A, i.e.

[0020] A = a·b

[0021] The obtained features are then input into local attention and non-local attention, respectively.

[0022] L(A)=β(PWConv1(δ(β(PWConv2(A)))))

[0023]

[0024] The kernel sizes of PWConv1 and PWConv2 are respectively... and β represents batch normalization, δ is the sigmoid function, i represents the output position, and its response value is calculated by enumerating j; the F function calculates A. i and A j The similarity between them; the G function calculates the representation of the feature map at position j, and the final NL(A) is normalized by the response factor D(A);

[0025] Then, the features obtained through local attention and non-local attention are added bitwise, and after passing through an activation function, they are multiplied element-wise with the original feature A to produce the refined feature A':

[0026]

[0027] Where M(A) represents the attention weights obtained from local attention and non-local attention; This indicates element addition; This indicates element-wise multiplication;

[0028] Finally, the characteristics of the fusion are represented as follows:

[0029]

[0030] The feature fusion module is used to fuse the features of the third and fourth convolutional blocks of the template branch and the search branch, respectively, as shown below:

[0031]

[0032]

[0033] Where h represents the attention-based feature fusion function.

[0034] Furthermore, in step D, a cross-layer calibration module is used to calibrate deep features, and the features obtained after deep fusion are used to calibrate the features of the last residual block; specifically:

[0035] Each and As input Y, take and As input T; first, perform a feature transformation on Y based on S2:

[0036] Y' = Up(f2(Y)) = Up(Y*S2)

[0037] Then, we obtain vectors Q and B, as follows:

[0038] Q=f4(f3(T)·δ(T')=S4*((T*S3)·δ(T'))

[0039] B = f1(T) = T * S1

[0040] Where f1, f2, f3, and f4 represent traditional 2D convolutional layers with different kernel sizes, and S1, S2, S3, and S4 represent convolutional kernels of different sizes; Up(.) is the bilinear interpolation operator;

[0041]

[0042]

[0043] Where K represents the cross-layer calibration function.

[0044] Further, in step E, deep cross-correlation operations are performed on the features of the third convolutional block of the template branch and the search branch, the features fused from the template branch and the search branch, and the features calibrated from the template branch and the search branch, respectively, to obtain three similarity response maps. The specific formulas are as follows:

[0045]

[0046]

[0047]

[0048] Furthermore, in step F, after the three similarity response maps are stitched together, dimensionality reduction is performed using a 1×1 convolution.

[0049] The present invention also provides a target tracking system based on cross-layer feature calibration fusion, including a memory, a processor, and computer program instructions stored in the memory and executable by the processor. When the processor executes the computer program instructions, it can implement the above-described method steps.

[0050] Compared with existing technologies, this invention has the following advantages: This invention provides a target tracking method and system based on calibration feature fusion, which includes an attention-based feature fusion module and a cross-layer calibration module. The method first fuses features from deep and shallow networks using the attention-based feature fusion module, ensuring that the fused features contain both rich semantic and visual information, while retaining key information learned by the shallow network. Then, the cross-layer calibration module calibrates the features learned by the deep network. This overcomes the problem that key information learned by the shallow network may be forgotten during the deep neural network learning process, and that the last residual block of ResNet-50 often contains a lot of redundant background information. Experiments have shown that the method of this invention is effective and achieves a real-time tracking speed exceeding 52 FPS, effectively improving the performance of target tracking. Attached Figure Description

[0051] Figure 1 This is a network model framework diagram of an embodiment of the present invention.

[0052] Figure 2 This is a framework diagram of the attention-based feature fusion module in an embodiment of the present invention.

[0053] Figure 3 This is a framework diagram of the cross-layer calibration module in an embodiment of the present invention. Detailed Implementation

[0054] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0055] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0056] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0057] This embodiment provides a target tracking method based on cross-layer feature calibration fusion, and its implemented network model is as follows: Figure 1 As shown. This method includes the following steps:

[0058] A. Construct a twin convolutional neural network, which includes a template branch and a search branch with shared weights; input the template region image and the search region image into the template branch and the search branch of the twin convolutional neural network, respectively;

[0059] B. The modified feature extraction network ResNet50 is used to extract features from the template branch and the search branch respectively, and the extracted features include the features of the third convolutional block and the features of the fourth convolutional block;

[0060] C. Use the attention-based feature fusion module to fuse the features of the third and fourth convolutional blocks of the template branch, and fuse the features of the third and fourth convolutional blocks of the search branch respectively;

[0061] D. The template branch and the search branch respectively use the cross-layer calibration module to calibrate the features of the fourth convolutional block using the fused features;

[0062] E. Perform deep cross-correlation operations on the features of the third convolutional block of the template branch and the search branch, the features fused from the template branch and the search branch, and the features calibrated from the template branch and the search branch, respectively, to obtain three similarity response maps;

[0063] F. Combine the three similarity response maps and then perform dimensionality reduction.

[0064] G. Input the dimensionality-reduced similarity response map into the prediction head for classification and regression, and finally obtain the location of the target.

[0065] In this embodiment, the modified feature extraction network resnet50 is achieved by changing the effective stride of the last two layers of the resnet50 network to 8, and changing the convolution method of the last two layers to dilated convolution, so as to reduce the stride without shrinking the receptive field.

[0066] In step B, features of the third convolutional block of the template branch are extracted respectively. Features of the fourth convolutional block And the features of the third convolutional block in the search branch. Features of the fourth convolutional block

[0067] In step C, an attention-based feature fusion module is used to fuse the features. and get and integration and get

[0068] In this embodiment, the attention-based feature fusion module is as follows: Figure 2As shown. The specific implementation method of the attention-based feature fusion module is as follows:

[0069] Given two input features a and b, we first perform a simple initial feature integration on a and b to obtain feature A, i.e.

[0070] A = a·b

[0071] The obtained features are then input into local attention and non-local attention, respectively.

[0072] L(A)=β(PWConv1(δ(β(PWConv2(A)))))

[0073]

[0074] The kernel sizes of PWConv1 and PWConv2 are respectively... and β represents batch normalization, δ is the Sigmoid function, and i represents the output position, such as a spatial, temporal, or spatiotemporal index, whose response value is calculated by enumerating j; the F function calculates A. i and A j The similarity between them; the G function calculates the representation of the feature map at position j, and the final NL(A) is normalized by the response factor D(A).

[0075] Then, the features obtained through local attention and non-local attention are added bitwise, and after passing through an activation function, they are multiplied element-wise with the original feature A to produce the refined feature A':

[0076]

[0077] Where M(A) represents the attention weights obtained from local attention and non-local attention; This indicates element addition; This indicates element-wise multiplication.

[0078] Finally, the characteristics of the fusion are represented as follows:

[0079]

[0080] The feature fusion module is used to fuse the features of the third and fourth convolutional blocks of the template branch and the search branch, respectively, as shown below:

[0081]

[0082]

[0083] Where h represents the attention-based feature fusion function.

[0084] In this embodiment, the cross-layer calibration module is as follows: Figure 3 As shown. In step D, a cross-layer calibration module is used to calibrate deep features, and the features obtained after deep fusion are used to calibrate the features of the last residual block; specifically:

[0085] Each and As input Y, take and As input T; first, perform a feature transformation on Y based on S2:

[0086] Y' = Up(f2(Y)) = Up(Y*S2)

[0087] Then, we obtain vectors Q and B, as follows:

[0088] Q=f4(f3(T)·δ(T')=S4*((T*S3)·δ(T'))

[0089] B = f1(T) = T * S1

[0090] Where f1, f2, f3, and f4 represent traditional 2D convolutional layers with different kernel sizes, and S1, S2, S3, and S4 represent convolutional kernels of different sizes; Up(.) is the bilinear interpolation operator.

[0091]

[0092]

[0093] Where K represents the cross-layer calibration function.

[0094] In step E, deep cross-correlation operations are performed on the features of the third convolutional block of the template branch and the search branch, the features fused from the template branch and the search branch, and the features calibrated from the template branch and the search branch, respectively, to obtain three similarity response maps. The specific formulas are as follows:

[0095]

[0096]

[0097]

[0098] In step F, the three similarity response maps are stitched together and then dimensionality is reduced using a 1×1 convolution.

[0099] In step G, the dimensionality-reduced similarity response map is input into the prediction head for classification and regression, and finally the location of the target is obtained.

[0100] This embodiment also provides a target tracking system based on cross-layer feature calibration fusion, including a memory, a processor, and computer program instructions stored in the memory and executable by the processor. When the processor executes the computer program instructions, it can perform the above-described method steps.

[0101] In this embodiment, the results of the method proposed in this invention are compared with those of other methods. Table 1 shows a comparison of the results of the method proposed in this invention with other face attribute recognition methods on the UAV20L dataset. Table 2 shows a comparison of the results of the method proposed in this invention with other face attribute recognition methods on the UAV123 dataset.

[0102] Table 1 compares the recognition results of the method of this invention with other methods on the UAV20L dataset.

[0103]

[0104] Table 2 compares the recognition results of the method of this invention with other methods on the UAV123 dataset.

[0105]

[0106]

[0107] in:

[0108] MemTrack corresponds to the method proposed by T. Yang, ABChan et al., Learning dynamic memory networks for object tracking, in: Proceedings of the European conference on computer vision (ECCV), 2018, pp. 152–167;

[0109] Siam FC corresponds to Luca Bertinetto, Jack Valmadre The method proposed by F. Henriques, Andrea Vedaldi, Philip HS et al. (Fully-Convolutional Siamese Networks for Object Tracking. In: ECCV Workshop (2016));

[0110] CFNet in J.Valmadre,L.Bertinetto,J.Henriques,A.Vedaldi,PHTorrEnd-to-end representation learning for correlation filter basedtracking,in:Proceedings ofthe IEEE Conference on Computer Vision and PatternRecognition(CVPR),2017,pp.2805–2813.);

[0111] DNAnet and S. Yun, J. Choi, Y. Yoo, K. Yun, J. Young ChoOverview of Action-Decision Networks for Visual Tracking with Deep Reinforcement Learning, in:Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition(CVPR),2017,pp.2711–2720.)4

[0112] Siam RPN and B. Li, J. Yan, W. Wu, Z. Zhu, and X. HuObjectives for High PerformanceVisual Tracking with Siamese Region Proposal Network, in:Proceedings of theIEEE Conference on Computer Vision and Pattern Recognition(CVPR),2018,pp.8971–8980.)4

[0113] VITAL corresponds to the method proposed by Song, C. Ma, X. Wu, L. Gong, L. Bao, W. Zuo, C. Shen, R. W. Lau, M.-H. Yang et al. (Vital: Visual tracking via adversarial learning, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 8990–8999.).

[0114] DP-Siam corresponds to the method proposed by MH Abdelpakey, MS Shehata et al. (Dp-siam: Dynamicpolicy siamese network for robust object tracking, IEEE Transactions on Image Processing 29(2019)1479–1492.).

[0115] Siam RPN++ corresponds to the method proposed by B. Li, W. Wu, Q. Wang, F. Zhang, J.

[0116] HIFT corresponds to the method proposed by Z. Cao, C. Fu, J. Ye, B. Li, Y. Li et al. (HiFT: Hierarchical feature transformer for aerial tracking, in: Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2021, pp. 1–10.);

[0117] Siam CAR corresponds to the method proposed by Dongyan Guo, Jun Wang, Ying Cui, Zhenhua Wang, Shengyong Chen, et al. (SiamCAR: Siamese Fully Convolutional Classification and Regression for Visual Tracking, In: CVPR (2020)).

[0118] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0119] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0120] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0121] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0122] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A target tracking method based on cross-layer feature calibration fusion, characterized in that, Includes the following steps: A. Construct a Siamese convolutional neural network, which includes a template branch and a search branch with shared weights; input the template region image and the search region image into the template branch and the search branch of the Siamese convolutional neural network, respectively; B. The modified feature extraction network ResNet50 is used to extract features from the template branch and the search branch, respectively. The extracted features include the features of the third convolutional block and the features of the fourth convolutional block. C. Use the attention-based feature fusion module to fuse the features of the third and fourth convolutional blocks of the template branch, and fuse the features of the third and fourth convolutional blocks of the search branch respectively; D. The template branch and the search branch respectively use the cross-layer calibration module to calibrate the features of the fourth convolutional block using the fused features; E. Perform deep cross-correlation operations on the features of the third convolutional block of the template branch and the search branch, the features fused from the template branch and the search branch, and the features calibrated from the template branch and the search branch, respectively, to obtain three similarity response maps; F. Combine the three similarity response maps and then perform dimensionality reduction. G. Input the dimensionality-reduced similarity response map into the prediction head for classification and regression, and finally obtain the location of the target.

2. The target tracking method based on cross-layer feature calibration fusion according to claim 1, characterized in that, In step B, the modified feature extraction network resnet50 is achieved by changing the effective stride of the last two layers of the resnet50 network to 8, and changing the convolution method of the last two layers to dilated convolution, so as to reduce the stride without shrinking the receptive field.

3. The target tracking method based on cross-layer feature calibration fusion according to claim 1, characterized in that, In step B, features of the third convolutional block of the template branch are extracted respectively. Features of the fourth convolutional block and the features of the third convolutional block in the search branch. Features of the fourth convolutional block ; In step C, an attention-based feature fusion module is used to fuse the features. and get and integration and get .

4. The target tracking method based on cross-layer feature calibration fusion according to claim 3, characterized in that, In step C, the specific implementation method of the attention-based feature fusion module is as follows: Given two input features a and b, we first perform a simple initial feature integration on a and b to obtain feature A, i.e. The obtained features are then input into local attention and non-local attention, respectively. in, and The kernel sizes are respectively and ; Indicates batch normalization, It is the Sigmoid function, where i represents the output position, and its response value is calculated by enumerating j; the F function is calculated. and The similarity between them; the G function calculates the representation of the feature map at position j, and the final NL(A) is normalized by the response factor D(A); Then, the features obtained through local attention and non-local attention are added bitwise, and after passing through an activation function, they are multiplied element-wise with the original feature A to produce refined features. : Where M(A) represents the attention weights obtained from local attention and non-local attention; This indicates element addition; This indicates element-wise multiplication; Finally, the characteristics of the fusion are represented as follows: The feature fusion module is used to fuse the features of the third and fourth convolutional blocks of the template branch and the search branch, respectively, as shown below: Where h represents the attention-based feature fusion function.

5. The target tracking method based on cross-layer feature calibration fusion according to claim 4, characterized in that, In step D, a cross-layer calibration module is used to calibrate deep features, and the features obtained after deep fusion are used to calibrate the features of the last residual block. Specifically: Each and As input Y, take and As input T; first, perform a feature transformation on Y based on S2: Then, we obtain vectors Q and B, as follows: Where f1, f2, f3, and f4 represent traditional 2D convolutional layers with different kernel sizes, and S1, S2, S3, and S4 represent convolutional kernels of different sizes; Up(.) is the bilinear interpolation operator; Where K represents the cross-layer calibration function.

6. The target tracking method based on cross-layer feature calibration fusion according to claim 5, characterized in that, In step E, deep cross-correlation operations are performed on the features of the third convolutional block of the template branch and the search branch, the features fused from the template branch and the search branch, and the features calibrated from the template branch and the search branch, respectively, to obtain three similarity response maps. The specific formulas are as follows: 。 7. The target tracking method based on cross-layer feature calibration fusion according to claim 1, characterized in that, In step F, the three similarity response maps are stitched together and then dimensionality is reduced using a 1×1 convolution.

8. A target tracking system based on cross-layer feature calibration fusion, characterized in that, It includes a memory, a processor, and computer program instructions stored in the memory and executable by the processor, which, when executed by the processor, enable the implementation of the steps of the method as described in any one of claims 1-7.

Images