A twin network long-range tracking method with enhanced positioning capability

Abstract

The application provides a twin network long-range tracking method for enhancing positioning capability, relates to the technical field of target tracking, and comprises the following steps: S1, a first frame of a target video is acquired, and a target template is obtained by cropping a labeled region of the first frame; S2, a target extraction branch of a twin network is used for feature extraction on the target template to obtain a target feature map; S3, a search region feature map is obtained; S4, the target feature map and the search region feature map are sent into a region proposal network to generate a series of regression boxes and corresponding target scores; S5, a regression box region corresponding to the highest target score is selected and put into a feature strengthening verifier to calculate a verification score; and S6, a sliding window is used to sequentially re-detect in a global picture, and the detected target is continuously tracked. The algorithm is based on the SPLT algorithm, the network structure of a re-detection module and a verifier is redesigned, and the problem that the SPLT algorithm is prone to tracking drift for a target is solved.

Description

Technical Field

[0001] This invention relates to the field of target tracking technology, and more particularly to a twin network long-range tracking method that enhances positioning capabilities. Background Technology

[0002] Object tracking is a crucial research problem in computer vision, with wide applications in video surveillance, behavior analysis, and visual navigation. Designing a robust and efficient tracker is a very challenging task, with major challenges including occlusion, lighting variations, rotation, and motion blur. Numerous algorithms have been proposed to address these challenges; however, most existing trackers focus on short-range tracking tasks, where the tracked object is almost always within the camera's field of view. Compared to traditional short-range tracking, long-range tracking—handling video with frame lengths reaching minutes and frequent object disappearance and reappearance—is more challenging and closer to real-world applications, thus attracting more research attention in recent years.

[0003] Because long-range tracking has more applications than short-range tracking, numerous long-range tracking algorithms have been proposed in recent years. The SPLT (Skimming-Perusal Long-term Tracking) algorithm is one such example. SPLT proposes a novel local tracking module that accurately captures the tracked object within a local search region. This module consists of an efficient bounding box regressor based on SiameseRPN and a robust offline-trained validator based on deep feature embedding. Furthermore, the algorithm designs a re-detection module that effectively selects the most probable local region from a densely sampled sliding window, accelerating the re-detection process within the image range when the target is missing.

[0004] In actual testing, it was found that the SPLT algorithm is prone to tracking drift when re-detecting the target because the re-detection module cannot accurately locate the target. Furthermore, the validator cannot effectively distinguish between the background and the target, resulting in tracking the background and thus affecting the overall performance of the target tracking algorithm. Summary of the Invention

[0005] In view of this, the purpose of this invention is to propose a twin network long-range tracking method to enhance positioning capabilities, so as to solve the technical problem that tracking drift easily occurs when using existing re-detection modules to detect targets after they disappear and reappear in existing long-range tracking.

[0006] The technical means employed in this invention are as follows:

[0007] A long-range tracking method using twin networks to enhance localization capabilities includes the following steps:

[0008] S1. Obtain the first frame of the target video, and crop the marked area of ​​the first frame to obtain the target template;

[0009] S2. Use the target extraction branch of the Siamese network to extract features from the target template to obtain a target feature map;

[0010] S3. During the tracking phase, subsequent frames are cropped to form the search region, and the features of the search region are extracted to obtain the search region feature map.

[0011] S4. The target feature map and the search region feature map are fed into the region proposal network to generate a series of regression boxes and corresponding target scores;

[0012] S5. Select the regression box region corresponding to the highest target score and put it into the feature enhancement validator to calculate the validation score;

[0013] If the obtained validator score is less than the set validator threshold, continue tracking;

[0014] When the obtained validator score is greater than the set validator threshold, it is determined that the target may have disappeared, and a center search is used. When the center search does not find the target, the feature enhancement re-detection module is used to search for the target.

[0015] S6. In the feature enhancement re-detection method, a sliding window is used to re-detect the global image in sequence, and the detected targets are continued to be tracked.

[0016] Furthermore, the regression box region corresponding to the highest target score is placed into the feature enhancement validator to calculate the validation score, including the following steps:

[0017] The feature-enhanced validator uses ResNet50 to extract features;

[0018] Multi-branch, multi-scale convolution is performed on the Block2, Block3, and Block4 features respectively;

[0019] The obtained multi-scale features are then fused together with features of the same scale.

[0020] The merged features are pooled.

[0021] By implementing full connectivity, the tracker's ability to make judgments is improved, thereby enhancing its tracking performance.

[0022] Furthermore, multi-branch, multi-scale convolutions are performed on the Block2, Block3, and Block4 features, including the following steps:

[0023] The features extracted from Block 2 undergo multi-branch processing. Branch 1 consists of 512 convolutional operations of size 1x1 with a stride of 1; Branch 2 consists of pooling operations of size 2x2 with a stride of 2, followed by 1024 convolutional operations of size 1x1 with a stride of 1. The features extracted from Block 3 also undergo multi-branch processing. Branch 1 involves upsampling by doubling the row and column size, followed by 512 convolutional operations of size 1x1 with a stride of 1; Branch 2 consists of 1x2 pooling operations of size 2x2 with a stride of 2, and then 1024 convolutional operations of size 1x1 with a stride of 1. The first branch performs 24 convolutional operations of size 1*1 with a stride of 1; the second branch performs 2048 convolutional operations of size 1*1 with a stride of 1. The features extracted by Block4 are processed in multiple branches. The first branch performs upsampling with rows and columns expanded by a factor of 2, followed by 1024 convolutional operations of size 1*1 with a stride of 1; the third branch performs 2048 convolutional operations of size 1*1 with a stride of 1.

[0024] Furthermore, the steps for finding targets using the feature enhancement re-detection module include:

[0025] The features extracted by MobileNet are upsampled. The upsampling branch consists of three convolutional layers: 1024 convolutional kernels of size 3*3 with a stride of 2; an upsampled convolutional layer with rows and columns expanded by two times; and 512 convolutional kernels of size 1*1 with a stride of 1. These layers are fused with the original feature layers extracted by MobileNet to improve the network's localization ability.

[0026] Furthermore, an asymmetric convolution branch is introduced, which consists of two convolutional layers;

[0027] The asymmetric convolution branch includes 512 convolution operations of size 1*3 with a stride of 1 and 512 convolution operations of size 3*1 with a stride of 1.

[0028] The asymmetric convolution branch is used to enhance the features of the target center region to improve tracking and localization capabilities.

[0029] The present invention also provides a storage medium comprising a stored program, wherein, when the program is executed, any of the above-described twin network long-range tracking methods for enhancing positioning capabilities are performed.

[0030] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the twin network long-range tracking method described above for enhancing positioning capabilities through the computer program.

[0031] Compared with the prior art, the present invention has the following advantages:

[0032] In the re-detection stage, this invention enhances the features extracted by MobileNet using three cascaded operation layers. These layers consist of 1024 convolutional operations of size 3x3 with a stride of 2, upsampling convolutional operations that double the row and column size, and 512 convolutional operations of size 1x1 with a stride of 1. These enhancements are then fused with the original feature layers extracted by MobileNet to improve the network's localization capabilities.

[0033] This invention introduces an asymmetric convolution module. The asymmetric convolution branch consists of two convolutional layers, namely 512 convolutional operations of size 1*3 with a stride of 1 and 512 convolutional operations of size 3*1 with a stride of 1. This asymmetric convolution branch is used to enhance the features of the target center region, thereby further improving the localization capability.

[0034] The validator part of this invention first extracts features using ResNet50. Then, it performs multi-branch, multi-scale convolution operations on the Block 2, Block 3, and Block 4 features respectively. Next, it fuses features of the same scale in the obtained multi-scale features. Furthermore, it performs pooling operations on the fused features. Finally, it performs a fully connected layer. This improves the tracker's judgment ability and further enhances the algorithm's performance. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 This is an overall flowchart of the method of the present invention;

[0037] Figure 2 This is a diagram of the feature enhancement verifier network structure of the improved method of the present invention;

[0038] Figure 3 This is a network structure diagram of the feature enhancement re-detection module in the improved method of the present invention;

[0039] Figure 4 This is a first tracking effect diagram of the method of the present invention.

[0040] Figure 5 This is a diagram showing the second tracking effect of the method of the present invention.

[0041] Figure 6 This is a diagram showing the third tracking effect of the method of the present invention.

[0042] Figure 7 This is a diagram showing the fourth tracking effect of the method of the present invention. Detailed Implementation

[0043] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0044] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0045] like Figures 1 to 7 As shown, this invention provides a long-range tracking method using twin networks to enhance positioning capabilities. The specific steps are as follows:

[0046] Step 1: Obtain the first frame of the video. Crop the marked area in the first frame to a size of 128*128, and use it as the target template, i.e., the real target. The marked area is a region determined by the coordinates of the top-left point of the given real target bounding box, as well as its width and height.

[0047] Step 2: Extract features from the cropped target using the target extraction branch of the Siamese network to obtain the target feature map. The feature map is 1*1 in size and has 512 channels.

[0048] Step 3: During the tracking phase, subsequent frames are cropped to serve as the search region. The cropped size is 300*300. Features of the search region are extracted to obtain the feature map of the search region. The feature map size is 19*19 and the number of channels is 512.

[0049] Step 4: Feed the feature maps of the target and the search region into the Region Proposal Network (RPN) to generate a series of bounding boxes and corresponding target scores. The local tracker uses the SiamRPN network, extracts features using MobileNet, and further utilizes the RPN to generate scores and bounding boxes.

[0050] Step 5: Select the regression box region corresponding to the highest target score and place it in the validator. If the obtained validator score is less than the set validator threshold, continue tracking. If the validator score is greater than the set validator threshold, it is determined that the target may have disappeared, and a center search is used. If the center search does not find the target, then the re-detection module is used to search for the target. Figure 2 As shown, the validator first extracts features using ResNet50. Then, it performs multi-branch, multi-scale convolutions on the Block 2, Block 3, and Block 4 features. Specifically, the features extracted from Block 2 undergo multi-branch processing: branch one consists of 512 convolutions of size 1x1 with a stride of 1; branch two consists of pooling operations of size 2x2 with a stride of 2, followed by 1024 convolutions of size 1x1 with a stride of 1. Similarly, the features extracted from Block 3 undergo multi-branch processing: branch one involves upsampling by doubling the row and column size, followed by 512 convolutions of size 1x1 with a stride of 1; branch two consists of 1024 convolutions of size 1x1 with a stride of 1; and branch three consists of pooling operations of size 2x2 with a stride of 2, followed by 2048 convolutions of size 1x1 with a stride of 1. The features extracted by Block4 undergo multi-branch processing. Branch one involves upsampling by a factor of two in both rows and columns, followed by 1024 convolutional operations of size 1x1 with a stride of 1. Branch two involves 2048 convolutional operations of size 1x1 with a stride of 1. Next, features of the same scale are fused from the obtained multi-scale features. Further, the fused features are pooled, and finally, a fully connected layer is applied. During validator training, the dataset used is ILSVRC2015_VID, which requires processing and saving as a TFRecord. The epoch is set to 70, the batch size to 32, and the learning rate to 0.005.

[0051] Step 6: In the re-detection phase, a sliding window is used to re-detect targets sequentially across the entire image. Detected targets continue to be tracked. For example... Figure 3As shown, in the re-detection stage, the features extracted by MobileNet are enhanced using three cascaded operation layers: 1024 convolutional operations of size 3*3 with a stride of 2, upsampling convolutional operations that double the row and column size, and 512 convolutional operations of size 1*1 with a stride of 1. These are then fused with the original feature layers extracted by MobileNet to improve the network's localization ability. Furthermore, an asymmetric convolution module is introduced, consisting of two convolutional layers: 512 convolutional operations of size 1*3 with a stride of 1, and 512 convolutional operations of size 3*1 with a stride of 1. This asymmetric convolution branch is used to enhance the features of the target center region, further improving the localization ability. During training, the dataset used is ILSVRC2015_VID, with an epoch of 45, a batch size of 32, a learning rate of 0.001, a momentum optimizer with a momentum of 0.9, and a binary cross-entropy loss function.

[0052] The present invention also provides a storage medium comprising a stored program, wherein, when the program is executed, a twin network long-range tracking method for enhancing positioning capabilities is performed.

[0053] The present invention also provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes a twin network long-range tracking method for enhanced positioning capabilities through the computer program.

[0054] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0055] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0056] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0057] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0058] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0059] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0060] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A long-range tracking method using twin networks to enhance positioning capabilities, characterized in that, Includes the following steps: S1. Obtain the first frame of the target video, and crop the marked area of ​​the first frame to obtain the target template; S2. Use the target extraction branch of the Siamese network to extract features from the target template to obtain a target feature map; S3. During the tracking phase, subsequent frames are cropped to form the search region, and the features of the search region are extracted to obtain the search region feature map. S4. The target feature map and the search region feature map are fed into the region proposal network to generate a series of regression boxes and corresponding target scores; S5. Select the regression box region corresponding to the highest target score and put it into the feature enhancement validator to calculate the validation score; If the obtained validator score is less than the set validator threshold, continue tracking; When the obtained validator score is greater than the set validator threshold, it is determined that the target may have disappeared, and a center search is used. When the center search does not find the target, the feature enhancement re-detection module is used to search for the target. S6. In the feature enhancement re-detection method, a sliding window is used to re-detect the global image in sequence, and the detected targets are continued to be tracked. The region of the regression box corresponding to the highest target score is placed into the feature enhancement validator to calculate the validation score, including the following steps: The feature-enhanced validator uses ResNet50 to extract features; Multi-branch, multi-scale convolution is performed on the Block2, Block3, and Block4 features respectively; The obtained multi-scale features are then fused together with features of the same scale. The merged features are pooled. By implementing full connectivity, the tracker's ability to make judgments is improved, thereby enhancing the tracker's tracking performance; Multi-branch, multi-scale convolutions are performed on the Block2, Block3, and Block4 features, including the following steps: The features extracted from Block2 are processed through multiple branches. Branch 1 consists of 512 branches with a size of 1.

1. A convolution operation with a stride of 1; branch two is a convolution operation with a stride of 2.

2. A pooling operation with a step size of 2 is performed, followed by 1024 pools of size 1.

1. A convolution operation with a stride of 1 is performed; the features extracted by Block 3 are processed through multiple branches. Branch one involves upsampling by doubling the row and column expansion, followed by 512 passes of size 1.

1. A convolution operation with a stride of 1; branch two consists of 1024 convolutions of size 1.

1. A convolution operation with a stride of 1; branch three is a convolution operation with a stride of 2.

2. A pooling operation with a step size of 2 is performed, followed by 2048 pools of size 1.

1. A convolution operation with a stride of 1 is performed; the features extracted by Block 4 are processed through multiple branches. Branch one involves upsampling by doubling the row and column expansion, followed by 1024 passes of size 1.

1. A convolution operation with a stride of 1; branch two consists of 2048 convolutions of size 1.

1. Convolution operation with a stride of 1.

2. The twin network long-range tracking method for enhanced positioning capability according to claim 1, characterized in that, The steps for finding targets using the feature-enhanced re-detection module include: The features extracted by MobileNet are upsampled. The upsampling branch has three convolutional layers, each consisting of 1024 convolutional layers of size 3.

3. Convolutional kernels with a stride of 2; upsampling convolutional layers with rows and columns expanded by a factor of 2; 512 convolutional layers of size 1 1. It consists of convolutional kernels with a stride of 1, which are fused with the original feature layers extracted by MobileNet to improve the network's localization capability.

3. The twin network long-range tracking method for enhanced positioning capability according to claim 2, characterized in that, An asymmetric convolutional branch is introduced into the re-detection module, which consists of two convolutional layers. The asymmetric convolutional branches include 512 branches of size 1.

3. Convolution operations with a stride of 1 and 512 operations of size 3 1. Convolution operation with a stride of 1; The asymmetric convolution branch is used to enhance the features of the target center region to improve tracking and localization capabilities.

4. A storage medium, characterized in that, The storage medium includes a stored program, wherein, when the program is executed, it performs the twin network long-range tracking method for enhanced positioning capability as described in any one of claims 1 to 3.

5. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the twin network long-range tracking method for enhanced positioning capabilities as described in any one of claims 1 to 3 through the computer program.

Images