Single target tracking method, system, electronic device and medium

Abstract

The present invention provides a single target tracking method, system, electronic device and medium, which belongs to the field of computer vision technology. The method adopts a weight-sharing Swim-Transformer as the backbone network to extract features from templates, dynamic templates and search areas. After improving the feature quality through the feature enhancement module, the enhanced features are input into the spatiotemporal attention Transformer encoding and decoding layers for fusion. The fused features are sent to the classification and regression branches to jointly obtain the tracking results. In addition, the system regularly checks whether the preset update cycle has been reached, and evaluates the detection results based on the classification confidence, and updates the initial template with the cropped tracking results when necessary. The present invention can effectively cope with complex scenarios such as target occlusion and environmental changes, significantly improve tracking accuracy and robustness, is applicable to a variety of application scenarios, and improves the practicality and reliability of computer vision technology.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a single target tracking method, system, electronic equipment and medium. Background Art

[0002] With the widespread adoption of technologies such as surveillance cameras, smartphones, and autonomous driving, real-time and accurate target tracking has become a crucial research topic in computer vision. In this context, single-target tracking, as a major branch of target tracking research, has attracted widespread attention. The task of single-target tracking is to locate the target in the first frame of a video and accurately locate its position and trajectory in subsequent frames. Traditional target tracking methods, including frame difference methods, optical flow methods, and Kalman filtering, often rely on hand-crafted features. Hand-crafted features not only require extensive expert knowledge but also perform poorly in adapting to complex scene changes, making them unable to meet increasingly demanding tracking requirements. Neural networks are widely used in the current target tracking field due to their excellent feature extraction capabilities, good generalization, and high detection accuracy.

[0003] Transformer, as an emerging neural network structure, was first proposed in 2017. Its revolutionary feature lies in the introduction of the self-attention mechanism, which greatly improves the efficiency and performance of sequence data processing. Subsequently, in 2020, Transformer was successfully introduced into the field of computer vision. By using the self-attention mechanism to process image tasks, it can excellently capture the global contextual information of the image, thus demonstrating great application potential in this field. Although the Transformer model performs well in processing sequence data, it lacks modeling of spatiotemporal information and local information when processing continuous frame image sequences, which can easily lead to problems such as unstable tracking targets and inability to fully capture the motion characteristics of the target, thereby reducing tracking accuracy. At the same time, Transformer-based single-target trackers usually only use the information of the initial template, which means that tracking is easily affected by target occlusion or environmental changes, further reducing the accuracy of the tracking results.

[0004] Therefore, how to effectively respond to the above technical challenges has become a difficult problem that practitioners in this field urgently need to solve. Summary of the Invention

[0005] The purpose of the present invention is to provide a single target tracking method, system, electronic device and medium, which can effectively cope with complex scenarios such as target occlusion and environmental changes, and improve tracking accuracy and robustness.

[0006] To achieve the above object, the present invention provides a single target tracking method, comprising the following steps:

[0007] Step S1: Use the weight-sharing Swim-Transformer as the backbone network for feature extraction to extract features from the template, dynamic template, and search area;

[0008] Step S2: enhancing the extracted features through a feature enhancement module;

[0009] Step S3: input the enhanced search frame feature sequence, template frame feature sequence, and dynamic template frame feature sequence into the spatiotemporal attention Transformer encoding layer and Transformer decoding layer for feature fusion;

[0010] Step S4: The fused features are input to the classification branch and the regression branch at the same time, and the two branches jointly obtain the tracking results;

[0011] Step S5: Check whether the preset update cycle has been reached. When the update cycle has been reached, evaluate the detection results based on the confidence of the classification, and determine whether to crop the tracking results to replace the initial template. When the update cycle has not been reached, the dynamic template remains unchanged, and the system continues to wait for the next frame of data to trigger subsequent updates.

[0012] Preferably, in step S1, Swim-Transformer is used as the backbone network for feature extraction to perform feature extraction on the template, dynamic template, and search area. The specific operations are as follows:

[0013] When processing a video sequence, we first check whether the current frame is the first frame. If it is the first frame, we set it as the initial dynamic template. The template, dynamic template, and search area are input into the Swim-Transformer backbone network for feature extraction.

[0014] Preferably, in step S2, the feature enhancement module includes three dilated convolution branches with different dilation rates and a global adaptive average pooling branch;

[0015] The three dilated convolution branches all use 3×3 convolution kernels with dilation rates of 1, 2, and 4, respectively. The global adaptive average pooling branch includes a layer of global adaptive average pooling, a layer of batch normalization, a ReLU activation function, and an upsampling operation. The outputs of the four branches are concatenated, integrated through 1×1 convolution, and finally output.

[0016] Preferably, in step S3, the spatiotemporal attention Transformer encoding layer introduces a spatiotemporal attention hybrid module to enhance the extraction of spatiotemporal information; the search frame feature sequence, the template frame feature sequence, and the dynamic template frame feature sequence are input into the spatiotemporal attention hybrid module after sinusoidal position encoding, and are subjected to residual connection and layer normalization processing; the output features are spliced ​​and input into multiple Transformer encoding layers to further optimize the features. The formula for the whole process is:

[0017] Z′=LN(Z+STAM(Z)) (1);

[0018] X′=LN(X+STAM(X)) (2);

[0019] Y′=LN(Y+STAM(Y)) (3);

[0020] U 1 =Concat(Z′,X′,Y′) (4);

[0021] U L+1 =LN(LN(U L +MSA(U L ))+FFN(LN(U L +MSA(U L )))) (5);

[0022] Among them, Z, X, Y represent the search frame feature sequence, template frame feature sequence and dynamic template frame feature sequence; Z′, X′, Y′ represent the results of the search frame feature sequence, template frame feature sequence and dynamic template frame feature sequence after being processed by the spatiotemporal attention hybrid module, residual connection and layer normalization; U 1 Indicates the result after feature splicing; U L+1 Represents the output of the Lth Transformer encoding layer without embedding operation; STAM represents the spatiotemporal attention hybrid module operation; MSA represents the multi-head self-attention operation; LN represents layer normalization; FFN represents the feedforward neural network.

[0023] Preferably, the spatiotemporal hybrid attention module includes a temporal self-attention module and a spatial self-attention module; the input features are sent to these two modules at the same time to extract temporal and spatial information respectively; then, the features extracted by the two modules are added to obtain the final output result of the spatiotemporal hybrid attention module.

[0024] Preferably, in step S5, when the update cycle is reached, the detection result is evaluated according to the confidence of the classification, and it is determined whether to crop the tracking result to replace the initial template. The specific operation is as follows:

[0025] Extract the highest confidence score from the classification results and compare the highest confidence score with a preset threshold;

[0026] When the highest confidence exceeds the threshold, the tracking result is cropped according to the template format and the dynamic template is updated;

[0027] When the highest confidence score is lower than the threshold, no further action is taken and the processing continues directly to the next frame.

[0028] The present invention also provides a single target tracking system, comprising:

[0029] The feature extraction module uses the weight-sharing Swim-Transformer as the backbone network for feature extraction to extract features from the template, dynamic template, and search area.

[0030] A feature enhancement module is used to enhance the features extracted by the feature extraction module;

[0031] The feature fusion module is used to input the enhanced search frame feature sequence, template frame feature sequence, and dynamic template frame feature sequence into the spatiotemporal attention Transformer encoding layer and Transformer decoding layer for feature fusion;

[0032] The prediction head module receives the fused features and inputs them into the classification branch and the regression branch. The two branches jointly produce the tracking results.

[0033] The update decision module is used to check whether the preset update cycle has been reached, evaluate the detection results based on the confidence of the classification, and determine whether to crop the tracking results to replace the initial template.

[0034] The present invention also provides a computer device, comprising: a memory and a processor; the memory stores a computer program, and the processor implements the steps of the above-mentioned single target tracking method when executing the computer program.

[0035] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned single target tracking method are implemented.

[0036] Therefore, the present invention employs the aforementioned single-target tracking method, system, electronic device, and medium. Technical effects: The present invention addresses the problem that Transformer single-target trackers tend to ignore spatiotemporal and local information. The method comprises three components: a Swim-Transformer-based feature extraction network, a feature fusion network based on a spatiotemporal attention Transformer encoder and decoder, and a prediction head. First, the search region, template, and dynamic template are input into the Swim-Transformer backbone network for feature extraction. The extracted features are then enhanced using a feature enhancement module to ensure that both global and local information are considered during feature extraction. The enhanced features are then input into a feature fusion network based on a spatiotemporal attention Transformer encoder and decoder. This designed feature fusion network not only achieves multimodal fusion but also, through the introduction of a spatiotemporal attention hybrid module, enables the fusion network to effectively focus on temporal and spatial information in the sequence, thereby improving tracking accuracy. The fused features are then fed into the prediction head, which comprises two branches: classification and regression. These two branches share the same input features and ultimately generate tracking results. The classification confidence level determines whether the search results need to be trimmed and the initial template replaced. Dynamic templates are introduced to achieve a dynamic loop in the tracking process, enhancing the integration of temporal information. This method, based on the Transformer single-target tracking method, incorporates spatiotemporal and local features, achieving superior tracking results. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is a flow chart of a single target tracking method of the present invention;

[0038] Figure 2 This is the structure diagram of the spatiotemporal attention Transformer encoding layer;

[0039] Figure 3 It is the feature extraction module. DETAILED DESCRIPTION

[0040] The technical solution of the present invention is further described below with reference to the accompanying drawings and embodiments.

[0041] Unless otherwise defined, technical or scientific terms used in the present invention shall have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs.

[0042] Example 1

[0043] like Figure 1 FIG. 1 is a flow chart of a single target tracking method according to the present invention, which specifically includes the following steps:

[0044] Step S1: Use the weight-sharing Swim-Transformer as the backbone network for feature extraction to extract features from the template, dynamic template, and search area. The specific operations are as follows:

[0045] When processing a video sequence, the first step is to check whether the current frame is the first one. If not, the update cycle is determined. If so, the highest confidence score is obtained based on the classification output, and it is determined whether it exceeds a set threshold. If it is above the threshold, the tracking results are cropped according to the template format, the dynamic template is updated, and it is input into the Swim-Transformer backbone network to update the dynamic template features. If it is below the threshold, the dynamic template is not updated and the next frame is processed. If it is the first frame, it is set as the initial dynamic template. The template, dynamic template, and search area are input into the Swim-Transformer backbone network for feature extraction.

[0046] The Swin-Transformer network serves as the backbone for feature extraction and performs object tracking in three stages. Stage 1 extracts preliminary features from the input image and focuses on the target region. Stage 2 models target features and integrates context information to enhance the relationship between the target and the background. Stage 3 utilizes the optimized features for precise target positioning and tracking, generating the final target bounding box position and related tracking information.

[0047] Step S2: Enhance the extracted features through a feature enhancement module.

[0048] like Figure 3 As shown in the figure, the feature enhancement module includes three dilated convolution branches with different dilation rates and a global adaptive average pooling branch;

[0049] The three dilated convolution branches all use 3×3 convolution kernels with dilation rates of 1, 2, and 4, respectively. The global adaptive average pooling branch includes a layer of global adaptive average pooling, a layer of batch normalization, a ReLU activation function, and an upsampling operation. The outputs of the four branches are concatenated, integrated through 1×1 convolution, and finally output.

[0050] Step S3: Input the enhanced search frame feature sequence, template frame feature sequence, and dynamic template frame feature sequence into the spatiotemporal attention Transformer encoding layer and Transformer decoding layer for feature fusion.

[0051] like Figure 2As shown in the figure, the spatiotemporal attention Transformer encoding layer introduces a spatiotemporal attention hybrid module to enhance the extraction of spatiotemporal information; the search frame feature sequence, template frame feature sequence, and dynamic template frame feature sequence are input into the spatiotemporal attention hybrid module after sinusoidal position encoding, and are subjected to residual connection and layer normalization processing; the output features are spliced ​​and input into multiple Transformer encoding layers to further optimize the features. The formula for the whole process is:

[0052] Z′=LN(Z+STAM(Z)) (1);

[0053] X′=LN(X+STAM(X)) (2);

[0054] Y′=LN(Y+STAM(Y)) (3);

[0055] U 1 =Concat(Z′,X′,Y′) (4);

[0056] U L+1 =LN(LN(U L +MSA(U L ))+FFN(LN(U L +MSA(U L )))) (5);

[0057] Among them, Z, X, Y represent the search frame feature sequence, template frame feature sequence and dynamic template frame feature sequence; Z′, X′, Y′ represent the results of the search frame feature sequence, template frame feature sequence and dynamic template frame feature sequence after being processed by the spatiotemporal attention hybrid module, residual connection and layer normalization; U 1 Indicates the result after feature splicing; U L+1 Represents the output of the Lth Transformer encoding layer without embedding operation; STAM represents the spatiotemporal attention hybrid module operation; MSA represents the multi-head self-attention operation; LN represents layer normalization; FFN represents the feedforward neural network.

[0058] The spatiotemporal hybrid attention module includes a temporal self-attention module and a spatial self-attention module; the input features are fed into these two modules at the same time to extract temporal and spatial information respectively; then, the features extracted by the two modules are added together to obtain the final output result of the spatiotemporal hybrid attention module.

[0059] Step S4: The fused features are input to the classification branch and the regression branch at the same time, and the two branches jointly obtain the tracking results.

[0060] Step S5: Check whether the preset update period has been reached.

[0061] When the update cycle is reached, the highest confidence score is extracted from the classification results and compared with the preset threshold;

[0062] When the highest confidence exceeds the threshold, the tracking result is cropped according to the template format and the dynamic template is updated;

[0063] When the highest confidence score is lower than the threshold, no further action is taken and the processing continues directly to the next frame.

[0064] When the update period is not reached, the dynamic template remains unchanged and the system continues to wait for the next frame of data to trigger subsequent updates.

[0065] Example 2

[0066] A single target tracking system, comprising:

[0067] The feature extraction module uses the weight-sharing Swim-Transformer as the backbone network for feature extraction to extract features from the template, dynamic template, and search area.

[0068] A feature enhancement module is used to enhance the features extracted by the feature extraction module;

[0069] The feature fusion module is used to input the enhanced search frame feature sequence, template frame feature sequence, and dynamic template frame feature sequence into the spatiotemporal attention Transformer encoding layer and Transformer decoding layer for feature fusion;

[0070] The prediction head module receives the fused features and inputs them into the classification branch and the regression branch. The two branches jointly produce the tracking results.

[0071] The update decision module is used to check whether the preset update cycle has been reached, evaluate the detection results based on the confidence of the classification, and determine whether to crop the tracking results to replace the initial template.

[0072] If the above functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0073] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0074] More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering, or processing in another suitable manner as necessary, and then stored in a computer memory.

[0075] It is worth noting that the contents not elaborated in detail in the present invention are all prior art and are well known to those skilled in the art.

[0076] Therefore, the present invention adopts the above-mentioned single target tracking method, system, electronic device and medium. First, a dynamic template update strategy is used to allow the tracker to dynamically update the template according to the target's motion and appearance changes, thereby maintaining the accuracy and stability of tracking and improving the tracker's adaptability to the target. Second, a feature enhancement module is introduced to help capture the key features of the target and further optimize the performance of the tracker. Third, a spatiotemporal attention mechanism is introduced into the feature fusion network, enabling it to effectively capture the dynamic changes of the target in time and space, helping the tracker to better understand the target's motion trajectory and appearance changes, and enhancing the tracker's generalization ability in complex situations.

[0077] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit the same. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solutions of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A single target tracking method, characterized in that: The following steps are involved: Step S1: Use the weight-sharing Swim-Transformer as the backbone network for feature extraction to extract features from the template, dynamic template, and search area; Step S2: enhancing the extracted features through a feature enhancement module; Step S3: input the enhanced search frame feature sequence, template frame feature sequence, and dynamic template frame feature sequence into the spatiotemporal attention Transformer encoding layer and Transformer decoding layer for feature fusion; Step S4: The fused features are input to the classification branch and the regression branch at the same time, and the two branches jointly obtain the tracking results; Step S5: Check whether the preset update cycle has been reached. If the update cycle has been reached, evaluate the detection results based on the confidence of the classification and determine whether to crop the tracking results and replace the initial template. If the update cycle has not been reached, the dynamic template remains unchanged and the system continues to wait for the next frame of data to trigger a subsequent update. The feature enhancement module includes three dilated convolution branches with different dilation rates and a global adaptive average pooling branch; The three dilated convolution branches all use a 3×3 convolution kernel with dilation rates of 1, 2, and 4, respectively. The global adaptive average pooling branch includes a layer of global adaptive average pooling, a layer of batch normalization, a ReLU activation function, and an upsampling operation. The outputs of the four branches are concatenated, integrated through a 1×1 convolution, and finally output. In step S3, the spatiotemporal attention Transformer encoding layer introduces a spatiotemporal attention hybrid module to enhance the extraction of spatiotemporal information; the search frame feature sequence, template frame feature sequence, and dynamic template frame feature sequence are input into the spatiotemporal attention hybrid module after sinusoidal position encoding, and are subjected to residual connection and layer normalization processing; the output features are spliced ​​and input into multiple Transformer encoding layers to further optimize the features. The formula for the whole process is: （1）； （2）； （3）； （4）； （5）； in, 、 、 Represents search frame feature sequence, template frame feature sequence and dynamic template frame feature sequence; 、 、 Represents the results of the search frame feature sequence, template frame feature sequence, and dynamic template frame feature sequence after being processed by the spatiotemporal attention hybrid module, residual connection, and layer normalization; Represents the result after feature splicing; represents the output of the Lth Transformer encoding layer without embedding operation; Represents the spatiotemporal attention hybrid module operation; Represents a multi-head self-attention operation; Representation layer normalization; represents a feedforward neural network.

2. A single target tracking method according to claim 1, characterized in that: In step S1, Swim-Transformer is used as the backbone network for feature extraction to extract features from the template, dynamic template, and search area. The specific operations are as follows: When processing a video sequence, we first check whether the current frame is the first frame. If the current frame is the first frame, it is set as the initial dynamic template; the template, dynamic template and search area are input into the Swim-Transformer backbone network for feature extraction.

3. A single target tracking method according to claim 1, characterized in that: The spatiotemporal hybrid attention module includes a temporal self-attention module and a spatial self-attention module; The input features are fed into these two modules at the same time to extract temporal and spatial information respectively; then, the features extracted by the two modules are added together to obtain the final output result of the spatiotemporal hybrid attention module.

4. A single target tracking method according to claim 1, characterized in that: In step S5, when the update period is reached, the detection result is evaluated according to the confidence of the classification, and it is determined whether to crop the tracking result and replace the initial template. The specific operations are as follows: Extract the highest confidence score from the classification results and compare the highest confidence score with a preset threshold; When the highest confidence exceeds the threshold, the tracking result is cropped according to the template format and the dynamic template is updated; When the highest confidence score is lower than the threshold, no further action is taken and the processing continues directly to the next frame.

5. A single target tracking system, characterized in that: Used to perform the single target tracking method according to any one of claims 1 to 4, comprising: The feature extraction module uses the weight-sharing Swim-Transformer as the backbone network for feature extraction to extract features from the template, dynamic template, and search area. A feature enhancement module is used to enhance the features extracted by the feature extraction module; The feature fusion module is used to input the enhanced search frame feature sequence, template frame feature sequence, and dynamic template frame feature sequence into the spatiotemporal attention Transformer encoding layer and Transformer decoding layer for feature fusion; The prediction head module receives the fused features and inputs them into the classification branch and the regression branch. The two branches jointly produce the tracking results. The update decision module is used to check whether the preset update cycle has been reached, and evaluate the detection results based on the confidence of the classification to determine whether to crop the tracking results and replace the initial template.

6. A computer device comprising: memory and processor; The memory stores a computer program, wherein the processor implements the steps of the single target tracking method according to any one of claims 1 to 4 when executing the computer program.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the single target tracking method according to any one of claims 1 to 4 are implemented.

Images