Single-target tracking method and system, electronic equipment and medium

Abstract

The invention provides a single-target tracking method and system, electronic equipment and a medium, and belongs to the technical field of computer vision. According to the method, a Swim-Transform of weight sharing is adopted as a backbone network, and feature extraction is carried out on a template, a dynamic template and a search area. And after the feature quality is improved through a feature enhancement module, enhanced features are input into a space-time attention Transform coding and decoding layer for fusion. Fusion features are sent to classification and regression branches, and a tracking result is obtained together. In addition, the system regularly checks whether a preset updating period is reached or not, evaluates a detection result according to classification confidence, and updates the initial template by using a cut tracking result when necessary. The method can effectively cope with complex scenes such as target shielding and environment change, remarkably improves the tracking precision and robustness, is suitable for various application scenes, and improves the practicability and reliability of the computer vision technology.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a single target tracking method, system, electronic equipment and medium. Background Art

[0002] With the popularization of technologies such as surveillance cameras, smart phones, and autonomous driving, real-time and accurate target tracking has become an important research topic in the field of computer vision. In this context, single target tracking, as the main branch of target tracking research, has attracted widespread attention. The task of single target tracking is to calibrate the target in the first frame of the video and accurately locate the position and trajectory of the target in subsequent video frames. Traditional target tracking methods include frame difference method, optical flow method, and Kalman filtering, which often rely on manually designed features. Manually designed features not only require a lot of expert knowledge, but also perform poorly in adapting to complex scene changes, and cannot meet the increasingly stringent tracking requirements. Neural networks are widely used in the current target tracking field due to their excellent feature extraction ability, good generalization and high detection accuracy.

[0003] Transformer, as an emerging neural network structure, was first proposed in 2017. Its revolutionary feature lies in the introduction of the self-attention mechanism, which greatly improves the efficiency and performance of sequence data processing. Subsequently, in 2020, Transformer was successfully introduced into the field of computer vision. By using the self-attention mechanism to process image tasks, it can excellently capture the global contextual information of the image, thus demonstrating great application potential in this field. Although the Transformer model performs well in processing sequence data, it lacks modeling of spatiotemporal information and local information when processing continuous frame image sequences, which can easily lead to problems such as unstable tracking targets and inability to fully capture the motion characteristics of the target, thereby reducing tracking accuracy. At the same time, Transformer-based single target trackers usually only use the information of the initial template, which means that tracking is easily affected by target occlusion or environmental changes, further reducing the accuracy of tracking results.

[0004] Therefore, how to effectively respond to the above technical challenges has become a difficult problem that practitioners in this field urgently need to solve. Summary of the invention

[0005] The purpose of the present invention is to provide a single target tracking method, system, electronic device and medium, which can effectively cope with complex scenes such as target occlusion and environmental changes, and improve tracking accuracy and robustness.

[0006] To achieve the above object, the present invention provides a single target tracking method, comprising the following steps:

[0007] Step S1, using the weight-sharing Swim-Transformer as the backbone network for feature extraction, and performing feature extraction on the template, dynamic template and search area;

[0008] Step S2, enhancing the extracted features through a feature enhancement module;

[0009] Step S3, inputting the enhanced search frame feature sequence, template frame feature sequence and dynamic template frame feature sequence into the spatiotemporal attention Transformer encoding layer and Transformer decoding layer for feature fusion;

[0010] Step S4: The fused features are simultaneously input into the classification branch and the regression branch, and the two branches jointly obtain the tracking result;

[0011] Step S5, check whether the preset update cycle is reached. When the update cycle is reached, evaluate the detection result according to the confidence of the classification, and determine whether to crop the tracking result to replace the initial template; when the update cycle is not reached, the dynamic template remains unchanged, and the system continues to wait for the next frame of data to trigger subsequent updates.

[0012] Preferably, in step S1, Swim-Transformer is used as the backbone network for feature extraction to extract features from the template, dynamic template and search area. The specific operations are as follows:

[0013] When processing a video sequence, we first check whether the current frame is the first frame. If it is the first frame, we set it as the initial dynamic template; the template, dynamic template and search area are input into the Swim-Transformer backbone network for feature extraction.

[0014] Preferably, in step S2, the feature enhancement module includes three dilated convolution branches with different dilated rates and a global adaptive average pooling branch;

[0015] The three dilated convolution branches all use 3×3 convolution kernels, and the dilation rates are 1, 2, and 4 respectively; the global adaptive average pooling branch includes a layer of global adaptive average pooling, a layer of batch normalization, a ReLU activation function, and an upsampling operation. The outputs of the four branches are concatenated, integrated through 1×1 convolution, and finally output.

[0016] Preferably, in step S3, the spatiotemporal attention Transformer encoding layer introduces a spatiotemporal attention hybrid module to enhance the extraction of spatiotemporal information; the search frame feature sequence, the template frame feature sequence, and the dynamic template frame feature sequence are input into the spatiotemporal attention hybrid module after sinusoidal position encoding, and residual connection and layer normalization are performed; the output features are spliced ​​and input into multiple Transformer encoding layers to further optimize the features. The formula for the whole process is:

[0017] Z′=LN(Z+STAM(Z)) (1);

[0018] X′=LN(X+STAM(X)) (2);

[0019] Y′=LN(Y+STAM(Y)) (3);

[0020] U 1 =Concat(Z′,X′,Y′) (4);

[0021] U L+1 =LN(LN(U L +MSA(U L ))+FFN(LN(U L +MSA(U L )))) (5);

[0022] Among them, Z, X, and Y represent the search frame feature sequence, template frame feature sequence, and dynamic template frame feature sequence; Z′, X′, and Y′ represent the results of the search frame feature sequence, template frame feature sequence, and dynamic template frame feature sequence after being processed by the spatiotemporal attention hybrid module, residual connection, and layer normalization; U 1 Indicates the result after feature concatenation; U L+1 represents the output of the Lth Transformer encoding layer without embedding operation; STAM represents the spatiotemporal attention hybrid module operation; MSA represents the multi-head self-attention operation; LN represents layer normalization; FFN represents the feedforward neural network.

[0023] Preferably, the spatiotemporal hybrid attention module includes a temporal self-attention module and a spatial self-attention module; the input features are sent to these two modules at the same time to extract temporal and spatial information respectively; then, the features extracted by the two modules are added to obtain the final output result of the spatiotemporal hybrid attention module.

[0024] Preferably, in step S5, when the update cycle is reached, the detection result is evaluated according to the confidence of the classification, and it is determined whether to crop the tracking result to replace the initial template. The specific operation is as follows:

[0025] Extract the highest confidence from the classification results, and compare the highest confidence with a preset threshold;

[0026] When the highest confidence exceeds the threshold, the tracking result is cropped according to the template format and the dynamic template is updated;

[0027] When the highest confidence is lower than the threshold, no further action is taken and the next frame is processed directly.

[0028] The present invention also provides a single target tracking system, comprising:

[0029] The feature extraction module is used to use the weight-sharing Swim-Transformer as the backbone network for feature extraction to extract features from the template, dynamic template, and search area;

[0030] A feature enhancement module, used to enhance the features extracted by the feature extraction module;

[0031] A feature fusion module is used to input the enhanced search frame feature sequence, template frame feature sequence and dynamic template frame feature sequence into the spatiotemporal attention Transformer encoding layer and Transformer decoding layer for feature fusion;

[0032] The prediction head module is used to receive the fused features and input them into the classification branch and the regression branch. The two branches jointly obtain the tracking results.

[0033] The update decision module is used to check whether the preset update cycle has been reached, and evaluate the detection results according to the confidence of the classification to determine whether to crop the tracking results to replace the initial template.

[0034] The present invention also provides a computer device, comprising: a memory and a processor; the memory stores a computer program, and the processor implements the steps of the above-mentioned single target tracking method when executing the computer program.

[0035] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned single target tracking method are implemented.

[0036] Therefore, the present invention adopts the above-mentioned single target tracking method, system, electronic device and medium, and the technical effect is: the present invention is used to solve the problem that the Transformer single target tracker easily ignores spatiotemporal information and local information. The method as a whole includes three parts: a feature extraction network based on Swim-Transformer, a feature fusion network based on spatiotemporal attention Transformer encoder and Transformer decoder, and a prediction head. First, the search area, template and dynamic template are input into the Swim-Transformer backbone network for feature extraction, and then the extracted features are enhanced by the feature enhancement module to ensure that global and local information can be considered simultaneously when extracting features. The enhanced features are input into the feature fusion network based on spatiotemporal attention Transformer encoder and Transformer decoder. The designed feature fusion network can not only realize multimodal fusion, but also introduce the spatiotemporal attention hybrid module to enable the fusion network to effectively pay attention to the temporal and spatial information in the sequence to achieve the effect of improving tracking accuracy. The fused features are sent to the prediction head, which includes two branches: classification and regression. The two branches share the same input features and finally generate tracking results. The classification confidence is used to determine whether the search results need to be cropped and the initial template needs to be replaced. The dynamic template is introduced to realize the dynamic cycle of the tracking process to strengthen the introduction of temporal information. This method introduces spatiotemporal and local features based on the Transformer single target tracking method and has a good tracking effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A flow chart of a single target tracking method of the present invention;

[0038] Figure 2 This is the structure diagram of the spatiotemporal attention Transformer encoding layer;

[0039] Figure 3 It is the feature extraction module. DETAILED DESCRIPTION

[0040] The technical solution of the present invention is further described below through the accompanying drawings and embodiments.

[0041] Unless otherwise defined, technical or scientific terms used in the present invention shall have the common meanings understood by one having ordinary skills in the field to which the present invention belongs.

[0042] Embodiment 1

[0043] like Figure 1 FIG. 1 is a flow chart of a single target tracking method of the present invention, which specifically includes the following steps:

[0044] Step S1: Use the weight-sharing Swim-Transformer as the backbone network for feature extraction to extract features from the template, dynamic template, and search area. The specific operations are as follows:

[0045] When processing a video sequence, first check whether the current frame is the first frame. If it is not the first frame, determine whether the update cycle has arrived; if so, obtain the highest confidence based on the classification output result and determine whether it exceeds the set threshold. If it is higher than the threshold, crop the tracking result according to the template format, update the dynamic template, and input it into the Swim-Transformer backbone network to update the dynamic template features; if it is lower than the threshold, do not update the dynamic template and wait for the next frame to be processed. If it is the first frame, set it as the initial dynamic template; input the template, dynamic template and search area into the Swim-Transformer backbone network for feature extraction.

[0046] Swin-Transformer is used as the feature extraction backbone network, which is divided into three stages for target tracking. Stage 1 extracts the preliminary features of the input image and focuses on the target area. Stage 2 performs target feature modeling and context information fusion to enhance the relationship between the target and the background. Stage 3 uses the optimized features for accurate target positioning and tracking, generating the final target frame position and related tracking information.

[0047] Step S2: Enhance the extracted features through a feature enhancement module.

[0048] like Figure 3 As shown in the figure, the feature enhancement module includes three dilated convolution branches with different dilation rates and a global adaptive average pooling branch;

[0049] The three dilated convolution branches all use 3×3 convolution kernels, and the dilation rates are 1, 2, and 4 respectively; the global adaptive average pooling branch includes a layer of global adaptive average pooling, a layer of batch normalization, a ReLU activation function, and an upsampling operation. The outputs of the four branches are concatenated, integrated through 1×1 convolution, and finally output.

[0050] Step S3: input the enhanced search frame feature sequence, template frame feature sequence and dynamic template frame feature sequence into the spatiotemporal attention Transformer encoding layer and Transformer decoding layer for feature fusion.

[0051] like Figure 2As shown in the figure, the spatiotemporal attention Transformer encoding layer introduces a spatiotemporal attention hybrid module to enhance the extraction of spatiotemporal information; the search frame feature sequence, template frame feature sequence, and dynamic template frame feature sequence are input into the spatiotemporal attention hybrid module after sinusoidal position encoding, and residual connection and layer normalization are performed; the output features are concatenated and input into multiple Transformer encoding layers to further optimize the features. The formula for the whole process is:

[0052] Z′=LN(Z+STAM(Z)) (1);

[0053] X′=LN(X+STAM(X)) (2);

[0054] Y′=LN(Y+STAM(Y)) (3);

[0055] U 1 =Concat(Z′,X′,Y′) (4);

[0056] U L+1 =LN(LN(U L +MSA(U L ))+FFN(LN(U L +MSA(U L )))) (5);

[0057] Among them, Z, X, and Y represent the search frame feature sequence, template frame feature sequence, and dynamic template frame feature sequence; Z′, X′, and Y′ represent the results of the search frame feature sequence, template frame feature sequence, and dynamic template frame feature sequence after being processed by the spatiotemporal attention hybrid module, residual connection, and layer normalization; U 1 Indicates the result after feature concatenation; U L+1 represents the output of the Lth Transformer encoding layer without embedding operation; STAM represents the spatiotemporal attention hybrid module operation; MSA represents the multi-head self-attention operation; LN represents layer normalization; FFN represents the feedforward neural network.

[0058] The spatiotemporal hybrid attention module includes a temporal self-attention module and a spatial self-attention module; the input features are sent to these two modules at the same time to extract temporal and spatial information respectively; then, the features extracted by the two modules are added together to obtain the final output result of the spatiotemporal hybrid attention module.

[0059] Step S4: The fused features are simultaneously input into the classification branch and the regression branch, and the two branches jointly obtain the tracking result.

[0060] Step S5: Check whether the preset update cycle has been reached.

[0061] When the update cycle is reached, the highest confidence is extracted from the classification results, and the highest confidence is compared with the preset threshold;

[0062] When the highest confidence exceeds the threshold, the tracking result is cropped according to the template format and the dynamic template is updated;

[0063] When the highest confidence is lower than the threshold, no further action is taken and the next frame is processed directly.

[0064] When the update cycle is not reached, the dynamic template remains unchanged and the system continues to wait for the next frame of data to trigger subsequent updates.

[0065] Embodiment 2

[0066] A single target tracking system, characterized by comprising:

[0067] The feature extraction module is used to use the weight-sharing Swim-Transformer as the backbone network for feature extraction to extract features from the template, dynamic template, and search area;

[0068] A feature enhancement module, used to enhance the features extracted by the feature extraction module;

[0069] A feature fusion module is used to input the enhanced search frame feature sequence, template frame feature sequence and dynamic template frame feature sequence into the spatiotemporal attention Transformer encoding layer and Transformer decoding layer for feature fusion;

[0070] The prediction head module is used to receive the fused features and input them into the classification branch and the regression branch. The two branches jointly obtain the tracking results.

[0071] The update decision module is used to check whether the preset update cycle has been reached, and evaluate the detection results according to the confidence of the classification to determine whether to crop the tracking results to replace the initial template.

[0072] If the above functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program code.

[0073] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0074] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0075] It is worth noting that the contents not elaborated in detail in the present invention are all prior art and are well known to those skilled in the art.

[0076] Therefore, the present invention adopts the above-mentioned single target tracking method, system, electronic device and medium. First, a dynamic template update strategy is used to allow the tracker to dynamically update the template according to the movement and appearance changes of the target, thereby maintaining the accuracy and stability of tracking and improving the tracker's adaptability to the target. Second, a feature enhancement module is introduced to help capture the key features of the target and further optimize the performance of the tracker. Third, the spatiotemporal attention mechanism is introduced into the feature fusion network, so that it can effectively capture the dynamic changes of the target in time and space, help the tracker better understand the target's motion trajectory and appearance changes, and enhance the tracker's generalization ability in complex situations.

[0077] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A single target tracking method, characterized in that: The following steps are involved: Step S1, using the weight-sharing Swim-Transformer as the backbone network for feature extraction, and performing feature extraction on the template, dynamic template and search area; Step S2, enhancing the extracted features through a feature enhancement module; Step S3, inputting the enhanced search frame feature sequence, template frame feature sequence and dynamic template frame feature sequence into the spatiotemporal attention Transformer encoding layer and Transformer decoding layer for feature fusion; Step S4: The fused features are simultaneously input into the classification branch and the regression branch, and the two branches jointly obtain the tracking result; Step S5, check whether the preset update cycle is reached. When the update cycle is reached, evaluate the detection result according to the confidence of the classification, and determine whether to crop the tracking result to replace the initial template; when the update cycle is not reached, the dynamic template remains unchanged, and the system continues to wait for the next frame of data to trigger subsequent updates.

2. A single target tracking method according to claim 1, characterized in that: In step S1, Swim-Transformer is used as the backbone network for feature extraction to extract features from the template, dynamic template, and search area. The specific operations are as follows: When processing a video sequence, first check whether the current frame is the first frame. If the current frame is the first frame, set it as the initial dynamic template; input the template, dynamic template and search area into the Swim-Transformer backbone network for feature extraction.

3. A single target tracking method according to claim 2, characterized in that: In step S2, the feature enhancement module includes three dilated convolution branches with different dilation rates and a global adaptive average pooling branch; The three dilated convolution branches all use 3×3 convolution kernels, and the dilation rates are 1, 2, and 4 respectively; the global adaptive average pooling branch includes a layer of global adaptive average pooling, a layer of batch normalization, a ReLU activation function, and an upsampling operation. The outputs of the four branches are concatenated, integrated through 1×1 convolution, and finally output.

4. A single target tracking method according to claim 3, characterized in that: In step S3, the spatiotemporal attention Transformer encoding layer introduces a spatiotemporal attention hybrid module to enhance the extraction of spatiotemporal information; the search frame feature sequence, template frame feature sequence, and dynamic template frame feature sequence are input into the spatiotemporal attention hybrid module after sinusoidal position encoding, and residual connection and layer normalization are performed; the output features are concatenated and input into multiple Transformer encoding layers to further optimize the features. The formula for the whole process is: Z′=LN(Z+STAM(Z)) (1); X′=LN(X+STAM(X)) (2); Y′=LN(Y+STAM(Y)) (3); U 1 =Concat(Z′,X′,Y′) (4); IN L+1 =LN(LN(U L +MSA(U L ))+FFN(LN(U L +MSA(U L )))) (5); Among them, Z, X, and Y represent the search frame feature sequence, template frame feature sequence, and dynamic template frame feature sequence; Z′, X′, and Y′ represent the results of the search frame feature sequence, template frame feature sequence, and dynamic template frame feature sequence after being processed by the spatiotemporal attention hybrid module, residual connection, and layer normalization; U 1 Indicates the result after feature concatenation; U L+1 represents the output of the Lth Transformer encoding layer without embedding operation; STAM represents the spatiotemporal attention hybrid module operation; MSA represents the multi-head self-attention operation; LN represents layer normalization; FFN represents the feedforward neural network.

5. A single target tracking method according to claim 4, characterized in that: The spatiotemporal hybrid attention module includes a temporal self-attention module and a spatial self-attention module; The input features are sent to these two modules at the same time to extract temporal and spatial information respectively; then, the features extracted by the two modules are added together to obtain the final output result of the spatiotemporal hybrid attention module.

6. A single target tracking method according to claim 5, characterized in that: In step S5, when the update cycle is reached, the detection result is evaluated according to the confidence of the classification, and it is determined whether to crop the tracking result to replace the initial template. The specific operation is as follows: Extract the highest confidence from the classification results, and compare the highest confidence with a preset threshold; When the highest confidence exceeds the threshold, the tracking result is cropped according to the template format and the dynamic template is updated; When the highest confidence is lower than the threshold, no further action is taken and the next frame is processed directly.

7. A single target tracking system, characterized in that: include: The feature extraction module is used to use the weight-sharing Swim-Transformer as the backbone network for feature extraction to extract features from the template, dynamic template, and search area; A feature enhancement module, used to enhance the features extracted by the feature extraction module; A feature fusion module is used to input the enhanced search frame feature sequence, template frame feature sequence and dynamic template frame feature sequence into the spatiotemporal attention Transformer encoding layer and Transformer decoding layer for feature fusion; The prediction head module is used to receive the fused features and input them into the classification branch and the regression branch. The two branches jointly obtain the tracking results. The update decision module is used to check whether the preset update cycle has been reached, and evaluate the detection results according to the confidence of the classification to determine whether to crop the tracking results to replace the initial template.

8. A computer device comprising: Memory and processor; The memory stores a computer program, wherein the processor implements the steps of the single target tracking method according to any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the single target tracking method described in any one of claims 1 to 6 are implemented.

Images