An end-to-end single target tracking method based on multi-stage feature extraction
By using a multi-layer perceptron for shallow feature extraction in the first two stages of the single-target tracker and using a visual converter self-attention coding module for feature fusion in the third stage, the challenges in multi-scale information processing and real-time performance in the prior art are solved, and efficient and accurate target tracking is achieved.
Patent Information
- Application Number
- CN202311073647.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-08-24
- Publication Date
- 2025-04-22
- Estimated Expiration
- 2043-08-24
AI Technical Summary
Existing Transformer-based single-target trackers have challenges in handling multi-scale information and real-time, especially in resource consumption and computational complexity.
The end-to-end single-object tracking method of multi-stage feature extraction is adopted, and feature fusion and tracking paradigm learning is performed by using a multi-layer perceptron in the first two stages, and in the third stage, the cascading vision converter self-attention coding module is used for feature fusion and tracking paradigm learning.
This method effectively reduces the amount of computing and resource consumption, while improving the accuracy and real-time nature of target tracking, and better captures the correlation between the target and the search area.
Smart Images
Figure CN117115474B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of image processing, and in particular relates to an end-to-end single target tracking method based on multi-stage feature extraction. Background Art
[0002] Visual object tracking is one of the basic tasks in the field of computer vision. Its core goal is to estimate the spatial position of a specified target in each frame in a video sequence in real time and accurately draw the bounding box of the tracked target. In recent years, target tracking technology has made significant progress, but there are still many challenges in designing a simple and efficient end-to-end tracker. These challenges mainly come from the constraints of conditions in complex scenes such as scale changes, deformation, occlusion, etc. of the tracked target, as well as confusion and interference with similar targets. In practical applications, target trackers also need to maintain high accuracy and real-time performance. At the same time, target motion across time and space scales, as well as changes in target shape, need to be effectively modeled and processed by the tracker. These challenges greatly increase the difficulty of the tracking task.
[0003] Advanced transformer-based trackers usually use pre-trained models on general visual tasks to initialize network parameters. These visual tasks mainly include classification, detection, segmentation, etc. Using these backbone networks pre-trained on a large number of datasets to initialize model parameters can greatly reduce the cost of training trackers. Object tracking tasks are relatively specialized on these basic visual tasks, that is, downstream tasks. These transformer-based methods basically optimize and adjust the backbone network, which can be divided into two categories: (1) tracker design based on the general visual backbone network (ViT); (2) tracker design based on the improved visual backbone network, that is, the variant of ViT. At present, trackers based on these two methods have achieved good performance, but there are still some problems. First of all, the first method, since the original backbone network ViT does not have a multi-stage design, it directly divides the input image into 16×16 image blocks, which will lose some multi-scale information, which is very important for the tracker. The second method is designed based on the improved ViT. Some methods learn multi-scale information by introducing multiple stages, which alleviates the problems of the first method to a certain extent. However, in visual tasks, it is difficult to directly use the original Transformer to process the output image. This is mainly because the input image data size is generally large. If it is directly processed at the pixel level, the exponential complexity of the Transformer and the current hardware resources cannot support training. Due to the high redundancy of information in images, there is a method of dividing the image into blocks, which not only solves the problem of being unable to train, but also guarantees accuracy to a certain extent. Although this multi-stage processing method has achieved good results, it consumes a lot of resources and the real-time tracking is not very high. Summary of the invention
[0004] In view of the shortcomings of the prior art, the present invention proposes an end-to-end single target tracking method based on multi-stage feature extraction, the method comprising: obtaining a video to be identified and preprocessing it, obtaining the preprocessed image and inputting it into a trained end-to-end single target tracking model to obtain a target tracking result;
[0005] The training process of the end-to-end single target tracking model includes:
[0006] S1: Obtain the video to be identified and preprocess it to obtain a template frame and a search frame;
[0007] S2: Use a multi-layer perceptron to process the template frame and the search frame respectively to obtain primary template frame features and primary search frame features;
[0008] S3: Splicing the primary template frame features and the primary search frame features to obtain splicing features;
[0009] S4: Input the concatenated features into multiple cascaded visual transformer self-attention encoding modules for processing to obtain mixed features;
[0010] S5: Split the mixed features to obtain the search feature vector;
[0011] S6: Use the autoregressive prediction head to process the search feature vector to obtain the target tracking frame, i.e., the target tracking result;
[0012] S7: Calculate the total model loss and adjust the model parameters according to the total model loss to obtain a trained end-to-end single target tracking model.
[0013] Preferably, the process of preprocessing the video to be identified includes: cropping the video frame to be identified according to the center position of the real target frame, the cropping size is 4 times the real target frame, the cropped image is used as the search frame, and the image cropped from the first frame of the video is used as the template frame.
[0014] Preferably, the process of using a multi-layer perceptron to process the template frame and the search frame respectively includes: performing linear projection block processing on the template frame and the search frame to obtain flattened template block vectors and search block vectors; inputting the flattened template block vectors and search block vectors into the multi-layer perceptron for feature encoding to obtain primary template frame features and primary search frame features.
[0015] Furthermore, the process of performing linear projection block processing on the template frame and the search frame includes: adjusting the size of the template frame to 224×224×3, and adjusting the size of the search frame to 224×224×3; inputting the resized template frame and search frame into the convolution layer for processing, respectively, to obtain overlapping template block vectors and search block vectors; flattening the overlapping template block vectors and search block vectors to obtain flattened template block vectors and search block vectors.
[0016] Furthermore, the convolution kernel size of the convolution layer is 4 and the step size is 4.
[0017] Preferably, the processing process of the visual converter self-attention encoding module on data includes:
[0018] Perform linear projection on the input features to obtain the Q, K, and V vectors of the features, and perform the self-attention operation of the visual converter to obtain the attention feature vector;
[0019] Input the attention feature vector into the multi-layer perceptron for linear projection mapping to obtain linear features;
[0020] Add the linear features and input features to get the output features of the module.
[0021] Preferably, the total loss of the model is:
[0022]
[0023] Among them, Loss represents the total loss of the model, L represents the length of the target sequence, Q(·) represents the softmax operation, s represents the search frame, and t represents the template frame. represents the predicted target sequence, j represents the marker position, Represents the j-th position of the predicted target sequence.
[0024] The beneficial effects of the present invention are as follows: the present invention repeats the visual converter attention module multiple times to obtain a feature extractor based on the module, and deepens the network depth through multiple such serial modules. Thanks to the characteristics of self-attention itself, while performing feature extraction of the target template and the search area, the module can also realize feature fusion between the template feature vector and the search feature vector by setting position encoding. In addition, the present invention also uses a multi-stage structural design. Specifically, in the first two stages, a simple multi-layer perceptron network is used to perform shallow feature extraction on the input, and in the third stage, as the core of the algorithm, a self-attention encoding module based on the visual converter is used to learn the tracking paradigm. Such a design not only integrates feature information of different scales, but also reduces the amount of calculation, and at the same time learns rich feature representations, making full use of the feature modeling capability of the Transformer, and enhancing the expressiveness and robustness of the model. The present invention can better capture the correlation between the target and the search area, and improve the accuracy and real-time performance of the target tracking results. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] Figure 1 It is a flow chart of the end-to-end single target tracking method based on multi-stage feature extraction in the present invention;
[0026] Figure 2 Schematic diagram of the autoregressive position prediction process in the present invention. DETAILED DESCRIPTION
[0027] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0028] The present invention proposes an end-to-end single target tracking method based on multi-stage feature extraction. Figure 1As shown, the method includes the following contents: obtaining a video to be identified and preprocessing it, inputting the preprocessed image into a trained end-to-end single target tracking model, and obtaining a target tracking result; the end-to-end single target tracking model (HiViTrack) includes a multi-stage tracking paradigm learning module and an autoregressive prediction head, and the multi-stage tracking paradigm learning module is composed of a multi-layer perceptron in the first two stages and a cascaded visual converter self-attention encoding module in the third stage.
[0029] The current mainstream single target tracker is mainly composed of three main modules: (1) Backbone network, which is used to extract the common features of the target to be tracked and the search area; (2) Feature fusion module, which fuses the information of the tracked target and the feature information of the search area to achieve subsequent target positioning and perception; (3) Target bounding box accurate prediction module, which is used to generate the final tracking result, that is, the bounding box of the target. Among these three main modules, the core is the feature fusion module of the second part, which mainly integrates the feature information of the target to be tracked into the features of the search area, realizes the information communication between the template and the search area, guides the subsequent tracking process, and obtains the corresponding tracking box according to the specific tracking target.
[0030] Different from the existing methods, the core idea of the present invention is to ensure that the multi-stage end-to-end structure is used to learn multi-scale features, while using auxiliary multi-layer perceptrons in the first two stages to reduce computational overhead. The first two stages of the existing trackers based on multi-stage visual converters almost all use specially processed converters to solve the problem of high computational load. Although these methods have achieved good results, they have greatly increased computational complexity and resource overhead. The present invention believes that the first two stages learn shallow feature information, and the image is highly redundant, so there is no need to use algorithms like converters that consume a lot of resources to implement it. Based on this background, the present invention implements the first two stages with a multi-layer perceptron. While learning multi-scale information, it also completes the encoding of shallow features, achieving the same function as the converter, but greatly reducing resource overhead. The present invention focuses on the feature extraction and feature fusion based on the self-attention encoding module of the visual converter in the third stage, which well achieves a balance between high real-time and high precision.
[0031] The training process of the end-to-end single target tracking model includes:
[0032] S1: Obtain the video to be identified and preprocess it to obtain a template frame and a search frame.
[0033] The process of preprocessing the video to be identified includes: cropping the video frame to be identified according to the center position of the real target frame, the cropping size is 4 times the real target frame, using the cropped image as the search frame, and using the image cropped from the first frame of the video as the template frame. The label of the video to be identified for training is the center point coordinates (x, y) and position offset (w, h) obtained after the original annotation information of the real target frame is converted.
[0034] S2: Use a multi-layer perceptron to process the template frame and the search frame respectively to obtain primary template frame features and primary search frame features.
[0035] The process of using a multi-layer perceptron to process the template frame and the search frame respectively includes:
[0036] The template frame and the search frame are subjected to linear projection block processing to obtain flattened template block vectors and search block vectors; specifically: the process of linear projection block processing of the template frame and the search frame includes: adjusting the size of the template frame to 224×224×3, and adjusting the size of the search frame to 224×224×3; inputting the resized template frame and the search frame into the convolution layer for processing respectively to obtain overlapping template block vectors and search block vectors; flattening the overlapping template block vectors and search block vectors to obtain flattened template block vectors and search block vectors. Preferably, the convolution kernel size of the convolution layer is 4, and the step size is 4.
[0037] The flattened template block vector and the search block vector are input into two consecutive feature extractors, namely, multi-layer perceptrons, for feature encoding to obtain primary template frame features and primary search frame features.
[0038] S3: Concatenate the primary template frame features and the primary search frame features to obtain concatenated features.
[0039] The primary search frame features are concatenated with the primary template frame features to obtain concatenated features.
[0040] S4: The concatenated features are input into multiple cascaded visual transformer self-attention encoding modules for processing to obtain mixed features.
[0041] The output of the previous visual converter self-attention encoding module is the input of the next visual converter self-attention encoding module, and the final output is the output of the last visual converter self-attention encoding module; the processing process of a single visual converter self-attention encoding module on input data includes:
[0042] Perform linear projection on the input features to obtain the Q, K, and V vectors of the features, and perform the self-attention operation of the visual converter to obtain the attention feature vector;
[0043] Input the attention feature vector into the multi-layer perceptron for linear projection mapping to obtain linear features;
[0044] Add the linear features and input features to get the output features of the module.
[0045] The following will analyze the internal reasons why the proposed method can achieve both feature extraction and feature fusion from the perspective of self-attention mechanism. The output of the self-attention operation A can be written as:
[0046]
[0047] Where Q, K and V are query, key and value matrices respectively; z and x represent the template frame vector and search frame vector respectively. The calculation of attention weight can be further expanded as:
[0048]
[0049] Where W zx Represents the similarity measure between the template and the search area. Similarly, W zz , W xz and W xx Denotes the corresponding similarity measure, then the output A of the self-attention operation can be further recorded as:
[0050] A=[W zz V z +W zx V x ; W xz V z +W xx V x ]
[0051] Among them, W zx V x , W xz V z is responsible for aggregating the similarities of different image parts (feature fusion), while
[0052] W xx V x ,W zz V z Aggregate intra-image features based on the similarity of different image parts (feature extraction). Therefore, feature extraction and relationship modeling can be performed simultaneously through the self-attention mechanism.
[0053] The concatenated vector is input into multiple cascaded self-attention encoding modules of the above-mentioned visual converter, and the above-mentioned operations are performed multiple times to obtain a feature vector fused by deep coding, i.e., a hybrid feature; it fuses the template frame information (tracking target information) and the search frame information, and learns the tracking paradigm.
[0054] S5: Split the mixed features to obtain the search feature vector.
[0055] like Figure 2 As shown, the mixed features are split according to the flattened feature sizes corresponding to the template frame and the search frame to obtain the split template feature vector and the search feature vector, and the search feature vector is taken out and used as the input of the autoregressive prediction head.
[0056] S6: Use the autoregressive prediction head to process the search feature vector to obtain the target tracking box, i.e., the target tracking result.
[0057] The autoregressive prediction head receives the search feature vector output by S5 as input, and directly predicts the coordinate position information of the target; specifically, the autoregressive prediction head is implemented based on the transformer decoder, and directly obtains the target tracking box, that is, the center point coordinates of the target position box and the position offset on the two coordinate axes.
[0058] S7: Calculate the total model loss and adjust the model parameters according to the total model loss to obtain a trained end-to-end single target tracking model.
[0059] The total loss of the model is:
[0060]
[0061] Among them, Loss represents the total loss of the model, L represents the length of the target sequence, Q(·) represents the softmax operation, s represents the search frame, and t represents the template frame. represents the predicted target sequence, j represents the marker position, represents the jth position of the predicted target sequence, Indicates the value used to predict the current tag. The preceding subsequence of .
[0062] The input sequence is the target sequence with a position offset that omits the start marker BOS and the end marker EOS. This offset combined with causal masking ensures the autoregressive property of the sequence model. The target sequence can be regarded as a description of the object bounding box, and the training goal is to teach the model to read the description words based on the previous words.
[0063] During inference, the encoder perceives the search region in the template image and the subsequent video frame. The initial input to the autoregressive prediction head is the start token BOS, which tells the model to start generating. The model then reads the target sequence [x, y, w, h, EOS] token by token. For each token, the model samples from the vocabulary V according to the maximum likelihood, i.e. in It is a word in V.
[0064] Through the above loss function design, there is no need to design different loss functions for target box position regression and classification during training. This unified training structure allows the model to focus more on the target as a whole, rather than biased towards category loss or regression loss. Compared with the previous method of setting different penalties (regularization) for losses based on experience, the loss function used in this method is very helpful for model optimization and can obtain better results.
[0065] In summary, the present invention mainly improves the tracker based on multi-stage visual converter, by using a simple multi-layer perceptron to extract shallow features in the first two stages, and then using the improved visual converter to learn the tracking paradigm and target features in the third stage. The previous two-stage method uses a convolutional neural network backbone network to extract features from the template and the search area respectively in the first stage, and then uses a converter with manually initialized parameters to perform feature fusion and learn the tracking paradigm in the second stage. Different from it, the method proposed in the present invention is an end-to-end solution. Since the first two stages of the tracker based on the multi-stage converter are only designed to assist the third stage, there is no need to use the converter to model the feature information of these two stages. The multi-layer perceptron can effectively extract shallow features, and the multi-stage design can also well guarantee multi-scale learning. Therefore, this method takes the third stage as the core, makes full use of the feature modeling ability of the converter, and realizes feature fusion while extracting features, that is, learning the tracking paradigm. This method greatly reduces the amount of calculation and can ensure that the tracker has good performance.
[0066] The above embodiments further illustrate the purpose, technical solutions and advantages of the present invention in detail. It should be understood that the above embodiments are only preferred implementation modes of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made to the present invention within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. An end-to-end single target tracking method based on multi-stage feature extraction, characterized in that: include: Obtain the video to be identified and preprocess it, and input the preprocessed image into the trained end-to-end single target tracking model to obtain the target tracking result; The training process of the end-to-end single target tracking model includes: S1: Obtain the video to be identified and preprocess it to obtain a template frame and a search frame; S2: using a multi-layer perceptron to process the template frame and the search frame respectively, to obtain primary template frame features and primary search frame features; the process of using the multi-layer perceptron to process the template frame and the search frame respectively includes: performing linear projection block processing on the template frame and the search frame to obtain a flattened template block vector and a search block vector; inputting the flattened template block vector and the search block vector into two consecutive feature extractors, i.e., the multi-layer perceptron, to perform feature encoding, to obtain primary template frame features and primary search frame features; S3: Splicing the primary template frame features and the primary search frame features to obtain splicing features; S4: Input the concatenated features into multiple cascaded visual transformer self-attention encoding modules for processing to obtain mixed features; the visual transformer self-attention encoding module processes the data including: Perform linear projection on the input features to obtain the Q, K, and V vectors of the features, and perform the self-attention operation of the visual converter to obtain the attention feature vector; Input the attention feature vector into the multi-layer perceptron for linear projection mapping to obtain linear features; Add the linear features and input features to get the output features of the module; S5: Split the mixed features to obtain the search feature vector; S6: Use the autoregressive prediction head to process the search feature vector to obtain the target tracking frame, i.e., the target tracking result; S7: Calculate the total model loss and adjust the model parameters according to the total model loss to obtain a trained end-to-end single target tracking model.
2. The end-to-end single target tracking method based on multi-stage feature extraction according to claim 1 is characterized in that: The process of preprocessing the video to be identified includes: cropping the video frame to be identified according to the center position of the real target frame, the cropping size is 4 times the real target frame, the cropped image is used as the search frame, and the image cropped from the first frame of the video is used as the template frame.
3. The end-to-end single target tracking method based on multi-stage feature extraction according to claim 1 is characterized in that: The process of performing linear projection block processing on the template frame and the search frame includes: adjusting the size of the template frame to 224×224×3, and adjusting the size of the search frame to 224×224×3; inputting the resized template frame and search frame into the convolution layer for processing respectively to obtain overlapping template block vectors and search block vectors; flattening the overlapping template block vectors and search block vectors to obtain flattened template block vectors and search block vectors.
4. The end-to-end single target tracking method based on multi-stage feature extraction according to claim 3 is characterized in that: The convolution kernel size of the convolution layer is 4 and the stride is 4.
5. The end-to-end single target tracking method based on multi-stage feature extraction according to claim 1, characterized in that: The total loss of the model is: Among them, Loss represents the total loss of the model, L represents the length of the target sequence, Q(·) represents the softmax operation, s represents the search frame, and t represents the template frame. represents the predicted target sequence, j represents the marker position, Represents the j-th position of the predicted target sequence.