A natural language object tracking method based on a Transformer architecture

Abstract

The application discloses a natural language target tracking method based on a Transformer architecture, and the method comprises the following steps: step S1, data loading and processing; step S2, using an encoder and a decoder of a Transformer structure to extract the features of the input data in step S1; step S3, fusing the features of different modes; step S4, model training and reasoning to obtain a prediction result. The application proposes an end-to-end unified network based on Transformers, which uses two pairs of encoder and decoder structures to cooperatively learn reference and tracking. It can not only independently perform visual reference or tracking, but also perform language description-based tracking by fusing low-level and high-level features, so that higher robustness and higher precision are obtained.

Description

Technical Field

[0001] This invention relates to a target tracking method, and more particularly to a natural language target tracking method based on the Transformer architecture. Background Technology

[0002] The goal of natural language-based target tracking methods is to continuously locate targets described in natural language within video sequences. Unlike traditional target tracking tasks that select the target in the first frame of the video as a tracking template, language-based target tracking makes human-computer interaction more convenient and is more suitable for real-world applications. Previous image-template-based target tracking methods often suffer from decreased tracking accuracy due to environmental changes such as motion, lighting, and occlusion.

[0003] Language-based target tracking methods consist of two important subtasks: localization and tracking. Previous methods treated them as two independent subtasks, using existing localization and tracking branches to obtain results for subsequent tasks. For example, models like GTI and TNL2K utilize the existing SiamRPN++ target tracking model combined with language information to achieve language-based target tracking without significant model adjustments. In terms of model structure, the heterogeneous structure and staged training of previous methods allowed the localization and tracking branches to be optimized independently, without a collaborative learning process to enable the two modules to cooperate. Summary of the Invention

[0004] To address the shortcomings of the aforementioned technologies, this invention provides a natural language target tracking method based on the Transformer architecture. Specifically, it proposes an end-to-end Transformer dual-branch network that integrates the visual and linguistic features of the object entity and can perform target tracking given a target image template, a linguistic description, or both.

[0005] To solve the above technical problems, the technical solution adopted by this invention is: a natural language target tracking method based on the Transformer architecture, which includes the following steps:

[0006] Step S1: Load and process data;

[0007] Step S2: Use an encoder and decoder with a Transformer structure to extract features from the input data in step S1;

[0008] Step S3: Perform feature fusion of different modalities;

[0009] Step S4: Perform model training and inference to obtain prediction results.

[0010] Preferably, the specific steps of step S1 are as follows: input the target image template Z, the searched image sequence X, and the language description features Q; use the ResNet-50 network to extract the visual features of the input target image template Z and the searched image sequence X to obtain Z' and X'; and use the Roberta network to extract the language features of the language description Q to obtain Q'.

[0011] Preferably, in step S2, two pairs of encoder and decoder structures with a Transformer architecture are used to collaboratively learn the reference and tracking. The reference module and the tracking module require different combinations of visual search template feature map Z' from the visual backbone, visual search region feature map X', and language features Q' from the language module.

[0012] Preferably, step S2 specifically includes the following steps:

[0013] Step S21: X′∈R H×W×C′ , Q′∈R L×C′ The number of channels is reduced to C;

[0014] in, The visual search template feature map Z′ represents the region with dimensions H in its domain. z W z In a space of dimension C′, R H×W×C′ This represents the visual search region feature map X′ within a space with dimensions H and W, and dimension C′. L×C′ This indicates that the linguistic feature Q' lies within a space of length L and dimension C′ in the domain of linguistic features;

[0015] Step S22: Flatten and concatenate feature maps Z and X along the spatial dimensions to generate a sequence of length HW+H. z W z The tracking features of dimension C are used as input to the image template Transformer encoder;

[0016] Step S23: Concatenate the flattened feature map X′ and the language feature Q′ to form a referential feature with a sequence length of HW+L, which is used as the input of the language Transformer encoder;

[0017] Step S24: The Transformer structure is used as the encoder structure for the target image template branch and the language branch. It consists of N encoder layers, each of which consists of a multi-head self-attention module with a feedforward network.

[0018] Step S25: The decoder consists of M decoder layers, each of which consists of a self-attention network, a codec attention network, and a feedforward network.

[0019] Preferably, in step S24, a three-layer perceptron with a ReLU activation function is used to predict the normalized center coordinates, height, and width of the bounding box of the target object.

[0020] Preferably, step S3 involves feature fusion of visual and linguistic data features, which specifically includes the following steps:

[0021] Step S31: Fuse the search image features extracted by the template encoder with the template features, denoted as feature M. Z The fusion of search image features and language features is denoted as feature M. Q The first step in feature fusion is to combine feature M Z With feature M Q The initial fusion feature M is obtained by adding the features together. F :

[0022] M F =M Q +M Z Formula 1

[0023] Step S32: Use a 1*1 convolution to transform M F The number of channels is reduced to 1, and the normalized attention map F is obtained using the Softmax function. z Attention map F q It is to use feature 1-M F The formula obtained by performing the same operation is as follows:

[0024] F z =Softtmax(Conv(M F Formula 2

[0025] F q =Softmax(1-Conv(M) F Formula 3

[0026] Step S33: Using F q and F z Multimodal global context information is extracted using matrix multiplication, as shown in the following formulas:

[0027]

[0028]

[0029] F′ q and F′ z All are composed of feature map F q and Fz From global features M Q M Z Global multimodal feature information extracted from it;

[0030] Step S34: Set F′ q and F′ z Added to its original corresponding decoder output feature D q D z In this process, multimodal characteristics are enhanced and concatenated as input D to the fusion prediction head. F The formula is as follows;

[0031] D F =cat(F′) q +D q F′ z +D z Formula 6.

[0032] Preferably, step S4 specifically includes the following steps:

[0033] Step S41: Randomly select an image from the video as the target frame, find the previous frame of the target frame, and then crop and adjust the size of the two frames respectively to obtain the target image template Z and its corresponding searched image sequence X. Construct a training tuple (Z, Q, X) for model training.

[0034] Step S42: Perform inference: For each input image, the network takes the search region centered on the prediction or standard bounding box of the previous frame as input and outputs the prediction result corresponding to each branch, and obtains the fused output as the final result.

[0035] Preferably, in step S41, the three losses for tracking, referencing branch, and hybrid branch are calculated respectively, as shown in the following formula:

[0036]

[0037]

[0038]

[0039] The total loss L is as follows:

[0040] L=λ t L t +λ f L f +λ g L g Formula 10

[0041] Among them, L iouThis indicates the calculation of the intersection-union ratio loss, L1 indicates the calculation of the L1 normal form loss, b′ represents the true bounding box, b t b g b f These represent the prediction boxes for tracking, fusion, and referring to branches, respectively, and λ iou , λ1, λ t , λ f , λ g It is a hyperparameter for adjusting the loss, and it makes λ of the three branches... iou and Maintain consistency.

[0042] Preferably, in step S42, the first frame is fixed as the template image in subsequent video frames, and the target image template Z and the language description feature Q are fixed in all frames.

[0043] Compared with the prior art, the present invention has the following advantages:

[0044] 1. This invention proposes an end-to-end unified network based on Transformers, which employs two pairs of encoders and decoders to collaboratively learn visual references and tracking. It can not only perform visual references or tracking independently, but also perform language-based tracking by fusing low-level and high-level features, achieving higher robustness and accuracy.

[0045] 2. This invention proposes a novel integration module for fusing two branches using confidence scores. The confidence heads are trained independently, and the scores reflect the performance of each branch.

[0046] 3. This invention validates the performance of the proposed method on three popular datasets through extensive experiments: the LaSOT dataset, the language-annotated OTB100 dataset, and the TNL2K dataset. Compared with popular object tracking methods, it demonstrates significantly improved results and reaches the forefront of object tracking tasks.

[0047] 4. This invention improves upon the shortcomings of tracking solely based on the initial frame target template by introducing linguistic information, thus effectively complementing traditional target tracking methods. Language-based target tracking methods have wide applications in tasks such as video question answering and video text retrieval. Attached Figure Description

[0048] Figure 1 This is a schematic diagram of the overall framework of the model of this invention.

[0049] Figure 2 This is a schematic diagram illustrating the fusion of image templates and natural language features.

[0050] Figure 3This is a comparison chart of the tracking visualization results of TransNLT and SNLT. Detailed Implementation

[0051] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0052] This invention proposes a natural language target tracking model, TransNLT, based on the Transformer architecture. It utilizes a single-stage end-to-end model to unify cross-modal target detection and tracking tasks. The model consists of two independent referencing and tracking branches based on Transformer, which are connected and combined through a feature fusion module. The input of the model includes three parts: the sequence of searched images, the target image template, and the language description. However, the target image template and the language description do not necessarily need to exist at the same time. Either input can output a prediction result, but inputting both at the same time will bring more accurate results.

[0053] The overall architecture of the TransNLT model is as follows: Figure 1 As shown, given a target image template Z, a language description Q, and a sequence of images to be searched X, the model predicts the bounding box of the target object in the sequence of images to be searched based on the target image template and the language description. The model can extract image and causal features using any convolutional neural network (CNN) and language model. This invention uses the traditional ResNet-50 as the visual backbone network and Roberta as the language model. The three vectors, target image template, sequence of images to be searched, and language description, are represented as Z', X', and Q', respectively.

[0054] After feature extraction, visual and textual features are processed in two separate branches. The target image template Z and the searched image sequence X form target tracking features, while the language description Q and the searched image sequence X form target detection features. Inspired by the Transformer structure used in computer vision, this invention uses a similar Transformer encoder and decoder structure to process the above features. The two branches have identical encoder structures and almost identical decoders. Unlike the target tracking branch, after the pooling operation in the referential branch, the text sentence features are added to the object query vector introduced in DETR. Although structurally similar, the parameters of the two independent branches are not shared. Furthermore, the two branches can predict results independently, providing strong flexibility for the practical application of this algorithm model. They can each use the tracking features or referential features output by the decoder, along with a simple prediction head, to output the final result. The fusion module can fully utilize the advantages of the two features to complement each other, producing more accurate results. In the experiment, the fusion result of the two features was used as the final prediction result, while the prediction outputs of the two independent branches were only used for training.

[0055] Referencing branches and tracking branches:

[0056] like Figure 2 As shown, the tracking branch and the referencing branch each require different feature maps. For a single independent branch, the components include input preprocessing, an encoder, a decoder, and a prediction head. Specifically, after the target image template feature X′, the language description feature Q′, and the searched image sequence feature X′ are reduced to d dimensions, Z′ and X′ are flattened and concatenated with spatial coordinate information to generate the input features for the tracking branch, which serve as the input to the Transformer encoder. Similarly, the flattened Q′ and X′ are concatenated and used as the input features for the referencing branch. The referencing module and the tracking module require different combinations of visual search template feature map Z′ from the visual backbone, visual search region feature map X′, and language features Q′ from the language module. For a single branch module, the complete components include input preprocessing, an encoder, a decoder, and a prediction head.

[0057] First, let Z X′∈R H×W×C′ , Q′∈R L×C′ The number of channels is reduced to C; feature maps Z′ and X′ are flattened along the spatial dimension and concatenated to produce a sequence of length HW+H. z W z The tracking feature with dimension C is used as the input to the image template Transformer encoder. At the same time, the flattened feature map X′ and the language feature Q′ are concatenated to form a referential feature with a sequence length of HW+L, which is used as the input to the language Transformer encoder. The encoders for tracking and referential branches adopt the same structure as the original Transformer, consisting of N encoder layers, each consisting of a multi-head self-attention module and a feedforward network module.

[0058] The decoder structure of this model is similar to that of DETR. The tracking and referencing branch decoders take the object query vector and the enhanced feature sequence output by the encoder as inputs, respectively. Since the prediction result is a single bounding box, the number of object query vectors is set to 1, and the Hungarian Algorithm in DETR is removed. The decoder consists of M decoding layers, each of which consists of a self-attention module, an encoder attention module, and a feedforward network module.

[0059] Both the referencing and tracking branches use object query vectors to locate the target in global features. It's worth noting that the referencing branch also uses pooled linguistic features to enhance the object query vectors for more accurate predictions. The decoder output is used to predict the coordinates of the bounding box, employing a three-layer perceptron based on ReLU activation to predict normalized center point coordinates, as well as the height and width of the target object's bounding box.

[0060] Fusion Module:

[0061] To achieve better prediction performance by fusing features from the referencing and tracking branches across modalities, this invention proposes a fusion process based on cross-modal attention on the output features of the two branches. First, the outputs of the two branch encoders are split into M... Z and M Q Then, the two are combined and added together to obtain the initial fusion feature M. F .

[0062] M F =M Q +M Z Formula 1

[0063] The obtained M F For the sake of convenience in explaining the principle, the transformation and transpose of the matrix are omitted here, as ∈R is used as a preliminary fusion feature.

[0064] M is convolved using a 1x1 convolution (represented by Conv). F The number of channels is reduced to 1, and then the attention feature map is normalized using the Softmax function to obtain F. z Similarly, another branch yields F. q .

[0065] F z =Softmax(Conv(M) F Formula 2

[0066] F q =Softmax(1-Conv(M) F Formula 3

[0067] Then, these two feature maps are multiplied with the outputs of the two branch encoders to obtain global multimodal feature information.

[0068]

[0069]

[0070] Finally, this multimodal global information is merged and concatenated with the original decoder input features (represented by cat) to enhance the multimodal fusion features, which are then used as the input D of the final prediction head. F .

[0071] D F =cat(F′) q +D q F′ z +D z Formula Six

[0072] Training and reasoning:

[0073] This section introduces the training data and the settings for training and inference, and describes the relevant loss functions. During training, an image is randomly selected from the video sequence as the target frame, and its preceding frame is found. These two frames are cropped and resized to serve as the target image template Z and its corresponding search image X, respectively. Cropping and resizing are based on labeled data, and the search image region X is slightly larger than the target image template region Z. The final training inputs are Z, Q, and X. Training is end-to-end, consistent with DETR's use of a combination of L1 and GIoU losses. The difference lies in calculating the total loss across the tracking, referencing, and fusion branches. Therefore, this method ensures that the algorithm can output predictive results under different input conditions. Specifically, when the prior linguistic description is missing, the tracking branch is used for prediction; when the target template is lacking, the referencing branch is used; and when both prior conditions are met, the fusion branch is used. The total loss L is defined as follows:

[0074]

[0075]

[0076]

[0077] L=λ t L t +λ f L f +λ g L g Formula 10

[0078] Among them, L iou This indicates the calculation of the intersection-union ratio loss, L1 indicates the calculation of the L1 normal form loss, b′ represents the true bounding box, b t b g b f These represent the prediction boxes for tracking, fusion, and referring to branches, respectively, and λ iou , λ1, λ t , λ f , λ g It is a hyperparameter for adjusting the loss, and it makes λ of the three branches... iou and Maintain consistency.

[0079] In the inference phase, as is customary, the first frame's bounding box is used as the target image template Z, and the linguistic description Q is also given in advance. For each frame of the searched image, the model locates the searched region based on the prediction result or the bounding box of the previous frame, and then feeds this region into the model along with Z and Q. Each branch can output a prediction result, and here the output of the fusion branch is used as the final result.

[0080] This experiment was conducted on the OTB100, LaSOT, and TNL2K datasets, and the results are shown in Table 1.

[0081] Table 1 Comparison of Experimental Results of Leading Target Tracking Methods

[0082]

[0083]

[0084] As shown in Table 1, in the LaSOT dataset, the tracking success rate and accuracy obtained by the tracking experiment initialized with language description reached 0.51 and 0.51 respectively, which are slightly higher than the 0.51 and 0.49 of the TNL2K method. The tracking success rate and accuracy obtained by the tracking experiment initialized with language description and bounding box reached 0.60 and 0.63 respectively, which are 0.09 and 0.12 higher than the best results of the same task.

[0085] In the OTB100 dataset, the tracking success rate and accuracy obtained by the tracking experiment initialized with language description reached 0.54 and 0.70 respectively, which are close to the existing highest level of 0.54 and 0.78. The tracking success rate and accuracy obtained by the tracking experiment initialized with language description and bounding box reached 0.67 and 0.87 respectively, which are close to the existing highest level of 0.68 and 0.88.

[0086] In the TNL2K dataset, the tracking success rate, standard accuracy, and precision achieved using language description for initialization were 0.51, 0.68, and 0.49, respectively, representing significant improvements over the original TNL2K method, with increases of 0.4, 0.57, and 0.43. The tracking success rate, standard accuracy, and precision achieved using language description and bounding boxes for initialization were 0.57, 0.75, and 0.57, respectively, representing improvements of 0.15, 0.25, and 0.15 over the current highest level. Based on these experimental results, this invention effectively achieves target tracking using language and target image templates, reaching a state-of-the-art level.

[0087] Tracking visualization results such as Figure 3As shown, this invention is compared with the SNLT (Symphony Tracking Network based on Natural Language) method in a tracking visualization. The figure shows the tracking of "an athlete doing pole vaulting," "a green circuit board," and "a girl wearing a yellow shirt and purple pants." The ground truth bounding box, TransNLT bounding box, and SNLT bounding box represent the correct bounding box, the TransNLT tracking bounding box, and the SNLT tracking bounding box, respectively. As shown, TransNLT can provide a more precise tracking bounding box with a smaller range. Furthermore, due to the initialization of language description and the fusion of visual language features, TransNLT can correct accumulated tracking errors, achieving the goal of continuously tracking the correct target. It can also better handle tracking difficulties such as occlusion and camera changes.

[0088] This invention utilizes the isomorphic structure of the referential and tracking branches for collaborative training and designs an efficient fusion module to integrate the feature representations of the two branches, achieving better tracking results through the comprehensive interaction between referential and tracking information. Since the Transformer architecture exhibits a powerful ability to model long-range dependencies, which is crucial for many vision and language tasks, this invention leverages the Transformer as the underlying structure for the referential and tracking branches. This invention designs a pair of encoders and decoders to process template information and search boxes, while using a similar Transformer encoder-decoder structure (DETR) to process language information and search boxes. Furthermore, this invention proposes a fusion module to fuse the features of the two branches based on confidence scores, and for the first time introduces the Transformer architecture into a natural language-based object tracking task.

[0089] The above embodiments are not intended to limit the present invention, and the present invention is not limited to the examples given above. Any changes, modifications, additions or substitutions made by those skilled in the art within the scope of the technical solution of the present invention are also within the protection scope of the present invention.

Claims

1. A natural language object tracking method based on a Transformer architecture, characterized in that: The method comprises the following steps: Step S1: data loading and processing, the specific steps are: inputting a target image template Z, a searched picture sequence X and a language description feature Q, using a Resnet-50 network to extract visual features of the input target image template Z and the searched picture sequence X to obtain Z', X', using a Roberta network to extract language features of the language description Q to obtain Q'; Step S2: using an encoder and a decoder of a Transformer structure to extract features of the input data in step S1; that is, using two pairs of encoder and decoder structures of the Transformer structure to cooperatively learn reference and tracking, wherein the reference module and the tracking module need different combinations of the visual search template feature map Z' and the visual search region feature map X' from the visual backbone and the language feature Q' from the language module; The tracking branch and the referring branch need different feature maps, and for a single independent branch, respectively including input preprocessing, an encoder, a decoder and a prediction head; specifically, the target image template feature , the language description feature Q', and the searched picture sequence feature After the number of channels is reduced to d dimensions, and The flattened operation is performed, and after being spliced with spatial coordinate information, the tracking branch input feature is generated as the input of the Transformer encoder, and similarly, the spliced Q' and The flattened operation is performed, and after being spliced with spatial coordinate information, the tracking branch input feature is generated as the input of the Transformer encoder, and similarly, the spliced Q' and Step S3: performing feature fusion of different modalities; Step S4: model training and reasoning to obtain a prediction result.

2. The Transformer architecture based natural language object tracking method according to claim 1, wherein: The step S2 specifically comprises the following steps: Step S21: the number of channels of the channel group is reduced to C. , , the number of channels of the channel group is reduced to C. wherein, denotes the visual search template feature map In a space with domain length and width of , dimension of , denotes the visual search region feature map In a space with domain length and width of , dimension of , denotes the language feature Q' in a space with domain language feature length of dimension of ; Step S22: flattening and concatenating the feature maps and along the spatial dimensions, resulting in a sequence of length with dimension C of the tracking features, which is used as input to the image template Transformer encoder; Step S23: flattening the feature map and language features are concatenated to form a sequence of length denoting features as input to a language Transformer encoder; Step S24: using a Transformer structure as an encoder structure of the target image template branch and the language branch, which is composed of N encoder layers, and each layer is composed of a multi-head self-attention module with a feedforward network; Step S25: the decoder is composed of M decoder layers, and each layer is composed of a self-attention, a codec attention and a feedforward network.

3. The Transformer architecture based natural language object tracking method according to claim 2, characterized in that: In the step S24, a three-layer perceptron with a ReLU activation function is used to predict the normalized center coordinates, height and width of the target object bounding box.

4. The Transformer architecture based natural language object tracking method according to claim 1, wherein: The step S3 is feature fusion of visual and language data features, which specifically comprises the following steps: Step S31: fuse the search image features extracted in the template encoder with the template features, denoted as features , the search image features with the language features, denoted as features , the first step of the feature fusion adds features and features to obtain the initial fused features : = Formula One Step S32: Use a 1*1 convolution to... The number of channels is reduced to 1, and the Softmax function is used to obtain the normalized attention map. Attention map It is to feature The formula obtained by performing the same operation is as follows: Equation Two Equation Three Step S33: using and extract the multi-modal global context information by matrix multiplication respectively, the formula is as follows: Formula Four Formula Five and are global multi-modal feature information extracted from global features and from global features , global features Step S34: adding to its original corresponding decoder output features and to enhance multi-modal characteristics and connect them as inputs to the fusion prediction head , , , as follows; Formula Six.

5. The Transformer architecture based natural language object tracking method according to claim 1, wherein: The step S4 specifically comprises the following steps: Step S41: randomly selecting an image from a video as a target frame, finding a previous frame of the target frame, then respectively cropping and adjusting the sizes of the two frames to obtain a target image template Z and a corresponding searched picture sequence X, and constructing a training tuple (Z, Q, X) for model training; Step S42: reasoning: for each input image, the network takes the search region centered on the prediction or standard bounding box of the previous frame of the current frame as input, and outputs the prediction result corresponding to each branch, and takes the fusion output as the final result.

6. The Transformer architecture based natural language object tracking method according to claim 5, characterized in that: In the step S41, three losses of the tracking branch, the reference branch and the mixed branch are respectively calculated, and the formulas are as follows: Formula Seven Formula Eight Formula Nine obtaining total loss as follows: Formula Ten where, represents the calculation of intersection over union loss, represents the calculation of paradigm loss, represents the real bounding box, , , respectively represent the prediction box of the tracking, fusion, and reference branch, , , , , , is a hyperparameter that adjusts the loss, and makes the and of the three branches consistent.

7. The Transformer architecture based natural language object tracking method according to claim 5, characterized in that: In the step S42, the first frame is fixed as a template image in subsequent video frames, and the target image template Z and the language description feature Q are fixed in all frames.

Images