Visual object tracking method based on natural language and target state information

Abstract

The application discloses a visual target tracking method based on natural language and target state information, and comprises the following steps: step (1), constructing a training sample set; step (2), constructing a visual target tracking model based on natural language and target state information; step (3), adjusting parameters of an image-text encoder and loading pre-training weights to obtain features of text and a first template after fusion, features of a second template and features of a search image; step (4), fusing position information of a target in a sample set and boundary box information of the target into the features of the second template; step (5), obtaining features after joint modeling; step (6), obtaining tokens containing target position information after query; step (7), obtaining a predicted target boundary box regression result; and step (8), obtaining a final tracking result. The application effectively improves the tracking accuracy of a visual tracker based on natural language.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision tracking, and particularly relates to a visual target tracking method based on natural language and target state information. BACKGROUND

[0002] The purpose of target tracking using natural language is to estimate the state of the target in the subsequent video frame based on the given natural language description about the target in the first frame. Compared with the traditional visual tracking which uses a bounding box in the first frame to specify the tracking target, the natural language based target tracking provides a more user-friendly way to specify the tracking target. Therefore, the natural language based tracking has a wide application prospect in the fields of video monitoring and robots.

[0003] The key of the natural language based tracking is to make the image understand the semantic information of the text. The current commonly used method is the joint modeling method. The main idea of the joint modeling method is to concatenate the features of the natural language, the template image and the search image along the sequence dimension and then perform joint modeling through the self-attention mechanism. The joint modeling method is widely used, which uses the self-attention mechanism to establish the connection between the text and the image, so that the model can automatically learn the complex relationship between the two modalities.

[0004] Although the joint modeling method has achieved great success, the interaction between the text and the search image may mislead the tracker. Specifically, the language description about the target is based on the first frame, and when the target changes in the subsequent frame, the text description about the target may make the tracker track the wrong target. In addition, the natural language is based on the whole image to describe the target, while the input to the tracker is the search region near the target, so it is likely that the target in the search region will not be consistent with the target described by the natural language.

[0005] On the other hand, in recent years, visual language models have developed rapidly, but the field of natural language tracking lacks the application of these achievements.

[0006] Most of the current researches directly migrate the natural language field pre-trained feature extractor to extract text features. However, since the feature extractors of images and texts are trained using completely different data, the features extracted by the two feature extractors will be in different feature spaces, and there will be a serious semantic misalignment (the cosine similarity between dog and cat may be smaller).

[0007] In addition, there is a significant performance gap between natural language-based tracking and traditional visual tracking. First, natural language does not contain accurate target edge information compared to using a bounding box to specify the target in traditional visual tracking, so the accuracy of the target bounding box obtained through natural language is poor. Second, the natural language-based tracking model has similar parameters to traditional visual tracking models, but the amount of available training data is greatly reduced, resulting in poor generalization ability of the tracker. SUMMARY

[0008] In order to overcome the deficiencies of the prior art, the purpose of the present application is to provide a visual target tracking method based on natural language and target state information, which uses two templates to fuse with natural language and target state information respectively for tracking, effectively improves the tracking accuracy of the visual tracker based on natural language by combining different information, and solves the problem of mismatch between the text description of the target and the target in the video frame in the prior art and the lack of edge information related to the target in natural language.

[0009] In order to achieve the above purpose, the technical scheme adopted by the present application is:

[0010] The visual target tracking method based on natural language and target state information comprises the following steps:

[0011] Step (1): obtaining annotated training videos from a network open source dataset, and performing preprocessing operations on the training videos, and constructing a training sample set using the preprocessed video frames;

[0012] Step (2): constructing a visual target tracking model based on natural language and target state information using a target state information fusion module, a feature joint modeling module, a target query module, a classification regression network, and a pre-trained image-text encoder;

[0013] Step (3): adjusting the parameters of the image-text encoder and loading the pre-trained weights, using the image-text encoder to extract features from the text description, template image and search image in the input sample set, obtaining the features of the text and the first template fusion, the second template features and the search image features;

[0014] Step (4): fusing the position information of the target and the bounding box information of the target in the sample set into the second template features through the target state information fusion module;

[0015] Step (5): concatenating the search image features, the fusion features of the text and the first template, and the second template features with fusion information along the spatial dimension, and inputting them into the feature joint modeling module to obtain the joint modeling features;

[0016] Step (6): input the combined features and a learnable query token into a target query module to obtain a token containing target location information after query;

[0017] Step (7): obtain a predicted target bounding box regression result by using the token after query, the combined features, and the search image features;

[0018] Step (8): calculate a tracking loss from the real target bounding box and the predicted target bounding box, then iteratively correct the tracking result to optimize the model parameters, and load the optimized model parameters into the model to obtain the final tracking result.

[0019] In the step (1), the preprocessing includes random flipping, color jittering, window jittering, size scaling, and normalization.

[0020] The sample set includes a text description of the target, two template images, one search image, target location information and target bounding box information (generated from the location of the target in the template) corresponding to the two template images; the two template images are respectively referred to as a first template and a second template. The first template image is the first frame of a video sequence, which is fused with the text description, and the second template image is updated to a tracking result with higher accuracy, which is fused with the target location information and the bounding box information. The two templates differ in that they fuse different information, and are connected in that they are both used as templates to guide the tracker to track the target.

[0021] In the step (2), the image-text encoder includes 12 Transformer encoding layers; the target state information fusion module includes a linear mapping layer for mapping the template target state information into a feature space; the feature joint modeling module includes 6 Transformer encoding layers for modeling the relationship between the template and the search image; the target query module includes 6 Transformer decoding layers for searching the target from the modeled search image features; and the classification regression network is composed of a 4-layer convolutional neural network.

[0022] Specifically, first, the image-text encoder is used to extract and fuse the features of the first template and the text description, then the target state information module is used to fuse the features of the second template and the corresponding target location information and target bounding box information, then the fused first template features and second template features are spliced together along the spatial dimension and input into the feature joint modeling module, then the target query module is used to perform target query on the combined features, and finally the query result and the combined features are input into the classification regression network to obtain the target-background classification result and the target bounding box regression result.

[0023] In the step (3), the parameters of the image-text encoder include input image size, input channel number and output channel number.

[0024] In the step (4), the position information of the target and the bounding box information of the target are fused into the features of the second template by a target state information fusion module, and the implementation steps are as follows:

[0025] (4a1) The bounding box coordinates of the target are mapped to the template feature space (with the same size as the template feature) using a linear mapping layer.

[0026] (4a2) The target position information is multiplied element by element with the query token to obtain the target position information mapped to the template feature space.

[0027] (4a3) The target bounding box information and the target position information are directly added to the second template feature to obtain the second template feature after the target position information is fused.

[0028] In the step (5), the features are sequentially input into 6 Transformer encoding layers in the feature joint modeling module to obtain the modeled features, and the implementation steps of each Transformer encoding layer are as follows:

[0029] (5a1) The input features are mapped to query Q e , key K e and value V e by three matrices respectively.

[0030] (5a2) The self-attention of the input features is calculated according to Q e , K e and V e , as shown in the following formula:

[0031]

[0032] In the formula, d k represents the channel dimension of Q e , Attention e represents the result of feature self-attention calculation, and softmax is a mathematical operator.

[0033] (5a3) The output after attention is input into a feedforward neural network to obtain the output of the Transformer encoding layer, and the feedforward neural network is composed of two linear layers and an activation layer.

[0034] In step (6), the query token and features are sequentially input into six Transformer decoding layers in the target query module to obtain the queried token. The implementation steps of each Transformer decoding layer are as follows:

[0035] (6a1) Map the input query token to Q using a matrix. d The input features are mapped to K using two matrices. d V d ;

[0036] (6a2) According to Q d ,K d V d The cross-attention between the input query token and the input features is calculated as shown in the following formula:

[0037]

[0038] In the formula d k Q represents d Attention d This represents the result of cross-attention calculation;

[0039] (6a3) The output Attention after attention calculation d The input is fed into a feedforward neural network, which is used to introduce nonlinear transformations into the features to better adapt the features to the tracking task, and obtains the output of the Transformer decoding layer. The feedforward neural network consists of two linear layers and one activation layer.

[0040] In step (7), the queried token and the jointly modeled features are input into the classification and regression network. The queried token and the jointly modeled search image features are multiplied channel by channel and then summed to obtain the predicted target-background classification result. The queried token and the jointly modeled search image features are multiplied channel by channel and then input into a 4-layer convolutional neural network to obtain the predicted target bounding box regression result; as shown in the following formula:

[0041]

[0042] Q in the formula t This represents the token obtained after the query, with a size of 1×1×d. k , The search image features output by the joint feature modeling module are 1×n×d. k n represents The length of the feature, · operation representation in d k Perform element-wise multiplication along the dimension.

[0043] In step (8), the tracking loss is calculated from the real target bounding box and the predicted target bounding box, and the tracking loss is obtained by two loss functions:

[0044] The classification loss is calculated using the mean square error:

[0045]

[0046] In the above formula, represents the value of the real classification result at position (m, n), represents the value of the predicted classification result at position (m, n), and h and w are the height and width of the search region feature, respectively;

[0047] The bounding box regression loss is calculated using the GIoU loss:

[0048]

[0049] The regression network generates a target bounding box for each position on the search feature, and in the above formula, b m,n represents the target bounding box generated at position (m, n), represents the real target bounding box at position (m, n);

[0050] The total tracking loss is:

[0051] L pre = 100L cls + L giou .

[0052] Advantages of the present application:

[0053] First, the present application can specify a target using only natural language, and can also specify a target using natural language and a target bounding box at the same time, effectively improving the application range of the visual target tracking method.

[0054] Second, the present application proposes to limit the interaction between text and search images, by limiting the text to be fused only with the first template, and by fusing the text information and the target state information through the attention mechanism of the Transformer, effectively solving the problem that the text description about the target does not match the subsequent template due to changes in the position or shape of the target, and thus causing the text description to mislead the tracking process, effectively improving the accuracy of tracking.

[0055] Third, the present application proposes to add the state information of the target to the template, by adding the position information and the bounding box information of the target to the features of the template to guide the tracker to perceive the position and edge information of the target, effectively improving the tracking accuracy of the tracker. BRIEF DESCRIPTION OF DRAWINGS

[0056] Figure 1 is the overall implementation flowchart of the present application.

[0057] Figure 2 is the implementation flowchart of the target state information fusion module in the present application.

[0058] Figure 3 is a visualization schematic diagram of the tracking results of the tracker in the video sequence. DETAILED DESCRIPTION

[0059] The present application will be further described in detail below with reference to the accompanying drawings.

[0060] Embodiment one: with reference to the accompanying drawings Figures 1-2 The present application proposes a visual target tracking method based on natural language and target state information, which specifically includes the following steps:

[0061] Step 1) Obtain annotated training videos from network open source data sets, and perform preprocessing operations thereon, and use the preprocessed video frames to construct a training sample set, wherein the sample set includes a text description of the target, two template images, and one search image; the two template images are respectively denoted as the first template and the second template; the preprocessing operations used in this embodiment include random flipping, color jittering, window jittering, size scaling, and normalization;

[0062] Preprocessing the training data through various data enhancement methods can improve the generalization ability and robustness of the model in different scenarios; using the structure of two template images and one search image, one template fusing text information and one template fusing target state information, can help improve the adaptability to target dynamic changes.

[0063] Step 2) Use a pre-trained Beit3 image-text encoder, a target state information fusion module, a feature joint modeling module, a target query module, and a classification regression network to construct a visual target tracking model based on natural language and target state information; with reference to Figure 1 , the Beit3 image-text encoder contains 12 Transformer encoding layers from top to bottom; the target state information fusion module contains a linear mapping layer for mapping the template target state information into the feature space; the feature joint modeling module contains 6 Transformer encoding layers for modeling the relationship between the template and the search image; the target query module contains 6 Transformer decoding layers for searching the target from the modeled search image features; the classification regression network is composed of a 4-layer convolutional neural network;

[0064] The pre-trained Beit3 model can provide powerful image-text feature extraction and fusion capabilities, reduce training costs, and improve feature extraction quality; the hierarchical modular design facilitates fine control of information processing at each stage, which helps to accurately locate and robustly track.

[0065] Step 3) Adjust the parameters of the image-text encoder and load the pre-trained weights, use the image-text encoder to extract features from the input text description, template image and search image, obtain the features after the text is fused with the first template, the features of the second template and the features of the search image; In this embodiment, the parameters of the image-text encoder include input image size, input channel number and output channel number;

[0066] Using a pre-trained model to quickly obtain good initial feature representation can effectively improve training efficiency and model performance; by adapting input and output parameters to ensure that the encoder seamlessly interfaces with downstream tracking tasks, the compatibility and stability of the overall system are improved.

[0067] Step 4) The target state information fusion module fuses the target position information and the target bounding box information into the second template feature; refer to Figure 2 The specific implementation steps of fusing target state information in this embodiment are as follows:

[0068] (4a1) Use a linear mapping layer to map the target bounding box coordinates to the template feature space (same size as the template feature);

[0069] (4a2) Use the query token to multiply the target position information element by element to obtain the target position information mapped to the template feature space;

[0070] (4a3) Add the target bounding box information and the target position information directly to the second template feature to obtain the second template feature after fusing the target position information.

[0071] Injecting the target position information and the target bounding box information into the template feature can significantly improve the model's perception of the target's spatial features, thereby enhancing the model's ability to model target state changes and further improving the robustness to complex situations such as occlusion and scale changes.

[0072] Step 5) Concatenate the first template feature fused with the text, the second template feature, and the search image feature along the spatial dimension, and input it into the feature joint modeling module. In the module, the features are sequentially input into 6 Transformer encoding layers to obtain the modeled features; the specific implementation steps of the Transformer encoding layer in this embodiment are as follows:

[0073] (5a1) Map the input features to Q eK e V e Three values;

[0074] (5a2) According to Q e K e V e , the self-attention of the input features is calculated, as shown in the following formula:

[0075]

[0076] In the formula, d k represents the channel dimension of Q e , Attention e represents the result of the feature self-attention calculation, and softmax is a mathematical operator.

[0077] (5a3) The output after attention is input into a feedforward neural network composed of two linear layers and an activation layer to obtain the output of the Transformer encoding layer.

[0078] Jointly modeling the target information of the text modality and the RGB modality through self-attention helps to improve the adaptability of the model to target changes; the multi-layer Transformer encoder enhances the context modeling capability, enabling the model to fully understand the relevance between the template and the search image and improving the accuracy of target relevance modeling.

[0079] Step 6) Input the jointly modeled features and a query token into a target query module, in which the query token and the features are sequentially input into 6 Transformer decoding layers to obtain tokens containing target position information after query; in this embodiment, the specific implementation steps of the Transformer decoding layer are as follows:

[0080] (6a1) Map the input query token to Q d through a matrix, map the input features to K d V d through two matrices;

[0081] (6a2) According to Q d K d V d , calculate the cross-attention between the input query token and the input features, as shown in the following formula:

[0082]

[0083] In the formula, d k represents the channel dimension of Q d , Attentiond This represents the result of cross-attention calculation;

[0084] (6a3) The output after attention is input into a feedforward neural network to obtain the output of the Transformer decoding layer. The feedforward neural network consists of two linear layers and one activation layer.

[0085] Using query tokens to guide the Transformer decoder to find the target location in the modeled features helps to explicitly construct the target localization process, which can effectively improve the model's target retrieval efficiency and accuracy in complex scenarios.

[0086] Step 7) Input the queried token and the jointly modeled features into the classification and regression network. Multiply the queried token and the jointly modeled search image features channel by channel and then sum them to obtain the predicted target-background classification result. Multiply the queried token and the jointly modeled search image features channel by channel and then input them into a 4-layer convolutional neural network to obtain the predicted target bounding box regression result.

[0087] Step 8) Calculate the tracking loss using the true target bounding box and the predicted target bounding box, then iteratively refine the tracking results to optimize the model parameters. Load the optimized model parameters into the model to obtain the final tracking result. In this embodiment, the tracking loss is obtained from two loss functions, and the specific calculation formulas are as follows:

[0088] Calculate classification loss using mean squared error:

[0089]

[0090] In the above formula This represents the value of the true classification result at position (m, n). The value at position (m,n) represents the predicted classification result, and h and w are the height and width of the search region feature, respectively.

[0091] Calculate the bounding box regression loss using GIoU loss:

[0092]

[0093] The regression network generates a target bounding box for each location on the search feature, where b in the above formula... m,n This represents the target bounding box generated at position (m,n). This represents the actual target bounding box at position (m,n).

[0094] The total tracking loss is:

[0095] L pre =100Lcls +L giou .

[0096] By combining the classification loss with the GIoU regression loss to construct the overall optimization target, the classification accuracy and the boundary box prediction quality can be effectively improved; by training the model in an iterative optimization manner, the optimal parameters can be gradually converged, and the final tracking performance and stability can be significantly improved.

[0097] As shown in Figure 3 , the tracking results of the tracker in the video sequence are visualized, and the left first column is the specified target, and the right four columns are the tracking results of the tracker.

[0098] The part of the present application not described in detail belongs to the common knowledge of those skilled in the art.

[0099] The above is a specific description of the preferred implementation of the present application, but the present application is not limited to the described embodiments, and those skilled in the art can make various equivalent modifications or replacements without departing from the spirit of the present application. These equivalent modifications or replacements are all included in the scope defined by the claims of the present application.

Claims

1. A visual target tracking method based on natural language and target state information, characterized in that, Includes the following steps; Step (1): Obtain labeled training videos from open-source datasets on the network, perform preprocessing operations on the training videos, and construct a training sample set using the preprocessed video frames; Step (2): Construct a visual target tracking model based on natural language and target state information using the target state information fusion module, feature joint modeling module, target query module, classification and regression network, and pre-trained image-text encoder; In step (2), the image-text encoder contains 12 Transformer encoding layers; the target state information fusion module contains a linear mapping layer, which is used to map the template target state information into the feature space; The feature joint modeling module contains 6 Transformer encoding layers for modeling the relationship between the template and the search image; the target query module contains 6 Transformer decoding layers for finding the target from the modeled search image features; the classification and regression network consists of a 4-layer convolutional neural network. Step (3): Adjust the parameters of the image-text encoder and load the pre-trained weights. Use the image-text encoder to extract features from the text description, the first template image, the second template image and the search image in the input sample set to obtain the features of the text fused with the first template, the features of the second template and the features of the search image. The first template image is the first frame of the video sequence, which is fused with the text description. The second template image is continuously updated to reflect the tracking results with higher accuracy. Step (4): The target location information and target bounding box information in the sample set are fused into the second template feature through the target state information fusion module; Step (5): The search image features, the fusion features of the text and the first template, and the second template features of the fusion information are spliced ​​together along the spatial dimension and input into the feature joint modeling module to obtain the features after joint modeling; Step (6): Input the jointly modeled features and a learnable query token into the target query module to obtain a token containing the target location information after the query; Step (7): Obtain the predicted target bounding box regression result by using the queried token, the jointly modeled features, and the search image features; Step (8): Calculate the tracking loss from the real target bounding box and the predicted target bounding box, then iteratively correct the tracking results to optimize the model parameters, and load the optimized model parameters into the model to obtain the final tracking result.

2. The visual target tracking method based on natural language and target state information according to claim 1, characterized in that, In step (1), the preprocessing includes random flipping, color dithering, window dithering, size scaling, and normalization. The training sample set includes a text description of the target, two template images, a search image, and the target location information and target bounding box information corresponding to the two template images; the two template images are denoted as the first template and the second template, respectively. The first template image is the first frame of the video sequence, which is fused with the text description. The second template image is continuously updated to a more accurate tracking result, which is fused with the target's position information and bounding box information.

3. The visual target tracking method based on natural language and target state information according to claim 2, characterized in that, Step (2) Specifically, firstly, the image-text encoder extracts and fuses the features of the first template and the text description. Then, the target state information module fuses the features of the second template with the corresponding target location information and target bounding box information. Next, the fused features of the first and second templates are concatenated along the spatial dimension and input into the feature joint modeling module. Then, the target query module performs a target query on the jointly modeled features. Finally, the query results and the jointly modeled features are input into the classification and regression network to obtain the target-background classification results and the target bounding box regression results.

4. The visual target tracking method based on natural language and target state information according to claim 3, characterized in that, In step (3), the parameters of the image-text encoder include the input image size, the number of input channels, and the number of output channels.

5. The visual target tracking method based on natural language and target state information according to claim 4, characterized in that, In step (4), the target's position information and the target's bounding box information are fused into the features of the second template through the target state information fusion module. The steps are as follows: (4a1) Use a linear mapping layer to map the bounding box coordinates of the target to the template feature space; (4a2) Multiply the query token and the target location information element by element to obtain the target location information mapped to the template feature space; (4a3) The target bounding box information and the target location information are directly added to the second template feature to obtain the second template feature after fusing the target location information.

6. The visual target tracking method based on natural language and target state information according to claim 5, characterized in that, In step (5), the features are sequentially input into six Transformer encoding layers in the feature joint modeling module to obtain the modeled features. The implementation steps of each Transformer encoding layer are as follows: (5a1) The input features are mapped to the query using three matrices. ; (5a2) According to The self-attention mechanism for the input features is calculated using the following formula: in the formula express The channel dimension, This represents the result of feature self-attention calculation. It is a mathematical operator; (5a3) The output after attention is input into a feedforward neural network to obtain the output of the Transformer encoding layer, which consists of two linear layers and one activation layer.

7. The visual target tracking method based on natural language and target state information according to claim 6, characterized in that, In step (6), the query token and features are sequentially input into six Transformer decoding layers in the target query module to obtain the queried token. The implementation steps of each Transformer decoding layer are as follows: (6a1) Map the input query token to a matrix. The input features are mapped to two matrices. ; (6a2) According to The cross-attention between the input query token and the input features is calculated as shown in the following formula: in the formula express The channel dimension, This represents the result of cross-attention calculation; (6a3) Output after attention calculation The input is fed into a feedforward neural network, which is used to introduce nonlinear transformations into the features to better adapt the features to the tracking task, and obtains the output of the Transformer decoding layer. The feedforward neural network consists of two linear layers and one activation layer.

8. The visual target tracking method based on natural language and target state information according to claim 7, characterized in that, In step (7), the queried token and the jointly modeled features are input into the classification and regression network. The queried token and the jointly modeled search image features are multiplied channel by channel and then summed to obtain the predicted target-background classification result. The queried token and the jointly modeled search image features are multiplied channel by channel and then input into a 4-layer convolutional neural network to obtain the predicted target bounding box regression result; as shown in the following formula: in the formula This represents the token obtained after the query, with a size of [value missing]. , The search image features output by the joint feature modeling module are represented by a size of [size missing]. n represents The length of the feature Operation is indicated in Perform element-wise multiplication along the dimension.

9. The visual target tracking method based on natural language and target state information according to claim 8, characterized in that, In step (8), the tracking loss is calculated from the true target bounding box and the predicted target bounding box. The tracking loss is obtained from two loss functions: Calculate classification loss using mean squared error: In the above formula This indicates the true classification result at the location. The value at that location, This indicates the predicted classification result in the location. The value at that location, and These are the height and width of the search region features, respectively; Calculate the bounding box regression loss using GIoU loss: The regression network generates a target bounding box for each location on the search feature, as shown in the above formula. Indicates the location The target bounding box generated at that location, Indicates the location Find the true target bounding box; The total tracking loss is: 。

Images