A visual object tracking method based on multimodal large language model

Abstract

The present invention discloses a visual target tracking method based on a multimodal large language model, which utilizes the knowledge in the multimodal large language model to generate a text description of the tracked target, thereby enhancing the performance of visual tracking. First, the present invention proposes a text optimization process based on the feedback of the large language model, and optimizes the text description through multiple iterations to obtain an accurate text description. In addition, the present invention also proposes a foreground verification process, which improves the tracking accuracy by making full use of the relationship between the foreground target and the background target. Compared with the prior art, the advantages of the present invention are: a) there is no need to manually annotate the text description of the tracked target; b) it solves the hallucination problem of the large language model and the problem of misalignment between the visual and language modalities; c) it has strong robustness to extreme environments such as target deformation, motion blur and target occlusion.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and in particular relates to a visual target tracking method based on a multimodal large language model in the technical field of target tracking of digital images. Background Art

[0002] Visual object tracking technology can predict the state of an object in subsequent video frames given an initial object state. However, since objects in videos may face challenges such as shape changes and occlusions, robust tracking cannot be achieved using only visual information. Existing methods have the following limitations: 1) Manual text description of the tracked object is required; 2) The alignment between text and visual information is poor, and the tracker cannot fully understand the text information; 3) When the text description is inaccurate, the tracking performance will drop sharply. Summary of the invention

[0003] The present invention proposes a new visual target tracking method, which can use both text and visual information for robust tracking. At the same time, the present invention also proposes a text optimization process based on large language model feedback, which guides the large language model to generate more accurate text by converting the visual output of the tracker into text feedback. Compared with the existing methods, the method proposed by the present invention has the following advantages: 1) No manual text description of the tracking target is required. 2) The text and visual information have strong alignment capabilities, and the tracker can make full use of text information to improve tracking performance. 3) When the text description is inaccurate, the tracker can adaptively rely on visual information for tracking, and there will be no problem of drastic performance degradation.

[0004] The specific technical solution for achieving the purpose of the present invention is:

[0005] A visual target tracking method based on a multimodal large language model is characterized by comprising the following specific steps:

[0006] Step 1: Input the template image into the feedback prompt optimization process based on the large language model to obtain the foreground description T fore and background description T back ;

[0007] Step 2: Transform the t-th frame image x t and prospect description T fore Input into the language-based visual localization model to obtain a series of corresponding foreground candidate boxes Then take the t-th frame image x t and background description T back Input into the language-based visual localization model to obtain a series of corresponding background candidate boxes

[0008] Step 3: Foreground candidate box and background candidate boxes During the input foreground verification process, the image tracking target result of the tth frame is obtained in:

[0009] The step 1 specifically includes:

[0010] 1.1. Passing template image I to multimodal large language model F MLLM In the above example, we generate a description of the image content T init ;in represents a set of real numbers, H represents the height of the image, and W represents the width of the image;

[0011] T init =F MLLM (I)

[0012] F MLLM Represents a multimodal large language model;

[0013] 1.2. Mark the tracked target with a bounding box in image I and create a new image input I m , will I m and T init Pass it into the multimodal large language model to generate a prospect description and background description

[0014]

[0015] F MLLM Represents a multimodal large language model;

[0016] Among them, T init Represents the description of the template image content, I m Indicates the image after the tracking target is selected;

[0017] 1.3. Describe the prospects The template image I is passed into the language-based visual positioning model to obtain the target candidate box and alignment score Where N represents the number of candidate boxes, M represents The length of i represents the number of iterations, and the value range of i is [0-N iter ];

[0018] P z ,S z =F GVLM +(I,T fore )

[0019] 1.4. Classified as positive descriptive words and negative descriptors The classification method is as follows:

[0020]

[0021]

[0022]

[0023]

[0024] Among them, θ2 represents the alignment score threshold, θ1 and θ3 are the intersection-over-union ratio calculation thresholds for the foreground and background respectively;

[0025] It indicates the intersection over union (IoU) calculation between the nth candidate box and the real target area. z ,G) is calculated as follows:

[0026]

[0027] Among them, Area_of_Overlap(P z ,G) represents the candidate box P z and the intersection area between the real target area G, Area_of_Union(P z ,G) represents the candidate box P z and the union area between the true target area G;

[0028] According to S z and IoU z (P z ,G) comparison:

[0029] When S z ≤IoU z (P z ,G) Positive Description

[0030] When S z >IoU z (P z ,G) Positive Description

[0031] Thus, the prospect description Positive description in and negative description

[0032] 1.5. IoU z (Pz ,G) is compared with the set threshold ∈ to determine:

[0033] If IoU(P max ,G)<∈,P max represents the candidate box with the largest evaluation score, then and and After being passed into the multimodal large prediction model for optimization, a new prospect description is obtained Background description reserved And continue to repeat steps 1.3, 1.4 and 1.5;

[0034] If IoU(Pmax,G)≥∈, then the current accurate foreground description T fore and background description T back Output; if i>N iter , then the current outlook is expressed as and background description As foreground description T fore and background description T back Output;

[0035] The step 3 specifically includes:

[0036] 3.1. Calculate foreground candidate boxes Prospect score The process is as follows:

[0037]

[0038] Where max() means taking the maximum value of the parameters in the brackets, z means the target template in the first frame, is in the bounding box The cropped image inside, f is a convolutional neural network used to extract image features, f T represents transposed convolution; “||f||2” represents the modulus length of the f vector;

[0039] Calculation of background score As shown below:

[0040]

[0041] in, express and The intersection-and-union ratio between ; it is calculated as follows:

[0042]

[0043] Among them, A and B represent two regions, A∩B is their intersection, and A∪B is their union;

[0044] 3.3. Combined foreground score W fore and background score W back To determine the candidate box confidence W all =[s 1 ,s 2 ,…,s N ], the specific calculation method is as follows:

[0045]

[0046] 3.4. Score s i The highest foreground candidate The image x tracked at frame t t The output is the tracking target result

[0047] The present invention can solve the hallucination problem of large language models and automatically generate accurate text descriptions of tracking targets, thereby improving tracking effects. It is not necessary to manually describe the text descriptions of tracking targets.

[0048] The present invention can utilize the rich language information in the text to improve the performance of visual tracking and has strong robustness to extreme environments such as target deformation, motion blur, and target occlusion.

[0049] The present invention can improve the alignment between text and visual modality, so that the tracker can better understand the text information. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 is a flow chart of an embodiment of the present invention;

[0051] Figure 2 It is a flow chart of the present invention;

[0052] Figure 3 Schematic diagram of the language-based visual localization model.

[0053] Specific implementation method

[0054] The present invention is described in detail below with reference to the accompanying drawings and embodiments. Example

[0055] See also Figure 1 , a visual target tracking method based on a multimodal large language model of this embodiment includes the following steps:

[0056] Step 1: Input the template image into the feedback prompt optimization process based on the large language model to obtain the foreground description T fore

[0057] and background description T back ;

[0058] Step 2: Transform the t-th frame image x t and prospect description T fore Input into the language-based visual localization model to obtain a series of corresponding foreground candidate boxes Then take the t-th frame image x t and background description T back Input into the language-based visual localization model to obtain a series of corresponding background candidate boxes

[0059] Step 3: Foreground candidate box and background candidate boxes During the input foreground verification process, the image tracking target result of the tth frame is obtained

[0060] See also Figure 2 In step 1, the template image is input into the feedback prompt optimization process based on the large language model to obtain the foreground description T fore and background description T back , including the following steps:

[0061] 1.1. Pass in the template image In the multimodal large language model, generate a description of the image content T init . (H represents the height of the image, W represents the width of the image).

[0062] T init =F MLLM (I)

[0063] 1.2. Mark the tracked target with a bounding box in image I and create a new image input I m , will I m and T init Pass it into the multimodal large language model to generate a prospect description and background description

[0064]

[0065] Among them, T init Represents the description of the template image content, I m Represents the image after the tracking target is selected.

[0066] 1.3 See also Figure 3 , describe the prospect The template image I is passed into the language-based visual positioning model to obtain the target candidate box and alignment score Where N represents the number of candidate boxes, M represents The length of i represents the number of iterations, and the value range of i is [0-N iter ].

[0067] P z ,S z =F GVLM +(I,T fore )

[0068] Among them, the specific processing method of the language-based visual positioning model is as follows:

[0069] 1.3.1 Perform text feature extraction:

[0070] First, perform word embedding processing and input the text into Then, the word embedding is input into the BERT model (a bidirectional encoder representation model based on Transformer) to obtain the text features: (Here d l =768, M is the number of words in the reference language);

[0071] 1.3.2 Then perform image feature extraction:

[0072] Image is the number of color channels, H0 is the original height of the image, and W0 is the original width of the image) input into ResNet-50, and we get Then use 1×1 convolution to reduce the channel dimension of feature map f from C to d v =256, and a new feature map is obtained Then flatten f′ to get the image features

[0073] 1.3.3 Then perform text-image feature fusion:

[0074] The image feature f v As K, V, text features f l As Q input Transformer network, output fused multimodal feature representation

[0075] 1.3.4Finally, perform predictive regression:

[0076] The bounding box regression module aims to regress the position of the final bounding box in the image based on the model's understanding of the semantic alignment relationship between cross-modal information. The specific calculation method is as follows:

[0077]

[0078]

[0079] In the formula, is the output of each layer of the cross-modal fusion network, σ is the activation function, MLP multi-layer perceptron, P z is the predicted bounding box coordinates {x, y, w, h}, S z is the alignment score of the bounding box, (x, y) is the coordinate of the upper left corner of the bounding box, and w and h are the width and height of the bounding box respectively.

[0080] 1.4. Classified as positive descriptive words and negative descriptors The classification method is as follows:

[0081]

[0082]

[0083]

[0084]

[0085] Among them, θ2 represents the alignment score threshold, θ1 and θ3 are the intersection-over-union ratio calculation thresholds for the foreground and background respectively;

[0086] It indicates the intersection over union (IoU) calculation between the nth candidate box and the real target area. z ,G) is calculated as follows:

[0087]

[0088] Among them, Area_of_Overlap(P z ,G) represents the candidate box P z and the intersection area between the real target area G, Area_of_Union(P z ,G) represents the candidate box P z and the union area between the true target area G;

[0089] According to S z and IoU z (P z ,G) comparison:

[0090] When S z ≤IoU z (P z ,G) Positive Description

[0091] When S z >IoU z (P z,G) Positive Description

[0092] Thus, the prospect description Positive description in and negative description

[0093] 1.5. IoU z (P z ,G) is compared with the set threshold ∈ to determine:

[0094] If IoU(P max ,G)<∈,P max represents the candidate box with the largest evaluation score, then and and After being passed into the multimodal large prediction model for optimization, a new prospect description is obtained Background description reserved And continue to repeat steps 1.3, 1.4 and 1.5;

[0095] If IoU(Pmax,G)≥∈, then the current accurate foreground description T fore and background description T back Output; if i>N iter , then the current outlook is expressed as and background description As foreground description T fore and background description T back Output;

[0096] The hyperparameter settings involved are as follows:

[0097] θ2 represents the alignment score threshold, θ1 and θ3 are the intersection-over-union ratio calculation thresholds for the foreground and background, respectively. In this embodiment, ∈=0.4, θ2=0.2, θ1=0.3, and θ3=0.1 are set.

[0098] Step 2: Transform the t-th frame image x t and prospect description T fore Input into the language-based visual localization model to obtain a series of corresponding foreground candidate boxes Then take the t-th frame image x t and background description T back Input into the language-based visual localization model to obtain a series of corresponding background candidate boxes

[0099] Step 3: Foreground candidate box and background candidate boxes During the input foreground verification process, the image tracking target result of the tth frame is obtained Specifically include:

[0100] 3.1 Calculate foreground candidate boxes Prospect score As shown below:

[0101]

[0102] Where max() means taking the maximum value of the parameters in the brackets, z means the target template in the first frame, is in the bounding box The cropped image inside, f is a convolutional neural network used to extract image features, f T represents transposed convolution; “||f||2” represents the modulus length of the f vector;

[0103] 3.2: Calculate background candidate boxes The background score W back = As shown below:

[0104]

[0105] in, express and The intersection over union ratio (Intersection over Union) between them is calculated as follows:

[0106]

[0107] Among them, A and B represent two areas, A∩B is their intersection, and A∪B is their union.

[0108] 3.3: Combined foreground score W fore and background score W back To determine the candidate box confidence W all =[s 1 ,s 2 ,…,s N ], the specific calculation method is as follows:

[0109]

[0110] 3.4: Score s i The highest foreground candidate The image x tracked at frame t t The output is the tracking target result

[0111] In order to verify the effectiveness of the method proposed in this example, an ablation study was performed on three variants of the model:

[0112] Visual Tracking Model (VTM), which uses a visual tracker exclusively to perform tracking tasks. In the ablation study, MixFormer-1k is used as the visual tracker. Manually Annotated Text Model (MATM), which uses manually annotated text in the dataset as input for semantic tracking. Due to the lack of background descriptions generated by the RPO module, only foreground proposals P are generated. fore , and use W fore To select the tracking results, the Single Iterative Model (SRIM) uses only the foreground and background text descriptions generated by the first iteration of the RPO module for tracking.

[0113] First, MATM achieved similar performance to VTM on all three datasets, indicating that manually annotated text is suboptimal for performing visual-language tracking. SRIM was then compared with VTM or MATM, and moderate performance improvements were observed on the three datasets. These improvements verify the effectiveness of the semantic tracking module and foreground verification module proposed in this embodiment. The text description obtained directly from MLLM may be inaccurate and noisy. However, the method of this embodiment effectively alleviates this noise by utilizing the semantic matching ability of GVLM. In addition, when the method of this embodiment is compared with VTM and MATM, it is found that the method of this embodiment has significant performance improvements on all datasets. These improvements indicate that the proposed feedback prompt optimization process based on a large language model can utilize the rich knowledge of MLLM to generate accurate target descriptions, and the generated descriptions are even better than the manually annotated descriptions provided by the dataset. Therefore, the performance gain of the method of this embodiment relative to SRIM verifies the effectiveness of iterative optimization of the feedback prompt optimization process based on a large language model. The comparison results are as follows:

[0114]

Claims

1. A visual target tracking method based on a multimodal large language model, characterized by comprising the following specific steps: Step 1: Input the template image I into the feedback prompt optimization process based on the large language model to obtain the foreground description T fore and background description T back ; Step 2: Transform the t-th frame image x t and prospect description T fore Input into the language-based visual localization model to obtain a series of corresponding foreground candidate boxes Then take the t-th frame image x t and background description T back Input into the language-based visual localization model to obtain a series of corresponding background candidate boxes Step 3: Foreground candidate box and background candidate boxes During the input foreground verification process, the image tracking target result of the tth frame is obtained in: The step 1 specifically includes: 1.

1. Passing template image I to multimodal large language model F MLLM In the above example, we generate a description of the image content T init ;in represents a set of real numbers, H represents the height of the image, and W represents the width of the image; T init =F MLLM (I) F MLLM Represents a multimodal large language model; 1.

2. Mark the tracked target with a bounding box in image I and create a new image input I m , will I m and T init Pass it into the multimodal large language model to generate a prospect description and background description F MLLM Represents a multimodal large language model; Among them, T init Represents the description of the template image content, I m Indicates the image after the tracking target is selected; 1.

3. Describe the prospects The template image I is passed into the language-based visual positioning model to obtain the target candidate box P z , and the alignment score S z , Where N represents the number of candidate boxes, M represents The length of i represents the number of iterations, and the value range of i is [0-N iter ]; P z ,S z =F GVLM (I,T fore ) 1.4 Classified as positive descriptive words and negative descriptors The classification method is as follows: Among them, θ2 represents the alignment score threshold, θ1 and θ3 are the intersection-over-union ratio calculation thresholds for the foreground and background respectively; It indicates the intersection over union (IoU) calculation between the nth candidate box and the real target area. z ,G) is calculated as follows: Among them, Area_of_Overlap(P z ,G) represents the candidate box P z and the intersection area between the real target area G, Area_of_Union(P $ ,G) represents the candidate box P $ and the union area between the true target area G; According to S z and IoU z (P z ,G) comparison: When S z ≤IoU z (P z ,G) Positive Description When S z >IoU z (P z ,G) Positive Description Thus, the prospect description Positive description in and negative description 1.

5. IoU z (P z ,G) is compared with the set threshold ∈ to determine: If IoU(P max ,G)<∈,P max represents the candidate box with the largest evaluation score, then and After being passed into the multimodal large prediction model for optimization, a new prospect description is obtained Background description reserved And continue to repeat steps 1.3, 1.4 and 1.5; If IoU(Pmax,G)≥∈, then the current accurate foreground description T fore and background description T back Output; if i>N iter , then the current prospect description and background description As foreground description T fore and background description T back Output; The step 3 specifically includes: 3.

1. Calculate foreground candidate boxes Prospect score The process is as follows: Where max() means taking the maximum value of the parameters in the brackets, z means the target template in the first frame, is in the bounding box The cropped image inside, f is a convolutional neural network used to extract image features, f f represents transposed convolution; ||f||2 represents the modulus length of the f vector; 3.

2. Calculate background candidate boxes Background score As shown below: in, express and The intersection-and-union ratio between ; it is calculated as follows: Among them, A and B represent two regions, A∩B is their intersection, and A∪B is their union; 3.

3. Combined foreground score W fore and background score W back To determine the candidate box confidence W all =[s 1 ,s 2 ,…,s N ], the specific calculation method is as follows: 3.

4. Score s i The highest foreground candidate box The image x tracked at frame t t The output is the tracking target result

Images