A visual cue based multi-modal instance-level understanding method and system

Abstract

This invention relates to the field of computer vision, specifically disclosing a multimodal instance-level understanding method and system based on visual cues. The method uses an interactive instance segmentation model and a video instance tracking model to perform spatiotemporal segmentation of specific instances and generate visual cues. Subsequently, a visual encoder encodes the video with visual cues, and a cross-modal connection module maps the visual representations to a multimodal representation space shared with language to obtain visual features. A text segmenter processes the input text to obtain corresponding text features. Finally, a large language model is used to uniformly model the visual and linguistic inputs to obtain fine-grained descriptions or question answers about specific instances. Compared with existing technologies, this invention achieves accurate localization and tracking of specific instances in the spatiotemporal dimension, improving the instance-level fine-grained understanding capability of video multimodal models.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and in particular to a multimodal instance-level fine-grained understanding method and system based on visual cues. Background Technology

[0002] Multimodal large models, a popular research area in artificial intelligence in recent years, have been widely applied in fields such as text-based question answering, human-computer dialogue, and artistic creation, greatly boosting productivity. Representative works include LLaVA, BLIP2, and Qwen2.5VL. After exploring various technical approaches in the early stages, the current model architecture in this field has gradually converged into a ternary structure consisting of a visual encoder, a modal connector, and a large language model.

[0003] Visual encoders typically employ a visual Transformer architecture. The process involves: first, dividing the input image into two-dimensional grid blocks, and then using a multilayer perceptron to map each image block from pixel space to feature space; second, to incorporate spatial position information between image blocks, positional encoding is performed on the feature sequence, commonly using two-dimensional absolute positional encoding and two-dimensional relative positional encoding; third, the positionally encoded image feature sequence is input into a visual encoder containing multiple encoder layers, and a self-attention mechanism is used to model the spatial and semantic information of the image; finally, the output of the last layer of the visual encoder is used as the visual representation of the image.

[0004] The primary function of modal connectors is to map visual representations to a shared multimodal representation space with language representations, thereby facilitating unified understanding and modeling of multimodal information in large language models. In the BLIP2 model, a Q-Former structure is used to achieve modal alignment, extracting key features from visual representations through learnable query sequences. In contrast, the LLaV series models use a simpler multilayer perceptron structure, directly mapping visual representations to a shared multimodal representation space.

[0005] As the core recognition and inference module of a multimodal large-scale model, the large language model is responsible for uniformly processing multimodal input information from images and text, and gradually generating response text through an autoregressive approach. During the training phase, large-scale image and text data and high-quality instruction-following data are typically used for supervision. Cross-entropy loss is used to align the model's predictions with the real labels, enabling the model to understand visual content and generate text descriptions or answers that semantically match the input query.

[0006] Despite significant progress in this field, existing video multimodal models, such as LLaVA-Video and Qwen2.5VL, still lack fine-grained instance-level understanding. Model performance degrades significantly when users focus on the spatiotemporal motion of specific instances. To address this, some methods, such as GPT4ROI and Ferret, embed local visual features of specific regions into the input query to explicitly specify the instances the model should focus on. However, such methods require additional local feature extraction modules and significantly increase the length of the model's input sequence, thereby increasing the computational overhead for training and inference. Summary of the Invention

[0007] This invention addresses the problem that existing multimodal large models struggle to focus on specific instance objects of interest to users, leading to performance degradation in fine-grained spatiotemporal understanding. This invention provides a method and apparatus for multimodal instance-level fine-grained understanding based on visual cues.

[0008] The objective of this invention can be achieved through the following technical solutions:

[0009] As a first aspect of the present invention, a multimodal instance-level understanding method based on visual cues is provided, comprising the following steps:

[0010] Obtain the video image to be understood and the query text input;

[0011] For the video image to be understood, the target instance is segmented by mask using an interactive instance segmentation model, and visual cues are generated on the original video image.

[0012] The video with visual cues and the corresponding query text are input into a multimodal fine-grained understander, which includes a visual encoder, a cross-modal connector, a word segmenter, and an autoregressive large language model. The training loss includes supervised fine-tuning loss, frame-level instance description loss, and video-level instance consistency loss.

[0013] A visual encoder encodes video images containing visual cues to obtain a sequence of visual lexical units; a cross-modal connector maps the visual representations to a multimodal space to obtain the visual features corresponding to the visual lexical units.

[0014] The input text is processed by a text segmenter, and the corresponding text features are obtained by querying the word embedding table.

[0015] Visual and textual features are input into an autoregressive large language model to generate fine-grained question answers or textual descriptions for the target instance.

[0016] As a preferred technical solution, the process of generating the visual cues is as follows:

[0017] The interactive instance segmentation module is used to segment the first frame of the video to obtain the mask of the target instance;

[0018] The instance mask of the first frame is used as a cue for the video instance tracking model to obtain the spatiotemporal trajectory mask of the target instance;

[0019] Based on the spatiotemporal trajectory mask of the target instance, visual cues of the instance are added to the corresponding area in the video frame to ensure the encoding consistency of the same instance across different frames.

[0020] As a preferred technical solution, the cross-modal connection module adopts a multilayer perceptron, which maps visual lexical units to a multimodal space through nonlinear transformation to obtain the visual features corresponding to the visual lexical units.

[0021] As a preferred technical solution, the text segmenter performs word segmentation on the input text to obtain a word index of the corresponding word element; the word index is then used to query the word embedding table to obtain the corresponding text features.

[0022] As a preferred technical solution, the method concatenates visual features and linguistic features, and inputs the concatenated sequence into a large language model for unified modeling and prediction; the large language model, in the manner of predicting the next word, autoregressively outputs the response content for the input query word by word, generating fine-grained question answers or text descriptions about specific instances.

[0023] As a preferred technical solution, the multimodal fine-grained understander freezes the shallow parameters of the visual encoder during training and only updates the deep parameters of the visual encoder, the cross-modal connectors, and the parameters of the large language model.

[0024] As a preferred technical solution, the loss function used in the training of the multimodal fine-grained understander is a weighted sum of supervised fine-tuning loss, frame-level instance description loss, and video-level instance consistency loss;

[0025] The supervised fine-tuning loss is based on instruction-following data. The input is a video and the corresponding question text, and the output is the predicted answer text. The difference between the model's prediction result and the standard answer is calculated using the cross-entropy loss function.

[0026] The frame-level instance description loss is trained based on frame-level instance description data. The input is a single-frame image with visual cues, and the output is a text description for a specific instance. The difference between the prediction result and the standard answer is calculated using the cross-entropy loss function.

[0027] The video-level instance consistency loss is trained based on video-level instance description data. The input consists of multiple video frames containing visual cues, and the output consists of text descriptions of target instances in each frame. A text encoder is used to obtain the text features of the target instance descriptions in each frame, and then the mean squared error loss function is used to constrain the consistency of the text descriptions of the same instance in different frames.

[0028] As a preferred technical solution, the multimodal fine-grained understander collects a multi-object tracking dataset and a video instance segmentation dataset during training, constructs instance visual cues based on instance mask annotations contained in the dataset, and uses a multimodal large model to perform multi-level fine-grained annotation on the video to generate a training dataset.

[0029] As a second aspect of the present invention, a visually cued multimodal understanding system is provided, the system executing the visually cued multimodal understanding method described above, including:

[0030] The visual cue module is used to process the original video frames, segment out specific instance objects in the first frame, track the spatiotemporal trajectory of the instance, and add visual cues in the corresponding area.

[0031] The visual encoding module is used to process video frames with visual cues to obtain visual lexical sequences;

[0032] The cross-modal connectivity module uses a multilayer perceptron for nonlinear transformation to obtain the visual features corresponding to visual lexical units;

[0033] The text encoding module is used to process the input text and obtain the corresponding text features;

[0034] The large language model module takes visual and textual features as input, and generates the final response output in an autoregressive manner through unified modeling across visual and language modalities.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] 1) By introducing explicit visual cues to inject instance binding information, this invention achieves precise positioning and tracking of specific instances in the spatiotemporal dimension. Without the need for additional complex local feature extraction modules, it significantly improves the instance-level fine-grained understanding capability of video multimodal models, effectively reduces the computational overhead of training and inference, and enhances the fusion effect of visual and linguistic features, thus achieving instance-level multimodal fine-grained understanding.

[0037] 2) In terms of model training, this invention designs frame-level instance description loss and video-level instance consistency loss: the former ensures that the model can accurately describe instance objects in a single frame, while the latter guarantees the semantic consistency of the same instance across different frames. By introducing visual cues to explicitly point to the target instance and combining two novel instance alignment losses, this invention effectively improves the model's performance in fine-grained instance-level understanding. Attached Figure Description

[0038] Figure 1 This is a flowchart of a multimodal instance-level fine-grained understanding method based on visual cues according to the present invention.

[0039] Figure 2 This is a schematic diagram of the multimodal fine-grained understanding model based on visual cues proposed in this invention.

[0040] Figure 3 This is a schematic diagram of the hierarchical fine-grained video annotation module proposed in this invention. Detailed Implementation

[0041] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0042] Example 1

[0043] This invention proposes a multimodal instance-level fine-grained understanding method based on visual cues. First, an interactive instance segmentation module generates a mask for the target instance, and a visual cue for the instance is constructed based on this mask. Then, a video containing the visual cues and the corresponding query text are input into a multimodal fine-grained understander, which includes a visual encoder, a cross-modal connector, a segmentation word segmenter, and an autoregressive large language model. The visual encoder encodes the image or video containing the visual cues, and the cross-modal connector maps the visual representation to the multimodal space to obtain the visual features corresponding to the visual tokens. Subsequently, the autoregressive large language model is used to uniformly model the input visual and linguistic tags, thereby outputting a fine-grained description or answer for a specific instance, achieving instance-level multimodal fine-grained understanding.

[0044] In terms of model training, this invention designs frame-level instance description loss and video-level instance consistency loss: the former ensures that the model can accurately describe instance objects in a single frame, while the latter guarantees the semantic consistency of the same instance across different frames. By introducing visual cues to explicitly point to the target instance and combining two novel instance alignment losses, this invention effectively improves the model's performance in fine-grained instance-level understanding.

[0045] like Figure 1 ,2 As shown, the multimodal instance-level fine-grained understanding method based on visual cues proposed in this invention specifically includes the following steps:

[0046] S1, such as Figure 3 As shown, firstly, a multi-object tracking dataset and a video instance segmentation dataset are collected as initial data. Instance-level visual cues are then drawn based on instance-level mask annotations contained in the data. These visual cues effectively improve the fine-grained annotation accuracy of the multimodal large model. Subsequently, a temporal window formed by two adjacent keyframes is used as input to the multimodal large model, and specific task cues are designed for the model to obtain multi-level fine-grained annotations of the video content. Specifically, this includes: 1) instance-level description, detailing the attributes, actions, and states of a single instance in a specific frame; 2) frame-level description, detailing the spatial relationships and action interactions between instances in a single frame; 3) temporal change description, describing the dynamic changes of instances between adjacent keyframes; and 4) video-level description, detailing the dynamic changes of instances throughout the entire video in both temporal and spatial dimensions.

[0047] S2. Use the interactive instance segmentation module to obtain the mask of a specific instance, and generate visual cues in the video based on the mask;

[0048] S3. Use a visual encoder to process video frames containing visual cues to obtain visual lexical units, and use a cross-modal connector to map the visual representations to a multimodal space to convert them into visual features.

[0049] S4. Use a text segmenter to process the input text and obtain the corresponding text features by querying the word embedding table.

[0050] S5. Use a large language model to uniformly model visual and text features, and generate fine-grained question answers or text descriptions for specific instance objects through autoregressive next lexical prediction.

[0051] The multimodal fine-grained understander is trained using the training data obtained in step S1 to improve its fine-grained understanding capabilities, resulting in trained model weights. Training losses include supervised fine-tuning loss, frame-level instance description loss, and video-level instance consistency loss. Finally, the trained model weights are loaded into the multimodal understanding network to instantiate the multimodal fine-grained understander; an interactive instance segmentation model is used to perform masked segmentation of specified instances, and visual cues are added to the video; the video with visual cues and the corresponding query text are input into the multimodal fine-grained understander to obtain the corresponding answer or response.

[0052] Furthermore, in step S2, the algorithm for constructing visual cues includes the following steps:

[0053] S2-1. Use the interactive instance segmentation module to segment the first frame image to obtain the mask of a specific instance;

[0054] S2-2. Use the first frame instance mask as a prompt from the video instance tracking module to obtain the spatiotemporal trajectory mask of a specific instance;

[0055] S2-3. Based on the spatiotemporal trajectory mask of the specific instance, construct an instance visual cue to ensure the encoding consistency of the unified instance across different frames.

[0056] Furthermore, in step S3, the algorithm for obtaining visual features includes the following steps:

[0057] S3-1. Process the image input using a visual encoder to obtain visual terms. For visual feature extraction, let... For the first 1 video frame, with a resolution of 1 Break it down into There are 1 image patch, where each image patch is 1 , Represents the size of image patch pairs. This represents the number of channels in the image. After obtaining these image patches, a visual encoder is used. To extract visual lexical units:

[0058]

[0059] (S3-2) Use cross-modal connectors to bridge visual lexical units into visual features and align them with text features in a shared feature space.

[0060] The cross-modal connectivity module employs a multilayer perceptron, which uses nonlinear transformations to map visual embeddings onto a multimodal feature space to obtain visual features.

[0061]

[0062] in, This represents a multilayer perceptron.

[0063] Furthermore, in step S4, the algorithm for obtaining text features includes the following steps:

[0064] S4-1. Use a text segmenter to segment the input text and obtain the word index of the corresponding word unit;

[0065] S4-2. Use the vocabulary index to query the word embedding table to obtain the corresponding text features. .

[0066] Furthermore, in step S5, the visual features obtained in step S3 are concatenated with the linguistic features obtained in step S4, and the concatenated sequence is input into the large language model for unified modeling and prediction. The large language model, in the manner of predicting the next word, autoregressively outputs the response content for the input query word by word to generate fine-grained question answers or text descriptions about specific instances.

[0067] For multimodal unified modeling, the extracted visual and textual features are directly input into the large language model as conditions, and the next response output is predicted one by one in an autoregressive manner.

[0068]

[0069] in, The output represents the model's response. The parameter is Large language models Indicates the length of the response output sequence; Indicates the generation of the first i Text features preceding each lexical unit; This represents the response content that has been generated before the generation of the i-th word.

[0070] Furthermore, during the training of the multimodal fine-grained understander, the shallow parameters of the visual encoder are frozen, and only the deep parameters of the visual encoder, the cross-modal connectors, and the parameters of the large language model are updated. The loss function used during model training consists of the following three parts:

[0071] Supervised fine-tuning loss: Training is performed based on instruction-following data. The input is a video and the corresponding question text, and the output is the predicted answer text. The loss is the cross-entropy between the model's predicted sequence and the standard answer reference sequence.

[0072]

[0073] in, This indicates the input video and the question text. For the first One predicted word unit, The length of the generated answer text sequence.

[0074] Frame-level instance description loss: Trained on frame-level instance description data, the input is a single-frame image with visual cues, and the output is a text description for a specific instance. The difference between the predicted result and the standard answer is calculated using the cross-entropy loss function.

[0075]

[0076] in, This refers to a single frame of an image with visual cues.

[0077] Video-level instance consistency loss: It is trained based on video-level instance description data. The input is multiple video frames containing visual cues, and the output is the text description of the target instance in each frame. The text encoder is used to obtain the text features of the instance description in each frame, and then the mean squared error (MSE) loss function is used to constrain the consistency of the text description of the same instance in different frames.

[0078] For the target instance in Description in the frame Text features are obtained through a text encoder. Then the consistency constraint loss is:

[0079]

[0080]

[0081] The final training loss is a weighted sum of the three losses:

[0082]

[0083] The frame-level instance description loss and video-level instance consistency loss designed in this invention ensure that the model can accurately describe instance objects in a single frame, while the latter guarantees the semantic consistency of the same instance across different frames. By introducing visual cues to explicitly point to the target instance and combining two novel instance alignment losses, this invention effectively improves the model's performance in fine-grained instance-level understanding.

[0084] Example 2

[0085] As another embodiment of the present invention, this embodiment provides a system for executing the multimodal instance-level fine-grained understanding method based on visual cues as described in Embodiment 1 above, the system comprising:

[0086] The visual cue module is used to process the original video frames, segment out specific instance objects in the first frame, track the spatiotemporal trajectory of the instance, and add visual cues in the corresponding area.

[0087] The visual encoding module is used to process video frames with visual cues to obtain visual lexical sequences;

[0088] The cross-modal connectivity module uses a multilayer perceptron for nonlinear transformation to obtain the visual features corresponding to visual lexical units;

[0089] The text encoding module is used to process the input text and obtain the corresponding text features;

[0090] The large language model module takes visual and textual features as input, and generates the final response output in an autoregressive manner through unified modeling across visual and language modalities.

[0091] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A multimodal instance-level understanding method based on visual cues, characterized in that the steps include... include: Obtain the video image to be understood and the query text input; For the video image to be understood, the target instance is segmented by mask using an interactive instance segmentation model, and visual cues are generated on the original video image. The video with visual cues and the corresponding query text are input into a multimodal fine-grained understander, which includes a visual encoder, a cross-modal connector, a word segmenter, and an autoregressive large language model. The training loss includes supervised fine-tuning loss, frame-level instance description loss, and video-level instance consistency loss. The supervised fine-tuning loss is based on instruction-following data. The input is a video and the corresponding question text, and the output is the predicted answer text. The difference between the model's prediction result and the standard answer is calculated using the cross-entropy loss function. The frame-level instance description loss is trained based on frame-level instance description data. The input is a single-frame image with visual cues, and the output is a text description for a specific instance. The difference between the prediction result and the standard answer is calculated using the cross-entropy loss function. The video-level instance consistency loss is trained based on video-level instance description data. The input is multiple video frames containing visual cues, and the output is the text description of the target instance in each frame. The text encoder is used to obtain the text features of the target instance description in each frame, and then the mean squared error loss function is used to constrain the consistency of the text description of the same instance in different frames. A visual encoder encodes video images containing visual cues to obtain a sequence of visual lexical units; a cross-modal connector maps the visual representations to a multimodal space to obtain the visual features corresponding to the visual lexical units. The input text is processed by a text segmenter, and the corresponding text features are obtained by querying the word embedding table. Visual and textual features are input into an autoregressive large language model to generate fine-grained question answers or textual descriptions for the target instance.

2. The method of claim 1, wherein, The process of generating the visual cues is as follows: The interactive instance segmentation module is used to segment the first frame of the video to obtain the mask of the target instance; The instance mask of the first frame is used as a cue for the video instance tracking model to obtain the spatiotemporal trajectory mask of the target instance; Based on the spatiotemporal trajectory mask of the target instance, visual cues of the instance are added to the corresponding area in the video frame to ensure the encoding consistency of the same instance across different frames.

3. The visual cue based multi-modal instance-level understanding method of claim 1, wherein, The cross-modal connector employs a multilayer perceptron, which maps visual lexical units to a multimodal space through nonlinear transformation to obtain the visual features corresponding to the visual lexical units.

4. The visual cue based multi-modal instance-level understanding method of claim 1, wherein, The text segmenter performs word segmentation on the input text to obtain the word index of the corresponding word unit; the word index is then used to query the word embedding table to obtain the corresponding text features.

5. The visual cue based multi-modal instance-level understanding method of claim 1, wherein, The method concatenates visual features with linguistic features and inputs the concatenated sequence into a large language model for unified modeling and prediction. The large language model, in an autoregressive manner, outputs response content for the input query word by word according to the next word prediction method, generating fine-grained question answers or text descriptions about specific instances.

6. The visual cue based multi-modal instance-level understanding method of claim 1, wherein, The multimodal fine-grained understander freezes the shallow parameters of the visual encoder during training and only updates the deep parameters of the visual encoder, the cross-modal connectors, and the parameters of the large language model.

7. The visual cue based multi-modal instance-level understanding method of claim 1, wherein, The multimodal fine-grained understander collects a multi-object tracking dataset and a video instance segmentation dataset during training. It constructs instance visual cues based on the instance mask annotations contained in the dataset and uses a multimodal large model to perform multi-level fine-grained annotations on the video to generate a training dataset.

8. A visual cue based multi-modal instance level understanding system, characterized in that, The system executes the multimodal instance-level understanding method based on visual cues as described in any one of claims 1-7, including: The visual cue module is used to process the original video frames, segment out specific instance objects in the first frame, track the spatiotemporal trajectory of the instance, and add visual cues in the corresponding area. The visual encoding module is used to process video frames with visual cues to obtain visual lexical sequences; The cross-modal connectivity module uses a multilayer perceptron for nonlinear transformation to obtain the visual features corresponding to visual lexical units; The text encoding module is used to process the input text and obtain the corresponding text features; The large language model module takes visual and textual features as input, and generates the final response output in an autoregressive manner through unified modeling across visual and language modalities.

Images