A general cross-modal reference representation method based on position index

Abstract

The application belongs to the technical field of multi-modal large models, and particularly relates to a general cross-modal reference representation method based on position indexes. The specific process is as follows: a cross-modal reference mapper and a position set encoder are set, and an attention reweighting module is introduced into a large language model; the cross-modal reference mapper is set to convert a reference area in an image or text into an index set I ref of the reference area in a token sequence; the position set encoder is set to encode the position index set I ref into a vector representation x ref ; the vector representation x ref is combined with a visual token vector encoded by a visual encoder and a text token vector mapped by a text token embedding to serve as an input sequence X of the large language model; the large language model performs feature interaction on the input sequence X to output a cross-modal reference representation; and in the feature interaction process, the attention reweighting module is used to dynamically weight an attention weight in the large language model.

Description

Technical Field

[0001] This invention belongs to the field of multimodal large model technology, specifically relating to a general cross-modal algebraic representation method based on position index. Background Technology

[0002] In recent years, Multimodal Large Language Models (MLLMs) have demonstrated outstanding cross-modal understanding and generative capabilities in tasks such as visual question answering, image and text generation, and multi-turn dialogue. These models generally employ structured modal encoders (such as CNN, ViT, and Audio Transformer) to uniformly map inputs from different modalities into vector sequences, and further fuse cross-modal information through the Transformer module to achieve semantic alignment and generative reasoning.

[0003] However, current mainstream MLLMs still face significant challenges in handling cross-modal local region referencing problems, mainly in the following two aspects:

[0004] 1. Limited modality and fixed referential form: Most existing works focus on region referential modeling of image modalities and only support fixed forms such as bounding boxes or masks. They cannot be generalized to richer modalities (such as text, video, tables, 3D scenes) and more flexible referential methods (such as arbitrary shape regions, click points, dynamic graffiti, etc.).

[0005] 2. Complex structure and high computational cost: To achieve region understanding, many methods need to introduce specific region encoders (such as region proposal networks, salient region detectors, etc.), resulting in complex model structures, high resource consumption, and difficulty in seamlessly integrating with general large language model architectures.

[0006] For example, Groma (ECCV 2024) proposed dividing the visual input into multiple candidate regions by introducing Region Encoder and Region Proposer modules. Each region computes an independent semantic representation and introduces surrogate tags to refer to the corresponding region. However, this method relies on accurate region proposal and independent region encoding, making it difficult to transfer to other modalities.

[0007] Alpha-CLIP (CVPR 2024) adds an alpha channel to the traditional CLIP model, drawing a mask region within this channel. The input values ​​for this channel range from 0 to 1, where 1 represents foreground and 0 represents background. This design allows the model to learn how to focus on specific regions during training. However, this mask-based referencing method is fixed to pixel-level image regions and cannot support more general cross-modal referencing needs such as video temporal ranges, table cell regions, or text sentence fragments.

[0008] While MLLMs such as LLaVA and MiniGPT-4 possess image-text alignment capabilities, they do not explicitly model the indexing and pointing relationships between regions when handling user references to local areas in an image (such as "this dog" or "its nose"). Instead, they rely solely on linguistic contextual reasoning, leading to ambiguity in referential understanding.

[0009] The above methods attempt to achieve local region referencing by encoding semantic vectors of specific regions of an image.

[0010] Although existing large multimodal models (MLLMs) have achieved significant results in tasks such as text and image understanding and cross-modal question answering, they still have the following significant shortcomings when dealing with cross-modal local region referencing tasks:

[0011] (1) It depends on modality-specific structures and has poor versatility.

[0012] Existing methods often rely on region coding structures specific to a particular modality (such as Groma using bounding box proposals and Alpha-CLIP using alpha mask channels), making it difficult to transfer to other modalities (such as text, video, 3D, etc.) and more complex referential forms (such as dynamic regions, irregular shapes, etc.).

[0013] There is a lack of a unified mechanism for representing modalities. Currently, there is no unified mechanism that can perform referential modeling of arbitrary regions of any modality in the same way. The current referential methods heavily rely on custom modality design and lack scalability and abstract representation capabilities.

[0014] (2) Complex structure and low reasoning efficiency

[0015] The Groma method introduces additional region encoding modules, region proposers, or multi-channel inputs, which increases the complexity of the model structure and resource consumption, making it unsuitable for direct integration and deployment in large model architectures.

[0016] (3) Difficult to support repetitive references in efficient multi-turn dialogues

[0017] Current methods cannot support semantically consistent region re-references without re-encoding the original modal input, hindering continuous understanding and efficient reasoning of region content in context during multi-turn dialogues. Summary of the Invention

[0018] In view of this, the present invention provides a general cross-modal reference representation method based on position index, which can realize cross-modal reference representation of local regions of any modality.

[0019] The technical solution for implementing the present invention is as follows:

[0020] Firstly, this invention provides a general cross-modal algebraic abbreviation representation method based on position indexing, the specific process of which is as follows:

[0021] We set up a cross-modal reference mapper and a location set encoder, and introduced an attention reweighting module into the large language model;

[0022] A cross-modal referential mapper is configured to convert referential regions in an image or text into a set of indices I in a token sequence. ref ;

[0023] The location set encoder is used to set the location index set I ref Encode as a vector representation x ref The visual token vector encoded by the visual encoder and the text token vector mapped by the text token embedding are combined and used as the input sequence X of the large language model.

[0024] The large language model performs feature interaction on the input sequence X and outputs a cross-modal reference representation; during the feature interaction process, the attention reweighting module is used to dynamically weight the assistant force weights in the large language model.

[0025] Optionally, the output of the cross-modal reference mapper described in this invention is:

[0026] I ref =ReferenceMapper(R,T,D)

[0027] Where R represents various forms of reference, T represents the modality-specific tokenizer used by the current multimodal large language model, and D represents multimodal content.

[0028] Optionally, the location set encoder of the present invention will use the location index set I ref Encode as a vector representation x ref The specific process is as follows: Location set I ref Index p of each position i Each token in the referred region is encoded using the ψ function, then summed to aggregate the semantics of all tokens in the region, and then... The function decodes the referential token to map it into the representation space of the text token.

[0029] Optionally, the dynamic reweighting of attention weights in the large language model described in this invention is as follows:

[0030]

[0031] Q = W Q X; K = W K X; V = W VX

[0032] Among them, W Q W K and W V Let d be the weight matrix. k It's the dimension of the Key, Mask causal It is the visible area mask. ref This is the visible area mask set for the referencing token; the non-referencing area is -∞.

[0033] Optionally, the Mask described in this invention ref for

[0034]

[0035] Secondly, the present invention provides a general cross-modal reference representation device based on position indexing, comprising a large language model, a cross-modal reference mapper, and a position set encoder.

[0036] A cross-modal referential mapper for converting referential regions in an image or text into a set of indices I in a token sequence. ref ;

[0037] A location set encoder is used to convert the location index set I ref Encode as a vector representation x ref The visual token vector encoded by the visual encoder and the text token vector mapped by the text token embedding are combined and used as the input sequence X of the large language model.

[0038] An attention reweighting module is introduced into the large language model to perform feature interaction on the input sequence X and output cross-modal reference representation; during the feature interaction process, the attention reweighting module is used to dynamically weight the assistant force weights in the large language model. Attached Figure Description

[0039] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 This is a flowchart illustrating the framework of the general cross-modal abbreviation representation method of the present invention.

[0041] Figure 2 This is a schematic diagram illustrating the dynamic reweighting of attention weights in the large language model of this invention. Detailed Implementation

[0042] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0043] It should be noted that, in the absence of conflict, the following embodiments and features can be combined with each other; and, based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0044] It should be noted that various aspects of embodiments within the scope of the appended claims are described below. It will be apparent that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this disclosure, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice the method. Additionally, this device and / or method can be implemented using structures and / or functionalities other than one or more of the aspects set forth herein.

[0045] like Figure 1 As shown, in order to realize the reference representation of local regions of any modality, this embodiment presents a general cross-modal reference representation method based on position index. The specific process is as follows: set up a cross-modal reference mapper and a position set encoder, and introduce an attention reweighting module in the large language model.

[0046] The following section uses the qwen2.5-vl multimodal large model as an example to explain in detail the introduced cross-modal referential mapper, location set encoder, and attention reweighting module.

[0047] Cross-modal reference mapper

[0048] This component is used to convert various forms of referential representations of any region in any modality (such as bounding boxes, masks, doodles, etc. in images, and mentions in text) into a set of their indices in a token sequence. The implementation is as follows:

[0049] For the image modality, assuming the model uses the ViT encoder to divide the image into 14x14 patches, and then performs 2x2 pooling, a total of 49 image tokens are obtained, arranged as the sequence [v1, v2, ..., v 49 When a user marks a bounding box region on the original image, the patch regions it contains can be calculated, and the corresponding token index set I can be deduced from this. ref .

[0050] For text modalities, the mention is referred to by a special tag. <mention> ...< / mention> The tokenizer can directly return its corresponding token index set I. ref .

[0051] Therefore, the output of the cross-modal referencing mapper is: I ref =ReferenceMapper(R,T,D), where R is various forms of reference, T is the modality-specific Tokenizer used by the current multimodal large model, and D is the multimodal content.

[0052] Reference Encoder (Location Set Encoder)

[0053] This module will use the above token index set I ref Encoded as a vector representation, used to replace the embedding that refers to the token (such as <|reference_pad|>).

[0054] For the location index set I ref Each position index p in i Using three-dimensional position encoding p i =(t i ,h i ,w i ) as its location feature.

[0055] Since the token index set has permutation invariance, meaning that the reference to a region is not affected by the order of the indices in the index set, the DeepSets theory is used to design the following structure:

[0056]

[0057] Let's take the Qwen2.5-VL model, which supports text, images, and videos, as an example. Where p... i Represents the set of locations I ref The three-dimensional positional encoding features of each location in the matrix; ψ, This is an MLP network with non-linear activation. Each token in the referred region is encoded using a ψ function, then summed to aggregate the semantics of all tokens in the region, and then... The function decodes the referential token to map it into the representation space of the text token.

[0058] At this point, the multimodal input sequence can be represented as:

[0059]

[0060] in, This represents a visual token vector encoded by a visual encoder (such as ViT). This represents a text token vector that has been mapped through text token embedding.

[0061] Reference-aware attention reweighting mechanism

[0062] The aforementioned multimodal input sequence can be fed into a large language model for feature interaction, thereby outputting the answer to the question, as shown in the figure above. This process can be formally represented as follows:

[0063] X output =LLM(X)

[0064] Here, LLM stands for Large Language Model. The core of current mainstream Large Language Models (LLMs) is typically built upon a Transformer architecture, and the core mechanism of the Transformer is self-attention, mathematically expressed as:

[0065]

[0066] Q = W Q X;

[0067] K = W K X;

[0068] V = W V X;

[0069] Where Q (Query), K (Key), and V (Value) are matrices obtained by linear transformation of the input sequence. k This is the dimension of the key, used to scale the dot product and prevent gradient vanishing. W Q W K and W V These are weight matrices, which are used to perform linear transformations on the input X during the attention mechanism's computation. These matrices map the input X to three different vector spaces: query (Q), key (K), and value (V), respectively, for subsequent calculation of the attention score.

[0070] Each layer of an LLM (such as a Transformer layer) can be represented as the following composite function:

[0071] LLM(X)=LayerNorm(FFN(LayerNorm(X+Attention(X))))

[0072] Wherein: FFN is a feedforward network (usually a two-layer MLP). LayerNorm is layer normalization.

[0073] To further constrain the attention of the referential token to the local semantics of its referential region, the attention weights in LLM are dynamically reweighted. The specific improved formula is as follows:

[0074]

[0075] Among them: Mask ref This is a visible mask set for the token being referenced; the non-representational area is -∞. causal It is the visible area mask of the original causal language model (each token can only see the tokens that appear in front of it), with the elements on the diagonal and below being 0, and the elements above the diagonal being -∞.

[0076] By reweighting the attention weights using a referential awareness approach, region referential constraints can be effectively fused without modifying the model parameters. Figure 2 As shown.

[0077]

[0078] Masl ref Given an n×n matrix, where n is the length of the input sequence, and the value of the i-th row and j-th column of the matrix is... It can be calculated using the formula above.

[0079] Compared with the prior art, the present invention has the following advantages:

[0080] 1. High versatility (modal independence + referential form independence): It encompasses cross-modal versatility and cross-referential form extensibility. For region objects of different modalities (text, images, videos, audio, etc.), they will be modeled as a set of index positions in a token sequence. This referential form is modal independent, therefore theoretically it can be extended to data of any modality supported by a large multimodal model. For different referential forms, they are mapped to a set of index positions through a modality-specific referential mapper. This step is rule-based; when a new referential form is introduced, a corresponding mapping rule can be added to enable the large multimodal model to support that referential form.

[0081] 2. Strong architectural compatibility and high computational efficiency. Compared to Groma / Alpha-CLIP, this method does not modify the basic model architecture, only introducing lightweight components: the token mapper only needs to perform simple token index mapping (with almost no computational burden); the ReferenceEncoder is a DeepSets structure constructed from MLP, with low computational complexity and support for batch operations; Attention reweighting can be directly implemented in the Transformer layer's attention mask without changing the model structure; therefore, this method is easy to integrate into various state-of-the-art multimodal models (such as Qwen-VL, MiniGPT-v2, Intern-VL, etc.); it can be flexibly plugged in and out, and does not depend on specific perception modules (such as region proposers); apart from introducing a very lightweight token encoder, no other parameters are introduced. The token mapper and attention reweighting add almost no computational cost to the model. Compared to Groma's method, it has higher computational efficiency and a more streamlined model structure.

[0082] 3. Supports multi-turn dialogue and KV-Cache mechanism to improve the reasoning efficiency of contextual reference understanding. Groma / Alpha-CLIP requires re-encoding the image region for each region reference and does not support cache reuse; while in the method of this invention, all modalities are tokenized, so the reference region is already in the cached KV, and can be directly referenced by position index + mask, which greatly improves reasoning efficiency;

[0083] Therefore, higher reasoning efficiency can be achieved in multi-turn text-image dialogues and multi-stage reasoning.

[0084] In summary, the above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A general cross-modal algebraic algebraic representation method based on position indexing, characterized in that, The specific process is as follows: We set up a cross-modal reference mapper and a location set encoder, and introduced an attention reweighting module into the large language model; A cross-modal referential mapper is configured to convert referential regions in an image or text into a set of their indices in a token sequence. ; The location set encoder is used to set the location index set. Encoded as a vector representation The visual token vector encoded by the visual encoder and the text token vector mapped by the text token embedding are combined as the input sequence of the large language model. ; Large language models for input sequences Perform feature interaction and output cross-modal reference representation; during the feature interaction process, use the attention reweighting module to dynamically weight the assistant force weights in the large language model; The dynamic reweighting of attention weights in the large language model is as follows: in, , and This is the weight matrix. It is the dimension of the key. It is the visible area mask. This is the visible area mask set for the token being referenced; the non-representational area is -∞. The for: 。 2. The general cross-modal reference representation method based on position indexing according to claim 1, characterized in that, The output of the cross-modal referencing mapper is: in, For various forms of reference, A modality-specific tokenizer used by current multimodal large language models. This is multimodal content.

3. The general cross-modal reference representation method based on position indexing according to claim 1 or 2, characterized in that, The location set encoder will output the location index set. Encoded as a vector representation The specific process is as follows: location set Index of each position Each token in the referred area is used The function encodes the representation, then sums them to aggregate the semantics of all tokens in the region, and then uses... The function decodes the referential token to map it into the representation space of the text token.

4. A universal cross-modal reference representation device based on position indexing, characterized in that, This includes large language models, cross-modal referential mappers, and location set encoders; A cross-modal referential mapper for converting referential regions in an image or text into a set of their indices in a token sequence. ; Location set encoder, used to convert location index sets Encoded as a vector representation The visual token vector encoded by the visual encoder and the text token vector mapped by the text token embedding are combined as the input sequence of the large language model. ; In large language models, an attention reweighting module is introduced to reweight the input sequence. Perform feature interaction and output cross-modal reference representation; during the feature interaction process, use the attention reweighting module to dynamically weight the assistant force weights in the large language model; The dynamic reweighting of attention weights in the large language model is as follows: in, , and This is the weight matrix. It is the dimension of the key. It is the visible area mask. This is the visible area mask set for the token being referenced; the non-representational area is -∞. The for: 。 5. The universal cross-modal reference representation device based on position indexing according to claim 4, characterized in that, The output of the cross-modal referencing mapper is: in, For various forms of reference, A modality-specific tokenizer used by current multimodal large language models. This is multimodal content.

6. The universal cross-modal reference representation device based on position indexing according to claim 4 or 5, characterized in that, The location set encoder will output the location index set. Encoded as a vector representation The specific process is as follows: location set Index of each position Each token in the referred area is used The function encodes the representation, then sums them to aggregate the semantics of all tokens in the region, and then uses... The function decodes the referential token to map it into the representation space of the text token.

Images