General cross-modal anaphora representation method based on position index

Abstract

The invention belongs to the technical field of multi-modal large models, and particularly relates to a universal cross-modal anaphora representation method based on position indexing. The method specifically comprises the following steps: setting a cross-modal anaphora mapper and a position set encoder, and introducing an attention reweighting module into a large language model; setting a cross-modal anaphora mapper for converting an anaphora region in the image or the text into an index set Iref of the anaphora region in the token sequence; a position set encoder is arranged and used for encoding a position index set Iref into a vector representation xref, and a visual token vector encoded by a visual encoder and a text token vector embedded and mapped through text Token are combined to serve as an input sequence X of the large language model; performing feature interaction on the input sequence X by the large language model, and outputting cross-modal anaphora representation; and in the feature interaction process, dynamically weighting the assistant force weight in the large language model by using an attention reweighting module.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of multimodal large models, and particularly relates to a general cross-modal reference representation method based on a position index. BACKGROUND

[0002] In recent years, multimodal large language models (MLLMs) have shown excellent cross-modal understanding and generation capabilities in tasks such as visual question answering, image-text generation, and multi-turn dialogue. Such models generally use structured modality encoders (such as CNN, ViT, Audio Transformer, etc.) to map inputs of different modalities into vector sequences, and further integrate cross-modal information through Transformer modules to achieve semantic alignment and generation reasoning.

[0003] However, current mainstream MLLMs still face great challenges in handling cross-modal local region reference problems, mainly in the following two aspects:

[0004] 1. Single modality, fixed reference form: Most existing work focuses on modeling region reference in the image modality, and only supports fixed forms such as bounding boxes or masks, which cannot be generalized to more diverse modalities (such as text, video, table, 3D scene) and more flexible reference methods (such as arbitrary shape regions, click points, dynamic scribbles, etc.).

[0005] 2. Complex structure, high computational cost: To achieve region understanding, many methods need to introduce specific region encoders (such as region proposal networks, salient region detectors, etc.), resulting in complex model structures, large resource consumption, and difficulty in seamless integration with general large language model architectures.

[0006] For example: Groma (ECCV 2024) proposes to divide the visual input into multiple candidate regions by introducing a Region Encoder and a Region Proposer module, calculate an independent semantic representation for each region, and use proxy labels to refer to the corresponding region. However, this method relies on accurate region proposals and independent region encoding, making it difficult to migrate to other modalities.

[0007] Alpha-CLIP (CVPR 2024) adds an Alpha channel to the traditional CLIP model and draws a mask region in this channel. The value range of the channel input is 0 and 1, where 1 represents the foreground and 0 represents the background. This design allows the model to learn how to focus on specific regions during training. This mask-based reference method is fixed to pixel-level image regions and cannot support more general cross-modal reference requirements such as video temporal range, table cell region, or text sentence segment.

[0008] Although MLLMs such as LLaVA and MiniGPT-4 have the ability to align images and text, when processing users' references to local areas in the image (such as "this dog" and "its nose"), they do not explicitly model the index and pointing relationship between regions, but only rely on language context reasoning, which leads to ambiguous reference understanding.

[0009] The above methods attempt to achieve reference to local areas by encoding semantic vectors of specific areas of the image.

[0010] Although existing multimodal large models (MLLMs) have achieved remarkable results in tasks such as image-text comprehension and cross-modal question answering, they still have the following significant shortcomings when dealing with cross-modal local region reference tasks:

[0011] (1) Dependence on modal-specific structures and poor versatility

[0012] Existing methods mostly rely on the region encoding structure of a specific modality (such as Groma using bounding box proposal and Alpha-CLIP using alpha mask channel), which makes it difficult to migrate to other modalities (such as text, video, 3D, etc.) and more complex reference forms (such as dynamic areas, irregular shapes, etc.).

[0013] Lack of a unified referential representation mechanism. Currently, there is no unified mechanism that can model referential representations of any region in any modality in the same way. Referential representations rely heavily on modality-specific design and lack scalability and abstract representation capabilities.

[0014] (2) Complex structure and low reasoning efficiency

[0015] The Groma method introduces additional region encoding modules, region proposers, or multi-channel inputs, which increases the complexity of the model structure and resource consumption, and is not suitable for direct integration and deployment in large model architectures.

[0016] (3) It is difficult to support repeated references in efficient multi-round conversations

[0017] Current methods cannot support semantically consistent region repeated references without re-encoding the original modal input, hindering the continuous understanding and efficient reasoning of regional content in context during multi-round dialogues. Summary of the Invention

[0018] In view of this, the present invention provides a general cross-modal referential representation method based on position index, which can realize the referential cross-modal representation of local areas of any modality.

[0019] The technical solutions for implementing the present invention are as follows:

[0020] In a first aspect, the present invention provides a general cross-modal referential representation method based on position indexing, the specific process of which is as follows:

[0021] Set up a cross-modal reference mapper and position set encoder, and introduce an attention reweighting module in the large language model;

[0022] Set up a cross-modal referent mapper to convert the referent region in the image or text into its index set in the token sequence I ref ;

[0023] Set the position set encoder to convert the position index set I ref Encoded as vector representation x ref , and combine the visual token vector encoded by the visual encoder and the text token vector mapped by the text token embedding as the input sequence X of the large language model;

[0024] The large language model performs feature interaction on the input sequence X and outputs a cross-modal referential representation; during the feature interaction process, the attention reweighting module is used to dynamically weight the auxiliary force weights in the large language model.

[0025] Optionally, the output of the cross-modal referential mapper of the present invention is:

[0026] I ref =ReferenceMapper(R,T,D)

[0027] Among them, R is various forms of reference, T is the modality-specific Tokenizer used by the current multimodal large language model, and D is the multimodal content.

[0028] Optionally, the position set encoder of the present invention converts the position index set I ref Encoded as vector representation x ref The specific process is: Position set I ref Each position index p in i Each token in the region is encoded using the ψ function, and then summed to aggregate the semantics of all tokens in the region, and then used The function decodes the reference token to map it into the representation space of the text token.

[0029] Optionally, the present invention dynamically reweights the attention weights in the large language model as follows:

[0030]

[0031] Q=W Q X;K=W K X; V = W VX

[0032] wherein, W Q , W K and W V are weight matrices, d k is the dimension of Key, Mask causal is the visible region mask, Mask ref is the visible region mask set for the token, and the non-reference region is -∞.

[0033] Optionally, the Mask ref of the present application is

[0034]

[0035] In a second aspect, the present application provides a general cross-modal reference representation device based on position index, which comprises a large language model, a cross-modal reference mapper and a position set encoder.

[0036] The cross-modal reference mapper is used to convert the reference region in the image or text into its index set I ref in the token sequence.

[0037] The position set encoder is used to encode the position index set I ref into vector representation x ref , and jointly encode the visual token vector encoded by the visual encoder and the text token vector mapped by the text token embedding as the input sequence X of the large language model.

[0038] The attention reweighting module is introduced into the large language model to perform feature interaction on the input sequence X, and output the cross-modal reference representation; in the feature interaction process, the attention weight in the large language model is dynamically weighted by the attention reweighting module. BRIEF DESCRIPTION OF DRAWINGS

[0039] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0040] Figure 1 It is the general cross-modal reference representation method framework flowchart of the present application;

[0041] Figure 2 It is the schematic diagram of dynamically reweighting the attention weight in the large language model of the present application. DETAILED DESCRIPTION

[0042] The embodiments of the present application will be described in detail below with reference to the drawings.

[0043] It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict; and all other embodiments obtained by those skilled in the art based on the embodiments in the present disclosure without creative labor are within the scope of protection of the present disclosure.

[0044] It should be noted that various aspects of the embodiments described below are within the scope of the appended claims. It should be apparent that the aspects described herein can be embodied in a wide variety of forms and that any specific structure and / or function described herein is merely illustrative. Based on the teachings herein one skilled in the art should appreciate that an aspect described herein can be implemented independently of any other aspects and that an aspect can be implemented both as any number of devices and / or as any number of methods. For example, an apparatus can be implemented using any number of the aspects described herein. In addition, an apparatus can be implemented using any number of other structures and / or functionalities not expressly described herein.

[0045] As shown in Figure 1 To realize the representation of the reference to the local area of any modality, the embodiment of the present application is a general cross-modal reference representation method based on position index, and the specific process is as follows: setting a cross-modal reference mapper and a position set encoder, and introducing an attention reweighting module in a large language model;

[0046] Next, taking qwen2.5-vl multi-modal large model as an example, the cross-modal reference mapper, the position set encoder and the attention reweighting module introduced are described in detail.

[0047] Cross-modal reference mapper (Reference Mapper)

[0048] This component is used to convert the reference representation of any area of any modality (such as bbox, mask, graffiti, etc. in image, mention in text) into its index set in token sequence. The implementation is as follows:

[0049] For the image modality, it is assumed that the model uses a ViT encoder to divide the image into 14x14 patches, and then performs 2x2 pooling, resulting in a total of 49 image tokens arranged in the sequence [v1, v2,..., v 49 ]. When the user labels a bounding box area on the original image, the patch area it contains can be calculated, and the corresponding token index set I ref is deduced accordingly.

[0050] For text modality, the mention is marked by special tokens <mention> ...< / mention> The tokenizer can directly return its corresponding token index set I ref .

[0051] Therefore, the output of the cross-modal reference mapper is: I ref = ReferenceMapper(R, T, D), where R is the reference in various forms, T is the modality-specific tokenizer used by the current multi-modal large model, and D is the multi-modal content.

[0052] Position set encoder (Reference Encoder)

[0053] This module encodes the token index set I ref mentioned above into a vector representation, which is used to replace the embedding of the reference token (such as <|reference_pad|>).

[0054] For each position index p ref in the position index set I i , use the three-dimensional position encoding p i = (t i , h i , w i ) as its position feature.

[0055] Since the token index set has permutation invariance, that is, the reference to the region will not be affected by the order of the index in the index set, the DeepSets theory is used to design the following structure:

[0056]

[0057] Take the Qwen2.5-VL model supporting text, picture, and video as an example. Among them, p i represents the three-dimensional position encoding feature of each position in the position set I ref ; ψ, is an MLP network with nonlinear activation. Each token in the mentioned region is encoded and represented using the ψ function, and then summed to converge the semantics of all tokens in the region, and then decoded using the function to map to the representation space of the text token.

[0058] At this time, the multi-modal input sequence can be represented as

[0059]

[0060] where, visual token vectors encoded by a visual encoder (e.g., ViT, etc.), text token vectors mapped from text tokens.

[0061] Reference-aware Attention Reweighting

[0062] The above multi-modal input sequence can be passed into a large language model for feature interaction, thereby outputting the answer to the question, as shown in the above figure. This process can be formally represented as

[0063] X output = LLM(X)

[0064] where LLM denotes a large language model. The core of the current mainstream large language model (LLM) is usually composed of a Transformer architecture, and the core mechanism of the Transformer is self-attention (Self-Attention), which is mathematically expressed as:

[0065]

[0066] Q = W Q X;

[0067] K = W K X;

[0068] V = W V X;

[0069] where: Q (Query), K (Key), and V (Value) are matrices obtained by linear transformation of the input sequence. d k is the dimension of the Key, used to scale the dot product to prevent gradient vanishing. W Q , W K , and W V are weight matrices, which are matrices used to linearly transform the input X in the attention mechanism calculation process. Through these matrices, the input X is mapped to three different vector spaces, namely Query (Q), Key (K), and Value (V), respectively, in order to calculate the attention score subsequently.

[0070] Each layer of the LLM (e.g., a Transformer layer) can be represented as the following composite function:

[0071] LLM(X) = LayerNorm(FFN(LayerNorm(X + Attention(X))))

[0072] Where: FFN is the feed-forward network (usually a two-layer MLP). LayerNorm is layer normalization.

[0073] To further constrain the attention of the referential token on the local semantics of the region it refers to. The attention weights in the LLM are dynamically reweighted. The specific improvement formula is as follows:

[0074]

[0075] Where: Mask ref is the visible region mask set for the referential token, and the non-referential region is -∞; Mask causal is the visible region mask of the original causal language model (each token can only see the tokens appearing before it), and the elements on the diagonal and below are 0, and the elements above the diagonal are -∞.

[0076] By reweighting the attention weights with referential awareness, the regional reference constraints can be effectively integrated without modifying the model parameters, as shown in Figure 2 .

[0077]

[0078] Masl ref is an n x n matrix, where n is the length of the input sequence, and the value of the i-th row and j-th column of the matrix can be calculated by the above formula.

[0079] Compared with the prior art, the present application has the following advantages:

[0080] 1. Strong universality (modality independent + referential form independent): includes cross-modal universality and cross-referential form scalability. For regional objects of different modalities (text, picture, video, audio, etc.), they will be modeled as a set of index positions in the token sequence. This referential form is modality independent, so it can theoretically be extended to any modality supported by a multi-modal large model. For different referential forms, they are mapped to a set of index positions through a modality-specific referential mapper. This step is rule-based, and when a new referential form is added, a mapping rule corresponding to the referential form can be added to support the referential form by the multi-modal large model.

[0081] 2、Architecture compatibility and high efficiency. Compared with Groma / Alpha-CLIP, this method does not modify the basic model architecture, only introduces a lightweight component: the reference mapper only needs to do a simple token index mapping (almost no computational burden); the ReferenceEncoder is a DeepSets structure constructed by MLP, with small computational complexity and supporting batch operation; the Attention reweighting can be directly implemented in the Transformer layer attention mask without model structure changes; therefore, this method is easy to integrate into various SOTA multi-modal models (such as Qwen-VL, MiniGPT-v2, Intern-VL, etc.); it can be flexibly plugged in and does not depend on specific perception modules (such as region proposer); in addition to introducing a very lightweight reference encoder, it does not introduce any other parameters. The reference mapper and attention reweighting in it almost do not increase the computational load of the model. Compared with the method of Groma, it has higher computational efficiency and a more streamlined model structure.

[0082] 3、Support multi-turn dialogue and KV-Cache mechanism to improve the reasoning efficiency of the context resolution. Groma / Alpha-CLIP needs to re-encode the image region for each region reference, and does not support cache reuse; in the method of the present application, all modalities are tokenized, so the reference region is in the cached KV, and can be directly referenced by position index + mask, greatly improving the reasoning efficiency.

[0083] Therefore, higher reasoning efficiency can be achieved in multi-turn image-text dialogue and multi-stage reasoning.

[0084] In summary, the above is only a preferred embodiment of the present application, and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A general cross-modal referential representation method based on position index, characterized in that: The specific process is: Set up a cross-modal reference mapper and position set encoder, and introduce an attention reweighting module in the large language model; Set up a cross-modal referent mapper to convert the referent region in the image or text into its index set in the token sequence I ref ; Set the position set encoder to convert the position index set I ref Encoded as vector representation x ref , and combine the visual token vector encoded by the visual encoder and the text token vector mapped by the text token embedding as the input sequence X of the large language model; The large language model performs feature interaction on the input sequence X and outputs a cross-modal referential representation; during the feature interaction process, the attention reweighting module is used to dynamically weight the auxiliary force weights in the large language model.

2. The universal cross-modal referential representation method based on position index according to claim 1, characterized in that: The output of the cross-modal referent mapper is: I ref =ReferenceMapper(R,T,D) Among them, R is various forms of reference, T is the modality-specific Tokenizer used by the current multimodal large language model, and D is the multimodal content.

3. The universal cross-modal referential representation method based on position index according to claim 1 or 2, characterized in that: The position set encoder sets the position index set I ref Encoded as vector representation x ref The specific process is: Position set I ref Each position index p in i Each token in the region is encoded using the ψ function, and then summed to aggregate the semantics of all tokens in the region, and then used The function decodes the reference token to map it into the representation space of the text token.

4. The universal cross-modal referential representation method based on position index according to claim 1 or 2, characterized in that: The dynamic reweighting of the attention weights in the large language model is: Q=W Q X;K=W K X;V=W V X Among them, W Q 、W K and W V is the weight matrix, d k is the dimension of Key, Mask causal Is the visible area mask, Mask ref It is the visible area mask set for the reference token, and the non-reference area is -∞.

5. The universal cross-modal referential representation method based on position index according to claim 1, characterized in that: The Mask ref for 6. A universal cross-modal referential representation device based on position index, characterized in that: Includes large language models, cross-modal referent mappers, and positional set encoders; Cross-modal referent mapper, used to convert the referent region in the image or text into its index set in the token sequence I ref ; Position set encoder, used to convert the position index set I ref Encoded as vector representation x ref , and combine the visual token vector encoded by the visual encoder and the text token vector mapped by the text token embedding as the input sequence X of the large language model; An attention reweighting module is introduced into the large language model to perform feature interaction on the input sequence X and output a cross-modal referential representation. During the feature interaction process, the attention reweighting module is used to dynamically weight the auxiliary force weights in the large language model.

7. The universal cross-modal referential representation device based on position index according to claim 6, characterized in that: The output of the cross-modal referent mapper is: I ref =ReferenceMapper(R,T,D) Among them, R is various forms of reference, T is the modality-specific Tokenizer used by the current multimodal large language model, and D is the multimodal content.

8. The universal cross-modal referential representation device based on position index according to claim 6 or 7, characterized in that: The position set encoder sets the position index set I ref Encoded as vector representation x ref The specific process is: Position set I ref Each position index p in i Each token in the region is encoded using the ψ function, and then summed to aggregate the semantics of all tokens in the region, and then used The function decodes the reference token to map it into the representation space of the text token.

9. The universal cross-modal referential representation device based on position index according to claim 6 or 7, characterized in that: The dynamic reweighting of the attention weights in the large language model is: Q=W Q X;K=W K X;V=W V X Among them, W Q 、W K and W V is the weight matrix, d k is the dimension of Key, Mask causal Is the visible area mask, Mask ref It is the visible area mask set for the reference token, and the non-reference area is -∞.

10. The universal cross-modal referential representation method based on position index according to claim 9, characterized in that: The Mask ref for:

Images