A visual attribute-aware representation aggregation method for multi-modal retrieval

Abstract

The application discloses a visual attribute perception representation aggregation method for multimodal retrieval, constructs a unified multimodal encoding backbone, realizes cross-modal interaction through self-attention, outputs context representation and defines a normalized retrieval space, constructs a role-based decoupling aggregation module, sets three types of learnable aggregation tokens, namely shared semantics, image-specific and text-specific, activates the corresponding role subset according to the mode, reads out independently through cross-attention, and then is pooled into a retrieval vector, constructs a general visual attribute alignment module, generates attribute description as a reference by using a visual language model in the training stage, and applies an InfoNCE contrast constraint to the image-specific representation in a low-dimensional subspace, jointly optimizes the main retrieval loss and the attribute alignment loss during training, adopts a warmup strategy to first establish a retrieval space and then introduce attribute supervision, removes the attribute alignment module during inference, and only uses the encoding backbone and the aggregation module to complete similarity calculation and sorting. The method can effectively alleviate the problem of insufficient visual attribute evidence modeling in multimodal retrieval.

Description

Technical Field

[0001] This invention belongs to the field of multimodal retrieval technology, specifically relating to a visual attribute-aware representation aggregation method for multimodal retrieval. Background Technology

[0002] Multimodal retrieval is a fundamental technology in the field of artificial intelligence, aiming to recall and query relevant evidence documents from heterogeneous corpora containing text and images. It broadly supports various upper-level applications such as open-domain question answering, general web search, and retrieval enhancement generation. Unlike traditional unimodal retrieval, which relies solely on text semantic matching, multimodal retrieval must simultaneously consider text semantic understanding and visual information parsing. It must accurately capture referential information such as entities and titles in the text while fully mining visual evidence such as color, spatial location, and object shape and state in images, achieving accurate relevance determination through multi-dimensional fusion of text and images. How to fully mine and effectively utilize refined visual features while ensuring the accuracy of text semantic retrieval is a core technical challenge that urgently needs to be overcome in the field of multimodal retrieval.

[0003] Early multimodal retrieval generally adopted a divide-and-conquer strategy, recalling candidate results separately through text and image retrieval units, and then fusing the candidate list and similarity scores in a post-processing stage. However, because the feature representation spaces and scoring distributions of different modalities are independent, this type of method is easily constrained by the incomparability of modality scores and the inconsistency of retrieval spaces, resulting in poor retrieval integration performance and low cross-modal matching accuracy. To overcome the modal fragmentation problem of the divide-and-conquer framework, subsequent research has shifted to constructing a unified, general multimodal dense semantic space. Methods such as UniVL-DR and UniIR use CLIP and BLIP models to encode text and images separately, and then generate a unified representation through routing, weighted fusion, etc., which alleviates the fusion problem to some extent. However, text and images are still processed by independent encoders, limiting the ability for deep cross-modal interaction. In recent years, projection architectures (represented by MARVEL, VISTA, and GME) have further projected visual features into the input embedding space of language models, enabling visual tokens and text tokens to complete contextual depth interactions within the same encoder, significantly improving the depth of modality fusion. Methods such as CIEA also optimize fusion results by mining complementary information about image-text differences. Nevertheless, due to the influence of image captions in real-world web corpora being mostly entity-level referential information, models still tend to focus on entity-level relevance matching, underutilizing the refined visual evidence in images.

[0004] Furthermore, aggregating fixed-dimensional representation vectors from the output of long sequence encoders is a crucial step in determining the quality of the underlying semantic space. Existing dense text retrieval models (such as DPR and BGE) generally employ single aggregation strategies such as CLS tokens and global mean pooling to generate single-vector representations to adapt to efficient ANN retrieval; multimodal retrieval models such as MARVEL and VISTA also follow this technical path. However, for heterogeneous multimodal documents, aggregating cross-modal shared semantics, textual factual clues, and image attribute evidence simultaneously through a single path can lead to competition between semantic signals from different sources, suppressing the expressive power of modality-specific evidence and further exacerbating the loss of fine-grained visual attribute information.

[0005] In summary, existing multimodal retrieval technologies have undergone iterative development from divide-and-conquer retrieval, independent encoding fusion, to projection-based deep interaction, solving problems such as inconsistency in modal spaces and insufficient shallow fusion interaction. However, in real-world heterogeneous webpage corpus scenarios, the following shortcomings still exist: First, the ability to model visual attribute-level features is insufficient. The model relies excessively on caption entity information to complete the matching, failing to effectively distinguish candidate documents with the same entity but different visual attributes, easily leading to retrieval mismatches and false recalls. Second, a single aggregation path triggers competition for multi-source semantic signals, and fine-grained visual attribute features are easily suppressed by entity semantic features, resulting in the loss of key visual information and failing to meet the requirements of high-precision, fine-grained retrieval. Therefore, there is an urgent need to provide a visual attribute-aware representation aggregation method for multimodal retrieval to address the aforementioned deficiencies of existing technologies. Summary of the Invention

[0006] To address the problems existing in the prior art, this invention provides a visual attribute-aware representation aggregation method for multimodal retrieval. This method introduces a role-based decoupling aggregation mechanism and a general visual attribute alignment module. While maintaining the semantic retrieval capability at the text entity level, it can effectively guide the image-specific aggregation path to more fully encode visual evidence related to attribute-type queries, effectively alleviating the problem of insufficient modeling of visual attribute evidence in multimodal retrieval.

[0007] To achieve the above objectives, this invention provides a visual attribute-aware representation aggregation method for multimodal retrieval, comprising the following steps: Step 1: Construct a unified multimodal coding backbone network; Construct a unified multimodal coding backbone network that maps query text, image documents and their corresponding captions to a shared representation space, and outputs token-level context representations; Step 2: Construct a role-decoupling aggregation module; set multiple learnable aggregation tokens on the document side to separate cross-modal shared semantics, visual attribute evidence and textual factual clues to different aggregation paths, and then pool them into the final document vector after being read out by the decoder through cross-attention. Step 3: Construct a general visual attribute alignment module; During the training phase, use the general visual attribute description generated by the visual language model as a language-side reference, and apply contrast constraints to the image-specific aggregated representation in a dedicated low-dimensional attribute subspace to enhance its attribute semantic expressive ability. Step 4: Construct a visual attribute perception representation aggregation model; connect the unified multimodal coding backbone network, the role decoupling aggregation module, and the general visual attribute alignment module to form a visual attribute perception representation aggregation model; Step 5: Joint training; Optimize the visual attribute perception representation aggregation model using training data. The main retrieval target and the attribute alignment target are jointly trained, and the attribute item weights are linearly warmed up with the training, so that the model first establishes the retrieval space and then gradually introduces attribute supervision. Step 6: Reasoning and Retrieval; Using the trained visual attribute perception representation aggregation model, remove the attribute alignment related modules, and only use the unified multimodal coding backbone network and the role decoupling aggregation module to encode the query and document, and complete the similarity calculation and ranking in the retrieval representation space.

[0008] As a preferred option, the process of constructing a unified multimodal coding backbone network in step 1 is as follows: S11: Construct a unified multimodal input sequence; For plain text input directly embedding its words As the backbone input, among which For word embedding layers; for inputs containing images, this is achieved by a frozen visual encoder. Image Divided into Each patch is generated and a patch-level visual representation is output. Each patch is represented by a linear projection layer. Mapping to the language model input embedding space yields a visual token sequence. ,in , , These are the learnable weight matrix and bias vector, respectively; Concatenate the visual token and caption token according to the prefix template, where the first... indivual The embedding of placeholders is determined by the projected visual representation. The sequence is filled in sequentially to obtain the embedded sequence. ,in , Special embeddings for identifying the start and end of image features. This indicates a text concatenation operation; The query and document use the same input construction method as described above, and each inputs a shared parameter encoding backbone to complete the encoding. S12: Encoder-based context modeling; via backbone encoder For input embedding sequence Perform context modeling to obtain token-level context representations. ,in For encoder parameters; Query and document sharing encoding backbone parameters; S13: Define the retrieval representation space; Define candidate corpus ,in It is a collection of plain text documents, each text document Text only For a collection of image documents, each image document From the image With corresponding caption composition; Calculate query Normalized representation vector Calculate documents Normalized representation vector ;in, and These are query encoding functions and document encoding functions, respectively. , For the final representation vector, To represent the feature dimension; With hyperspherical As a retrieval representation space, query With Documents The correlation is determined by the inner product. measure.

[0009] As a preferred option, the process of constructing the role decoupling aggregation module in step 2 is as follows: S21: Define a set of role-based aggregated tokens; Maintain a set of learnable role-based aggregate embeddings on the document side, defining the role set as follows: ,in Shared semantic roles are used to read common semantics that can be compared across modalities between image documents and text documents; As an image-side role, it is activated only in image documents, providing additional aggregation capacity for visual attribute evidence; As a text-side role, activated only within text documents, it provides additional aggregation capacity for textual factual evidence; each role Each corresponds to a learnable aggregate embedding. Each embedding is initialized either randomly or according to a normal distribution with a mean of zero. For different document types, from the complete set Select different subsets of roles; for image documents The subset of characters is For text documents The subset of characters is ; S22: Role-based decoding readout; Each aggregate token has its learnable embeddings As a decoder The input token is used to output the encoder. Cross-attention converges contextual information and uses diagonal self-attention masks. Each aggregated token is connected only to itself in self-attention to maintain role independence; For documents The decoder is located at the aggregate token position. The output hidden state is ,in It is the identity matrix. These are decoder parameters; S23: Pooling generates the final document vector; Hidden state of shared semantic token Image-side token hidden state The image document is obtained by taking the mean and normalizing. Search representation ; Hidden state of shared semantic token With text-side token hidden state The text document is obtained by taking the mean and normalizing. Search representation ; The query is processed through the same encoding process and consists of a single aggregated token. Read aloud: .

[0010] As a preferred option, the process of constructing the general visual attribute alignment module in step 3 is as follows: S31: Generate a general visual attribute description; For the Image document Visual language model In the attribute hints Generate visual attribute descriptions under the guidance of ,in guide Focusing on directly visible information in the image, emphasizing color, spatial relationships, shape, and clearly visible objects; the resulting visual attribute descriptions It is not included in the document input sequence, but only used as a language-side reference for image representation during training; S32: Encoding attribute reference; Describing visual attributes As a text query, it is encoded using the query encoding function. Encode and apply stopping gradient Fix it as a constant reference ; S33: Attribute subspace projection and alignment; The first Image document Image-side aggregation hidden state Its attribute reference is represented by the same low-dimensional projection head Mapping to the attribute subspace and normalizing it yields the visual lateral vectors. With attribute reference side vector The visual side and the reference side share the same projection head. This causes both to fall into the same attribute subspace; S34: Attribute alignment loss; valid sample set within the batch Above, InfoNCE loss is used for alignment. Positive samples are attribute references for the image itself, and negative samples are attribute references for other images in the same batch. An attribute alignment loss function is constructed. ,in Temperature hyperparameters for attribute alignment.

[0011] As a preferred option, the joint training process in step 5 is as follows: S51: Main search target; Within a unified representation space, move queries closer to relevant documents and push away irrelevant documents; let the batch size be... , For query Positive example documents, Construct the main retrieval loss function for its negative example set. ,in Temperature hyperparameters for primary retrieval loss; S52: Joint optimization; The main retrieval objective and the attribute alignment objective are jointly optimized, with attribute weights undergoing linear warmup during training. This allows the model to first establish a retrieval space and then gradually introduce attribute supervision to construct an overall loss function. ,in , For training steps, counting starts from 0. For warmup steps, For attribute alignment loss in the first... Dynamic weights of the steps for The maximum value.

[0012] As a preferred option, the reasoning and retrieval process in step 6 is as follows: S61: Inference coding; Load the model parameters saved after training, for the input query The normalized representation vector is obtained through a unified multimodal coding backbone network and a role-decoupling aggregation module. For each document in the candidate corpus The normalized representation vector is obtained through the encoding backbone and aggregation module. ; Visual language model Attribute hints With the projector head Neither of them participates in the reasoning process. The retrieval process still maintains the dual-encoder offline database building and online vector similarity retrieval, without increasing the deployment cost brought by the attribute alignment module. S62: Retrieval and sorting; For input query Represent it as a vector Representation vectors of all documents in the candidate corpus according to Calculate the inner product similarity, sort all documents in the database in descending order of similarity, and return the Top-K candidate documents as the search results.

[0013] Compared with the prior art, the present invention has the following technical advantages: 1. By constructing a visual attribute perception representation aggregation model based on a unified multimodal coding backbone, with a role-based decoupling aggregation module as the core and a general visual attribute alignment module as the training constraint, text and images are mapped to a shared representation space. This model provides differentiated aggregation paths for cross-modal shared semantics, visual attribute evidence, and textual factual clues. Simultaneously, by constraining image-specific aggregation representations through language-side attribute references, the model is guided to aggregate evidence related to attribute-based queries from visual features, achieving end-to-end mapping from raw multimodal inputs to normalized retrieval vectors. The model adopts a training and inference separation architecture. During training, the attribute alignment module guides the image aggregation path to learn attribute semantics; during inference, the training-specific module is removed, enabling the model to possess both rich visual attribute representation capabilities and efficient retrieval deployment performance.

[0014] 2. By introducing a role-based decoupling aggregation mechanism, shared semantic tokens, image-specific tokens, and text-specific tokens are set on the document side, providing differentiated aggregation paths for cross-modal shared semantics, visual attribute evidence, and textual factual clues. The two types of documents intersect using the shared semantic token, ensuring direct comparison of documents from different modalities under the shared semantic channel. Image-specific tokens and text-specific tokens are activated only in their respective modalities, providing independent aggregation capacity for modality-specific evidence. This mechanism effectively alleviates the problem of competition between semantic signals from different sources in traditional single-vector aggregation, preventing visual attribute information from being overwhelmed by entity semantics, and enhancing the modeling capability for visual attribute evidence while maintaining entity-level matching capabilities.

[0015] 3. Through a general visual attribute alignment module, based on a projective architecture, a visual language model is used to generate general visual attribute descriptions for images, such as color, spatial relationships, and shape. These descriptions are used as language-side references during the training phase. InfoNCE contrast constraints are applied to the image-specific aggregated representation within a dedicated low-dimensional attribute subspace, guiding the model to more fully encode visual evidence relevant to attribute-based queries. This module works in conjunction with a decoupled aggregation mechanism. Decoupled aggregation sets shared aggregation tokens, image-specific aggregation tokens, and text-specific aggregation tokens on the document side, providing an independent image-specific aggregation channel for attribute alignment. Attribute alignment provides attribute-level supervision signals to this channel, guiding the image-specific aggregated representation to align with the attribute descriptions in the subspace. The two are interdependent and synergistically enhance each other. This auxiliary supervision only participates in the training phase and not in the document input and index construction during the inference phase, thus not increasing model deployment costs. The retrieval process still maintains the dual-encoder offline database construction and online vector similarity retrieval, without increasing the deployment costs associated with the attribute alignment module.

[0016] 4. A linear warmup strategy is employed to jointly optimize the main retrieval target and the attribute alignment target. This allows the model to first establish a stable retrieval semantic space before gradually introducing attribute-level supervision signals. This avoids interference from attribute constraints on the optimization of the main retrieval task in the early stages of training, ensuring the stability of the training process and the final retrieval performance. Experimental results on multimodal retrieval tasks show that VARA achieves excellent performance on multiple evaluation metrics, and ablation studies further demonstrate the effectiveness of each module.

[0017] This method, through the collaborative design of a role-based decoupling aggregation mechanism and a general visual attribute alignment module, effectively guides the image-specific aggregation path to more fully encode visual evidence related to attribute-type queries while maintaining the semantic retrieval capability of text entity level. This effectively alleviates the problem of insufficient modeling of visual attribute evidence in multimodal retrieval and achieves excellent performance on multimodal retrieval tasks. Attached Figure Description

[0018] Figure 1 This is an overall flowchart of the present invention; Figure 2 This is a structural diagram of the visual attribute perception representation aggregation model in this invention. Detailed Implementation

[0019] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided below.

[0020] like Figure 1 and Figure 2 As shown, this invention provides a Visual Attribute-Aware Representation Aggregation (VARA) method for multimodal retrieval. It aims to address the core pain points of existing projective multimodal retrieval architectures, such as multi-source semantic signal competition caused by a single aggregation path and insufficient visual attribute-level evidence modeling capabilities. While ensuring retrieval efficiency, VARA enhances the model's ability to perceive and match fine-grained visual attributes. The overall model architecture of VARA comprises three core modules: a unified multimodal coding backbone network for encoding queries and candidate documents into the same representation space; a decoupled representation aggregation module for aggregating shared semantic, visual, and textual evidence from different document types; and a general visual attribute alignment module for enhancing the expressive power of image-side visual attributes. For any query and candidate document, the unified coding backbone first encodes them independently into token-level contextual representations. The aggregation module then introduces shared semantic anchors and modality-specific aggregation tokens on the document side, providing different readout paths for shared semantics and modality-specific evidence, and pooling them into the final document vector. During the training phase, the attribute alignment module uses the general visual attribute descriptions generated by the visual language model as semantic references to impose attribute-level semantic constraints on the image-specific aggregation paths.

[0021] Specifically, the following steps are included: Step 1: Construct a unified multimodal coding backbone network; Construct a unified multimodal coding backbone network that maps query text, image documents and their corresponding captions to a shared representation space, and outputs token-level context representations; As a preferred option, the process of constructing a unified multimodal coding backbone network is as follows: S11: Construct a unified multimodal input sequence; For any input instance, it can be viewed as a combination of an image (optional) and text, and encoded by a unified language model backbone. Specifically, for plain text input... directly embedding its words As the backbone input, among which For word embedding layer; For inputs containing images, the frozen visual encoder... Image Divided into Each patch is generated and a patch-level visual representation is output. To bridge the representation gap between the visual encoder and the language encoder, a linear projection layer is used to represent each patch. Mapping to the language model input embedding space yields a visual token sequence. ,in , , These are the learnable weight matrix and bias vector, respectively; Prefix Template Concatenate the visual token and the caption token, where the first... indivual The embedding of placeholders is determined by the projected visual representation. The sequence is filled in sequentially to obtain the embedded sequence. ,in , Special embeddings for identifying the start and end of image features. This indicates a text concatenation operation; The query and document use the same input construction method as described above, and each inputs a shared parameter encoding backbone to complete the encoding. S12: Encoder-based context modeling; via backbone encoder For input embedding sequence Perform context modeling to obtain token-level context representations. ,in Encoder parameters; query and document-shared encoding backbone parameters; Thus, visual tokens and text tokens achieve deep cross-modal interaction through multi-layered self-attention. The contextual representations of queries and documents are unified within the same semantic space, providing a shared semantic foundation for the retrieval representation space—this shared semantic space is the basis for subsequent normalized representation vectors. , This assumes that relevance can be directly compared through the inner product. The final retrieval vector is generated by the subsequent representation aggregation module from... Read it out.

[0022] S13: Define the retrieval representation space; Define candidate corpus ,in It is a collection of plain text documents, each text document Text only For a collection of image documents, each image document From the image With corresponding caption composition; Calculate query Normalized representation vector Calculate documents Normalized representation vector ; Enable query With Documents Encoding mapped to the same unit hypersphere ,in, and These are query encoding functions and document encoding functions, respectively. , For the final representation vector, To represent the feature dimension; With hyperspherical As a retrieval representation space, query With Documents The correlation is determined by the inner product. measure; In this technical solution, by constructing a unified input construction method and a shared encoding backbone, the word embeddings of plain text and the patch-level visual representations of images are linearly projected and concatenated into a unified sequence according to prefix templates, bridging the representational differences between vision and language. On this basis, a multi-layer self-attention mechanism of the backbone encoder enables deep cross-modal contextual interaction between visual tokens and text tokens, ensuring that the contextual representations of queries and documents fall within the same semantic space, providing a shared semantic foundation for subsequent direct comparison of inner product similarity. Simultaneously, by defining a retrieval representation space based on a unit hypersphere, the retrieval framework is unified with L2 normalization constraints and inner product metrics, laying a standardized mathematical foundation for the subsequent representation aggregation module to read the final retrieval vector from the contextual representation.

[0023] Step 2: Construct a role-decoupling aggregation module; set multiple learnable aggregation tokens on the document side to separate cross-modal shared semantics, visual attribute evidence and textual factual clues to different aggregation paths, and then pool them into the final document vector after being read out by the decoder through cross-attention. As a preferred approach, the process of constructing the role-based decoupling aggregation module is as follows: S21: Define a set of role-based aggregated tokens; The unified coding backbone outputs a token-level context representation. Dense retrieval requires compressing the data into a single vector. When this vector is generated using a single aggregation path, cross-modal shared semantics and modality-specific evidence share the same representation channel, potentially leading to competition between evidence signals from different sources. To mitigate this issue, a decoupled representation aggregation is proposed. Specifically, a set of learnable role-based aggregation embeddings is maintained on the document side. Each role embedding uses its own embedding as the decoder input position and is processed by the document encoder output. Cross-attention aggregates information related to a role from a complete contextual representation. The set of roles is defined as follows: ,in Shared semantic roles are used to read common semantics that can be compared across modalities between image documents and text documents; As an image-side role, it is activated only in image documents, providing additional aggregation capacity for visual attribute evidence; As a text-side role, activated only within text documents, it provides additional aggregation capacity for textual factual evidence; each role Each corresponds to a learnable aggregate embedding. Each embedding is initialized either randomly or according to a normal distribution with a mean of zero. For different document types, from the complete set Select different subsets of roles; for image documents The subset of characters is For text documents The subset of characters is ; and All with To share the intersection, and Each is activated only in its corresponding modality, providing independent aggregation capacity for modality-specific evidence; S22: Role-based decoding readout; Each aggregate token has its learnable embeddings As a decoder The input token of r is used to output the encoder. Cross-attention converges contextual information and uses diagonal self-attention masks. Each aggregated token is connected only to itself in self-attention to maintain role independence; For documents The decoder is located at the aggregate token position. The output hidden state is ,in It is the identity matrix. These are decoder parameters; S23: Pooling generates the final document vector; Hidden state of shared semantic token Image-side token hidden state The image document is obtained by taking the mean and normalizing. Search representation ; Hidden state of shared semantic token With text-side token hidden state The text document is obtained by taking the mean and normalizing. Search representation ; The query is processed through the same encoding process and consists of a single aggregated token. Read aloud: Thus, the document side is decoupled and aggregated from multiple role tokens, while the query side is read from a single token, and each query and each document ultimately correspond to a representation vector. This technical solution introduces a role-based decoupling aggregation mechanism, expanding document-side representation aggregation from a traditional single path to three independent paths: shared semantics, visual attributes, and textual facts. This effectively alleviates the signal overload problem caused by competition between cross-modal shared semantics and modality-specific evidence in a single aggregation vector. By maintaining a set of learnable role-based aggregation embeddings on the document side and employing a diagonal self-attention mask to ensure role independence for each aggregation token during decoding, image documents and text documents are compressed collaboratively by their respective specific roles and shared roles. This guarantees direct comparability of documents from different modalities under the shared semantic channel while preserving independent expressive capacity for visual attribute evidence and textual factual clues. Finally, by merging and normalizing the multi-role representations through mean pooling, a document retrieval vector with both cross-modal comparability and modality-specific information integrity is generated, providing a high-quality representation foundation for accurate matching of subsequent attribute-based queries.

[0024] Step 3: Construct a general visual attribute alignment module; During the training phase, use the general visual attribute description generated by the visual language model as a language-side reference, and apply contrast constraints to the image-specific aggregated representation in a dedicated low-dimensional attribute subspace to enhance its attribute semantic expressive ability. As a preferred option, the process of constructing a general visual attribute alignment module is as follows: S31: Generate a general visual attribute description; The captions of image documents primarily provide entity references (name, location, title, and source, etc.), with relatively insufficient coverage of observable attributes such as color, spatial relationships, and shape; they rely solely on image-specific aggregated tokens used during target retrieval training. The lack of attribute-level supervision signals may make the representation more prone to bias towards entity-level matching. To make Beyond entity semantics, this paper further encodes visual semantics that can be queried by attribute-based questions, proposing a general visual attribute alignment: using a general visual attribute description generated by a visual language model as an attribute semantic reference in the query domain, a contrast constraint is imposed on the image-specific aggregation path within a dedicated low-dimensional attribute subspace. Each image is assigned a general visual attribute description. For the first Image document Visual language model In the attribute hints Generate visual attribute descriptions under the guidance of ,in guide Focusing on directly visible information in the image, emphasizing color, spatial relationships, shape, and clearly visible objects; the resulting visual attribute descriptions It is not included in the document input sequence, but only used as a language-side reference for image representation during training; S32: Encoding attribute reference; To ensure that the reference and the actual retrieval query fall within the same semantic space, the visual attribute descriptions are... As a text query, it is encoded using the query encoding function. Encode and apply stopping gradient Fix it as a constant reference ; S33: Attribute subspace projection and alignment; The first Image document Image-side aggregation hidden state Its attribute reference is represented by the same low-dimensional projection head Mapping to the attribute subspace and normalizing it yields the visual lateral vectors. With attribute reference side vector The visual side and the reference side share the same projection head. This causes both to fall into the same attribute subspace; S34: Attribute alignment loss; valid sample set within the batch Above, InfoNCE loss is used for alignment. Positive samples are attribute references for the image itself, and negative samples are attribute references for other images in the same batch. An attribute alignment loss function is constructed. ,in Temperature hyperparameters for attribute alignment; By minimizing the attribute alignment loss function The model is trained such that for each image, its visual attribute representation is most similar to its own attribute description in the attribute subspace and is clearly distinguishable from the attribute descriptions of other images.

[0025] This technical solution introduces a general visual attribute alignment mechanism, utilizing a visual language model to generate general visual attribute descriptions for each image, focusing on directly visible information such as color, spatial relationships, and shape. These descriptions, after being frozen and encoded, serve as language-side attribute references, addressing the issue of missing attribute-level supervision signals caused by insufficient visual attribute coverage due to image captions emphasizing entity referential information. By applying InfoNCE contrast constraints to the image-specific aggregated representation and attribute reference within a dedicated low-dimensional attribute subspace, the solution guides the image-specific aggregated token to more fully encode visual evidence relevant to attribute-type queries. This allows it to further encode visual semantics that can be queried by attribute-type questions, beyond entity semantics, thus compensating for the shortcomings of models trained solely on retrieval targets that favor entity-level matching and neglect visual attributes. This module only functions during the training phase; the visual language model, attribute hints, and projector do not participate in forward computation during the inference phase, thus not increasing model deployment costs.

[0026] Step 4: Construct a visual attribute-aware representation aggregation model; connect the unified multimodal coding backbone network, the role decoupling aggregation module, and the general visual attribute alignment module to form a visual attribute-aware representation aggregation model; this model is based on the unified multimodal coding backbone network, with the role decoupling aggregation module as the core and the general visual attribute alignment module as the training constraint. The three work together to achieve a complete mapping of queries and documents from the original input to the normalized retrieval vector. In this technical solution, a unified multimodal coding backbone, a role-based decoupling aggregation module, and a general visual attribute alignment module are organically integrated into a complete visual attribute-aware representation aggregation model, achieving end-to-end mapping from raw multimodal inputs to normalized retrieval vectors. This model is based on a shared coding backbone, with role-based decoupling aggregation as its core architecture and attribute alignment as a training constraint. These three components complement each other and work collaboratively: the coding backbone provides the semantic foundation for cross-modal context representation; decoupling aggregation ensures that semantic signals from different sources are properly positioned and do not compete with each other; and attribute alignment injects attribute-level supervision into image-specific channels to compensate for insufficient visual attribute coverage in captions. After model construction, its training and inference phases have clearly defined responsibilities. During training, the attribute alignment module guides the image aggregation path to learn attribute semantics; during inference, the attribute-related modules do not participate in computation, enabling the model to possess both rich visual attribute representation capabilities and efficient retrieval deployment performance.

[0027] Step 5: Joint training; Optimize the visual attribute perception representation aggregation model using training data. The main retrieval target and the attribute alignment target are jointly trained, and the attribute item weights are linearly warmed up with the training, so that the model first establishes the retrieval space and then gradually introduces attribute supervision. As a preferred option, the joint training process is as follows: S51: Main search target; Within a unified representation space, move queries closer to relevant documents and push away irrelevant documents; let the batch size be... , For query Positive example documents, Construct the main retrieval loss function for its negative example set. ,in Temperature hyperparameters for primary retrieval loss; S52: Joint optimization; The main retrieval objective and the attribute alignment objective are jointly optimized, with attribute weights undergoing linear warmup during training. This allows the model to first establish a retrieval space and then gradually introduce attribute supervision to construct an overall loss function. ,in , For training steps, counting starts from 0. For warmup steps, For attribute alignment loss in the first... Dynamic weights of the steps for The maximum value; this warmup strategy enables the retrieval semantic space to be built with the main retrieval target as the main focus in the early stage of training, avoiding the interference of attribute constraints on the optimization of the main retrieval task in the early stage of training.

[0028] This technical solution achieves a balance between multi-objective collaboration and training stability during training by jointly optimizing the main retrieval objective and the attribute alignment objective, and dynamically controlling the weight of the attribute alignment loss using a linear warmup strategy. The main retrieval loss brings the query closer to positive documents and pushes away negative documents within a unified representation space, establishing a basic retrieval semantic space for the model; the attribute alignment loss guides the image-specific aggregation path to learn visual attribute evidence. By allowing the attribute weights to gradually increase from zero with each training step, the model focuses on establishing a stable cross-modal retrieval semantic space in the early stages of training. After the retrieval foundation is solid, attribute-level supervision signals are gradually introduced for fine-tuning, effectively avoiding interference from attribute constraints on the optimization of the main retrieval task in the early stages of training. This allows the model to ultimately possess both good entity-level semantic matching capabilities and the ability to express visual attribute evidence.

[0029] Step 6: Reasoning and Retrieval; Using the trained visual attribute perception representation aggregation model, remove the attribute alignment related modules, and only use the unified multimodal coding backbone network and the role decoupling aggregation module to encode the query and document, and complete the similarity calculation and ranking in the retrieval representation space.

[0030] As a preferred approach, the reasoning and retrieval process is as follows: S61: Inference coding; Load the model parameters saved after training, for the input query The normalized representation vector is obtained through a unified multimodal coding backbone network and a role-decoupling aggregation module. For each document in the candidate corpus The normalized representation vector is obtained through the encoding backbone and aggregation module. (Image document is) The text document is ); Visual language model Attribute hints With the projector head Neither of them participates in the reasoning process. The retrieval process still maintains the dual-encoder offline database building and online vector similarity retrieval, without increasing the deployment cost brought by the attribute alignment module. S62: Retrieval and sorting; For input query Represent it as a vector Representation vectors of all documents in the candidate corpus according to Calculate the inner product similarity, sort all documents in the database in descending order of similarity, and return the Top-K candidate documents as the search results.

[0031] In this technical solution, the separation of training and inference allows retrieval to be completed during the inference phase by simply loading the encoding backbone and aggregation module. Training-specific modules such as the visual language model, attribute hints, and projection heads do not participate in the forward computation of inference. The retrieval efficiency is consistent with the standard dual-encoder solution, without increasing deployment costs. Furthermore, thanks to the guidance of the image-specific aggregation path provided by the attribute alignment module during training, the normalized representation vector generated during inference already possesses rich visual attribute semantics. Cross-modal relevance can be directly and efficiently measured through inner product, balancing retrieval efficiency with visual attribute expressive power.

[0032] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A visual attribute-aware representation aggregation method for multimodal retrieval, characterized in that, Includes the following steps: Step 1: Construct a unified multimodal coding backbone network; Construct a unified multimodal coding backbone network that maps query text, image documents and their corresponding captions to a shared representation space, and outputs token-level context representations; Step 2: Construct a role-decoupling aggregation module; set multiple learnable aggregation tokens on the document side to separate cross-modal shared semantics, visual attribute evidence and textual factual clues to different aggregation paths, and then pool them into the final document vector after being read out by the decoder through cross-attention. Step 3: Construct a general visual attribute alignment module; During the training phase, use the general visual attribute description generated by the visual language model as a language-side reference, and apply contrast constraints to the image-specific aggregated representation in a dedicated low-dimensional attribute subspace to enhance its attribute semantic expressive ability. Step 4: Construct a visual attribute perception representation aggregation model; connect the unified multimodal coding backbone network, the role decoupling aggregation module, and the general visual attribute alignment module to form a visual attribute perception representation aggregation model; Step 5: Joint training; Optimize the visual attribute perception representation aggregation model using training data. The main retrieval target and the attribute alignment target are jointly trained, and the attribute item weights are linearly warmed up with the training, so that the model first establishes the retrieval space and then gradually introduces attribute supervision. Step 6: Reasoning and Retrieval; Using the trained visual attribute perception representation aggregation model, remove the attribute alignment related modules, and only use the unified multimodal coding backbone network and the role decoupling aggregation module to encode the query and document, and complete the similarity calculation and ranking in the retrieval representation space.

2. The visual attribute-aware representation aggregation method for multimodal retrieval according to claim 1, characterized in that, In step 1, the process of constructing a unified multimodal coding backbone network is as follows: S11: Construct a unified multimodal input sequence; For plain text input directly embedding its words As the backbone input, among which For word embedding layers; for inputs containing images, this is achieved by a frozen visual encoder. Image Divided into Each patch is generated and a patch-level visual representation is output. ; Each patch is represented using a linear projection layer. Mapping to the language model input embedding space yields a visual token sequence. ,in , , These are the learnable weight matrix and bias vector, respectively; Concatenate the visual token and caption token according to the prefix template, where the first... indivual The embedding of placeholders is determined by the projected visual representation. The sequence is filled in sequentially to obtain the embedded sequence. ,in , Special embeddings for identifying the start and end of image features. This indicates a text concatenation operation; The query and document use the same input construction method as described above, and each inputs a shared parameter encoding backbone to complete the encoding. S12: Encoder-based context modeling; via backbone encoder For input embedding sequence Perform context modeling to obtain token-level context representations. ,in For encoder parameters; Query and document sharing encoding backbone parameters; S13: Define the retrieval representation space; Define candidate corpus ,in It is a collection of plain text documents, each text document Text only For a collection of image documents, each image document From the image With corresponding caption composition; Calculate query Normalized representation vector Calculate documents Normalized representation vector ;in, and These are query encoding functions and document encoding functions, respectively. , For the final representation vector, To represent the feature dimension; With hyperspherical As a retrieval representation space, query With Documents The correlation is determined by the inner product. measure.

3. The visual attribute-aware representation aggregation method for multimodal retrieval according to claim 2, characterized in that, In step 2, the process of constructing the role decoupling aggregation module is as follows: S21: Define a set of role-based aggregated tokens; Maintain a set of learnable role-based aggregate embeddings on the document side, defining the role set as follows: ,in Shared semantic roles are used to read common semantics that can be compared across modalities between image documents and text documents; As an image-side role, it is activated only in image documents, providing additional aggregation capacity for visual attribute evidence; As a text-side role, activated only within text documents, it provides additional aggregation capacity for textual factual evidence; each role Each corresponds to a learnable aggregate embedding. Each embedding is initialized either randomly or according to a normal distribution with a mean of zero. For different document types, from the complete set Select different subsets of roles; for image documents The subset of characters is For text documents The subset of characters is ; S22: Role-based decoding readout; Each aggregate token has its learnable embeddings As a decoder The input token is used to output the encoder. Cross-attention converges contextual information and uses diagonal self-attention masks. Each aggregated token is connected only to itself in self-attention to maintain role independence; For documents The decoder is located at the aggregate token position. The output hidden state is ,in It is the identity matrix. These are decoder parameters; S23: Pooling generates the final document vector; Hidden state of shared semantic token Image-side token hidden state The image document is obtained by taking the mean and normalizing. Search representation ; Hidden state of shared semantic token With text-side token hidden state The text document is obtained by taking the mean and normalizing. Search representation ; The query is processed through the same encoding process and consists of a single aggregated token. Read aloud: .

4. The visual attribute-aware representation aggregation method for multimodal retrieval according to claim 1, characterized in that, In step 3, the process of constructing the general visual attribute alignment module is as follows: S31: Generate a general visual attribute description; For the Image document Visual language model In the attribute hints Generate visual attribute descriptions under the guidance of ,in guide Focusing on directly visible information in the image, emphasizing color, spatial relationships, shape, and clearly visible objects; the resulting visual attribute descriptions It is not included in the document input sequence, but only used as a language-side reference for image representation during training; S32: Encoding attribute reference; Describing visual attributes As a text query, it is encoded using the query encoding function. Encode and apply stopping gradient Fix it as a constant reference ; S33: Attribute subspace projection and alignment; The first Image document Image-side aggregation hidden state Its attribute reference is represented by the same low-dimensional projection head Mapping to the attribute subspace and normalizing it yields the visual lateral vectors. With attribute reference side vector The visual side and the reference side share the same projection head. This causes both to fall into the same attribute subspace; S34: Attribute alignment loss; valid sample set within the batch Above, InfoNCE loss is used for alignment. Positive samples are attribute references for the image itself, and negative samples are attribute references for other images in the same batch. An attribute alignment loss function is constructed. ,in Temperature hyperparameters for attribute alignment.

5. The visual attribute-aware representation aggregation method for multimodal retrieval according to claim 1, characterized in that, In step 5, the joint training process is as follows: S51: Main search target; Within a unified representation space, move queries closer to relevant documents and push away irrelevant documents; let the batch size be... , For query Positive example documents, Construct the main retrieval loss function for its negative example set. ,in Temperature hyperparameters for primary retrieval loss; S52: Joint optimization; The main retrieval objective and the attribute alignment objective are jointly optimized, with attribute weights undergoing linear warmup during training. This allows the model to first establish a retrieval space and then gradually introduce attribute supervision to construct an overall loss function. ,in , For training steps, counting starts from 0. For warmup steps, For attribute alignment loss in the first... Dynamic weights of the steps for The maximum value.

6. The visual attribute-aware representation aggregation method for multimodal retrieval according to claim 1, characterized in that, In step 6, the reasoning and retrieval process is as follows: S61: Inference coding; Load the model parameters saved after training, for the input query The normalized representation vector is obtained through a unified multimodal coding backbone network and a role-decoupling aggregation module. For each document in the candidate corpus The normalized representation vector is obtained through the encoding backbone and aggregation module. ; Visual language model Attribute hints With the projector head Neither of them participates in the reasoning process. The retrieval process still maintains the dual-encoder offline database building and online vector similarity retrieval, without increasing the deployment cost brought by the attribute alignment module. S62: Retrieval and sorting; For input query Represent it as a vector Representation vectors of all documents in the candidate corpus according to Calculate the inner product similarity, sort all documents in the database in descending order of similarity, and return the Top-K candidate documents as the search results.

Images