A Dynamic Granularity Multimodal Fine-Grained Alignment Method Based on Unbalanced Optimal Transmission

By adopting a dynamic granular multimodal alignment method based on unbalanced optimal transmission, we adaptively model the many-to-many relationship between multi-granular images and text, solving the problems of global average matching degradation and error propagation in existing technologies, and achieving efficient fine-grained semantic alignment.

CN123068979APending Publication Date: 2026-11-10SOUTHEAST UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611220563.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-12
Publication Date
2026-11-10

AI Technical Summary

Technical Problem

Existing technologies struggle to effectively establish fine-grained semantic correspondences between images and text, especially in multi-granularity scenarios where global average matching degradation and error propagation issues exist. Furthermore, reliance on external detection or segmentation tools increases system complexity.

Method used

We employ a dynamic granular multimodal alignment method based on unbalanced optimal transmission. By extracting global, regional, and block-level representations of the image and combining textual features of global, local, and detailed descriptions, we construct an unbalanced optimal transmission loss. This enables multi-granularity contrastive learning, adaptively models many-to-many soft alignment relationships, and suppresses background noise interference.

Benefits of technology

It achieves adaptive multi-granularity alignment without external tools, improves the accuracy and robustness of fine-grained alignment, simplifies the system structure, and overcomes the problems of global average matching degradation and error propagation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123068979A_ABST
    Figure CN123068979A_ABST
Patent Text Reader

Abstract

This invention discloses a dynamic granularity multimodal fine-grained alignment method based on imbalanced optimal transmission, comprising the following steps: extracting global-level features, region-level features, and block-level features of an image; extracting text features of global descriptive text, local descriptive text, and detailed descriptive text corresponding to the image; constructing a global contrastive learning loss and a dynamic granularity imbalanced optimal transmission loss between text terms and visual terms; calculating the similarity between multi-granularity image features and multi-granularity text features to obtain the final multi-granularity contrastive learning loss. This invention utilizes the multi-granularity features of images and text, defines a joint loss for dynamic granularity cross-modal fine-grained alignment based on imbalanced optimal transmission, and fully considers the correspondence between text features of different granularities and image features of corresponding granularities, thus achieving better fine-grained alignment results.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and multimodal machine learning technology, specifically relating to a dynamic granular multimodal fine-grained alignment method based on unbalanced optimal transmission. Background Technology

[0002] Fine-grained visual-linguistic alignment is an important application direction of multimodal artificial intelligence. It aims to establish precise correspondences between image regions and text semantic units, serving as an analytical method to describe cross-modal semantic associations. The alignment results clearly reflect the mapping patterns between visual content and linguistic descriptions, facilitating the understanding and discovery of visual-text semantic correspondence mechanisms. Compared to conventional image-text retrieval, fine-grained visual-linguistic alignment faces the following challenges: 1) Significant modal differences exist, with a substantial semantic gap between images and text, making it difficult to establish fine-grained semantic correspondences; 2) The granularity of visual representations is limited, making it difficult to capture valuable region-level semantic representations relying solely on global features; 3) The limited information capacity of the text makes it difficult to accommodate long text descriptions containing behavioral, scene context, and appearance attributes; 4) The high cost of acquiring fine-grained labeled data results in a limited number of effective training samples; 5) Significant background noise and illusory text interference make it difficult to handle dense, occluded, and relationally coupled targets. Therefore, conventional image-text retrieval processing models and methods cannot meet the needs of fine-grained visual-linguistic alignment. Based on the characteristics of cross-modal fine-grained semantics, it is necessary to further develop visual-linguistic fine-grained alignment technology, and on the basis of pattern recognition, deep learning, cross-modal representation learning, etc., to effectively establish local semantic correspondences that are difficult to distinguish in traditional global alignment methods.

[0003] In recent years, contrastive language-image pre-trained models (CLIP) have been widely used in visual-language understanding tasks due to their high generalization ability and zero-shot transfer capability. CLIP is a cross-modal pre-trained model with a dual-encoder structure, which is simpler and more efficient than traditional single-modal models and supervised learning-based fusion methods. Proposed by Radford et al. in 2021, CLIP maps images and corresponding text to a shared embedding space through large-scale contrastive learning, thus achieving significant breakthroughs in tasks such as zero-shot classification, image-text retrieval, and visual question answering.

[0004] While CLIP has achieved good global alignment results, it primarily processes global semantic information of images and text, neglecting region-level local information that plays a significant role in fine-grained understanding. To overcome this deficiency, many researchers have adopted a method combining region feature extraction and CLIP. This method inherits CLIP's excellent generalization performance. First, in the preprocessing stage, appropriate region features are extracted using a pre-trained object detector or segmentation model. Then, CLIP is used as the alignment model to achieve better fine-grained alignment results. However, the above methods still have problems: 1) With the rapid development of pattern recognition and deep learning, many region feature extraction methods have been successfully applied to the general image domain. However, images and text have rich multi-granular semantic information, and coarse-grained region segmentation is difficult to capture fine-grained semantic differences, making it difficult to extract appropriate region-level features. 2) In practice, the above methods heavily rely on pre-trained detection or segmentation models, and their alignment quality is limited by the performance ceiling of the detector, increasing system complexity and introducing the risk of error propagation. They are prone to missed detections and false detections for small, dynamic targets. 3) From the perspective of the matching mechanism, the classic optimal transmission (OT) requires a mass conservation constraint from the source set to the target set. In multi-granularity scenarios, this rigid constraint forces the transmission quality to be distributed approximately uniformly across all unit pairs, resulting in the alignment behavior degenerating into global average matching. At the same time, the one-to-one hard alignment mechanism cannot model the many-to-many relationship between visual and text multi-granularity representation sets, thus creating alignment noise between local details and global semantics.

[0005] Therefore, there is an urgent need for a cross-modal fine-grained alignment method that does not require external detection or segmentation tools, can adaptively model multi-granularity visual-text many-to-many correspondences, and can suppress background noise and illusory text interference. (Summary of the Invention)

[0006] The purpose of this invention is to provide a dynamic granularity multimodal fine-grained alignment method based on unbalanced optimal transmission, which includes the following steps:

[0007] The technical solution for achieving the objective of this invention is as follows:

[0008] A dynamic granular multimodal fine-grained alignment method based on unbalanced optimal transmission includes the following steps:

[0009] Step 1: Extract the global representation, region-level representation, and block-level representation of the image;

[0010] Step 2: Extract the three-granularity text features of the global description text, local description text, and detailed description text corresponding to the image;

[0011] Step 3: Construct a global contrastive learning loss and a dynamic granularity unbalanced optimal transfer loss between text and visual lexical units. Calculate the similarity between multi-granularity image features and multi-granularity text features to obtain the final multi-granularity contrastive learning loss. Use the final multi-granularity contrastive learning joint loss as the optimization objective. Update the model weights based on the gradient backpropagation algorithm using the contrastive learning joint loss. This loss is applied to the multi-granularity image-text alignment task.

[0012] A computer storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the above method.

[0013] An electronic device includes a memory and one or more processors, the memory being used to store one or more programs; when the one or more programs are executed by the one or more processors, they implement the steps of the above method.

[0014] Compared with the prior art, the significant advantages of this invention are: (1) This invention directly uses CLIP visual Transformer layer tokens of different depths to construct three levels of visual representation: global, regional, and block. It aggregates spatially related image blocks from the middle layer through attention pooling of learnable query vectors. It does not require external detection or segmentation tools, eliminates detection bottlenecks and error propagation, and has a simple system structure that supports offline feature pre-computation; (2) This invention introduces unbalanced optimal transmission and implements a quality relaxation mechanism through KL divergence edge relaxation. It allows partial matching and automatically concentrates the transmission quality on semantically related text-visual unit pairs, assigning near-zero weights to irrelevant units, thereby suppressing background noise and illusory text interference and overcoming the global average matching degradation problem caused by the classical optimal transmission quality conservation constraint; (3) This invention achieves dynamic similarity aggregation by weighting the pairwise cosine similarity through the optimal transmission plan. The optimal transmission plan adaptively learns the matching strength between text units and visual units, implicitly realizing dynamic granular routing, and modeling a many-to-many soft alignment relationship, breaking through the theoretical bottleneck of InfoNCE one-to-one hard alignment. Attached Figure Description

[0015] Figure 1 This is a diagram illustrating the overall framework of the dynamic granularity multimodal fine-grained alignment method based on unbalanced optimal transmission according to the present invention. Detailed Implementation

[0016] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are for illustrative purposes only and are not intended to limit the scope of the invention.

[0017] like Figure 1 As shown, this invention provides a dynamic granularity multimodal fine-grained alignment method based on unbalanced optimal transmission, specifically including the following steps:

[0018] Step 1: Extract the global representation, region-level representation, and block-level representation of the image;

[0019] Step 1 uses image encoder category terms as the global representation of the image, uses attention aggregation algorithm to aggregate the region-level representations of the intermediate layer of the image encoder, and uses the block terms output by the image encoder as the block-level representation.

[0020] The image encoder is based on a multi-layer visual Transformer, which initializes weights and biases by contrasting with a language-image pre-trained model, and the attention aggregation uses orthogonal initialization.

[0021] Take the category term [CLS] token features of the final layer of the visual Transformer as... ,in For the hidden layer dimensions of the visual Transformer, For the depth of the visual Transformer block, Projection layer inherited from the visual projection layer of the language-image pre-trained model Mapped to dimension The joint embedding space yields the global representation. :

[0022] (1)

[0023] The visual Transformer region-level representation method involves introducing Q orthogonally initialized learnable query vectors. ; For intermediate layer sets Each intermediate layer Take all of this layer Hidden states of image block words For each learnable query vector, perform scaled attention pooling, the first... The learnable query vector pair of the th Attention weights for each image block word for:

[0024] (2)

[0025] The attention weights are used to weight and sum the hidden states of all image block words in the corresponding layer, and then aggregated to obtain the first... Hidden status of each area :

[0026] (3)

[0027] Will After the intermediate layer Corresponding regional projection head Mapping to the joint embedding space yields image region representations. :

[0028] (4)

[0029] Visual Transformer block-level representation method: Take all of the final layers of the visual Transformer Hidden states of image block words The hidden state of each image block word is projected through an independent block-level projection head. Mapping to the joint embedding space, we get Block-level representation :

[0030] (5)

[0031] The multi-granularity visual representation set is as follows:

[0032] (6)

[0033] The total number of visual units is .

[0034] Step 2: Extract the text features of the global description text, local description text, and detailed description text corresponding to the image.

[0035] Step 2 describes three-granularity text features consisting of global description, local description, and detailed description. The global description depicts the overall scene of the image, each local description describes the structured annotation of a single entity in the image, and each detailed description describes detailed attributes such as behavior, background, species, and appearance. The formula for extracting any text feature is:

[0036] (7)

[0037] in To contrast the language-image pre-trained model text Transformer encoder, For describing text The sequence ends with a word in the final layer of the encoder. To unify the text projection matrix; the hierarchical text representation set is as follows:

[0038] (8)

[0039] in This refers to the text unit corresponding to the global description. For the first The text units corresponding to each partial description total Description, For the first The text units corresponding to the detailed descriptions total a total of Description, The total number of Chinese text units is .

[0040] Step 3: Calculate the similarity between multi-granularity image features and multi-granularity text features, construct the global contrastive learning loss and dynamic granularity unbalanced optimal transfer loss between text words and visual words, and obtain the final multi-granularity contrastive learning joint loss; use the final multi-granularity contrastive learning joint loss as the optimization objective, and update the model weights based on the contrastive learning joint loss using the gradient backpropagation algorithm. This loss is applied to the multi-granularity image-text alignment task.

[0041] The similarity between multi-granularity image features and multi-granularity text features in step 3 elements The definition is as follows:

[0042] (9)

[0043] in For the hierarchical text representation set, the first Each text unit The first in the multi-granularity visual representation set Each visual unit For the similarity metric function; the optimal transmission plan for the unbalanced optimal transmission problem. for:

[0044] (10)

[0045] in For the inner product term of the transmission cost, The optimal transmission matrix weights represent the transmission coefficients of the p-th text unit and the q-th visual unit under the current transmission plan. For entropy regularization, For generalized KL divergence, and Each of the following is a list of elements that are all 1s. Wei and 3D column vector, and These are the uniform prior distributions for the text side and the visual side, respectively. The entropy regularity coefficient is... and The KL relaxation strength coefficient controls the penalty strength for deviations of the row and column sums of the transmission plan from the prior distribution, which is the optimal transmission plan described in step 3. The solution is obtained by using entropy regularization Sinkhorn iteration.

[0046] Dynamic granularity refers to the optimal transmission plan. The elements are adaptively assigned transmission quality according to the sample content, so that the matching strength between text units of different granularities and visual units of different levels changes dynamically with the input, without the need to manually specify the granularity correspondence.

[0047] Step 3 describes the weighted aggregation of pairwise cosine similarities using the optimal transmission plan to obtain the image-text similarity. The formula is:

[0048] (11)

[0049] in All elements are of Matrix; Optimal transmission plan Middle elements For the first The text unit and the first Adaptive representation matching strength between visual units. For intra-batch... A set of image-text sample terms is used to construct... Unbalanced optimal transmission similarity matrix , its first Line number Column elements are , and The first batch The hierarchical text representation set of the nth sample and the nth A hierarchical visual representation set of samples; using temperature coefficients The first symmetric contrastive loss is constructed using symmetric InfoNCE. :

[0050] (12)

[0051] (13)

[0052] (14)

[0053] Constructing a global representation of in-batch samples The global similarity matrix is ​​used to construct a second symmetric contrastive loss using symmetric InfoNCE. :

[0054] (15)

[0055] in and The loss is calculated as follows: image-to-text comparison loss based on the global similarity matrix, and text-to-image comparison loss. The total loss is shown in the following formula:

[0056] (16)

[0057] in The weighting coefficients are used to balance fine-grained alignment and global alignment.

[0058] The technical means disclosed in this invention are not limited to those disclosed in the above embodiments, but also include technical solutions composed of any combination of the above technical features. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications are also considered within the scope of protection of this invention.

Claims

1. A dynamic granularity multimodal fine-grained alignment method based on unbalanced optimal transmission, characterized in that, Includes the following steps: Step 1: Extract the global representation, region-level representation, and block-level representation of the image; Step 2: Extract the three-granularity text features of the global description text, local description text, and detailed description text corresponding to the image; Step 3: Calculate the similarity between multi-granularity image features and multi-granularity text features, construct the global contrastive learning loss and dynamic granularity unbalanced optimal transmission loss between text words and visual words, and obtain the final multi-granularity contrastive learning joint loss. The final multi-granularity contrastive learning joint loss is used as the optimization objective. The gradient backpropagation algorithm is used to update the model weights based on the contrastive learning joint loss. This loss is applied to the multi-granularity image-text alignment task.

2. The dynamic granularity multimodal fine-grained alignment method based on unbalanced optimal transmission according to claim 1, characterized in that, Step 1 uses image encoder category terms as the global representation of the image, uses attention aggregation algorithm to aggregate the region-level representations of the intermediate layer of the image encoder, and uses the block terms output by the image encoder as the block-level representation.

3. The dynamic granularity multimodal fine-grained alignment method based on unbalanced optimal transmission according to claim 2, characterized in that, The image encoder is based on a multi-layer visual Transformer, which initializes weights and biases by contrasting with a language-image pre-trained model, and the attention aggregation uses orthogonal initialization. Take the category term [CLS] token features of the final layer of the visual Transformer as... ,in For the hidden layer dimensions of the visual Transformer, For the depth of the visual Transformer block, Projection layer inherited from the visual projection layer of the language-image pre-trained model Mapped to dimension The joint embedding space yields the global representation. : (1) The visual Transformer region-level representation method involves introducing Q orthogonally initialized learnable query vectors. ; For intermediate layer sets Each intermediate layer Take all of this layer Hidden states of image block words For each learnable query vector, perform scaled attention pooling, the first... The learnable query vector pair of the th Attention weights for each image block word for: (2) The attention weights are used to weight and sum the hidden states of all image block words in the corresponding layer, and then aggregated to obtain the first... Hidden status of each area : (3) Will After the intermediate layer Corresponding regional projection head Mapping to the joint embedding space yields image region representations. : (4) Visual Transformer block-level representation method: Take all of the final layers of the visual Transformer Hidden states of image block words The hidden state of each image block word is projected through an independent block-level projection head. Mapping to the joint embedding space, we get Block-level representation : (5) Multi-granularity visual representation set as follows: (6) The total number of visual units is .

4. The dynamic granularity multimodal fine-grained alignment method based on unbalanced optimal transmission according to claim 1, characterized in that, Step 2 describes three-granularity text features consisting of global description, local description, and detailed description. The global description depicts the overall scene of the image, each local description describes the structured annotation of a single entity in the image, and each detailed description describes detailed attributes such as behavior, background, species, and appearance. Any text feature... The extraction formula is: (7) in To contrast the language-image pre-trained model text Transformer encoder, For describing text The sequence ends with a word in the final layer of the encoder. To unify the text projection matrix; hierarchical text representation set for: (8) in This is the text unit corresponding to the global description. For the first The text units corresponding to each partial description total Item description, For the first The text units corresponding to the detailed descriptions total a total of Item description, The total number of Chinese text units is .

5. The dynamic granularity multimodal fine-grained alignment method based on unbalanced optimal transmission according to claim 1, characterized in that, The similarity between multi-granularity image features and multi-granularity text features in step 3 elements The definition is as follows: (9) in For the hierarchical text representation set, the first Each text unit The first in the multi-granularity visual representation set Each visual unit For the similarity metric function; the optimal transmission plan for the unbalanced optimal transmission problem. for: (10) in For the inner product term of transmission cost, The optimal transmission matrix weights represent the transmission coefficients of the p-th text unit and the q-th visual unit under the current transmission plan. For entropy regularization, For generalized KL divergence, and Each of the following is a list of elements that are all 1s. Wei and 3D column vector, and These are the uniform prior distributions for the text side and the visual side, respectively. The entropy regularity coefficient is... and The KL relaxation strength coefficient controls the penalty strength for deviations of the row and column sums of the transmission plan from the prior distribution, which is the optimal transmission plan described in step 3. The solution is obtained by using entropy regularization Sinkhorn iteration.

6. The dynamic granularity multimodal fine-grained alignment method based on unbalanced optimal transmission according to claim 5, characterized in that, Dynamic granularity refers to the optimal transmission plan The elements are adaptively assigned transmission quality according to the sample content, so that the matching strength between text units of different granularities and visual units of different levels changes dynamically with the input, without the need to manually specify the granularity correspondence.

7. The dynamic granularity multimodal fine-grained alignment method based on unbalanced optimal transmission according to claim 5, characterized in that, Step 3 describes the weighted aggregation of pairwise cosine similarities using the optimal transmission plan to obtain the image-text similarity. The formula is: (11) in It is a collection of multi-granularity visual representations. A hierarchical set of text representations All elements are of Matrix; Optimal transmission plan medium elements For the first The text unit and the first Adaptive representation matching strength between visual units; for batch... A set of image-text sample terms is used to construct... Unbalanced optimal transmission similarity matrix , its first Line number Column elements are , and The first batch The hierarchical text representation set of the nth sample and the nth A hierarchical visual representation set of each sample; using a temperature coefficient The first symmetric contrastive loss is constructed using symmetric InfoNCE. : (12) (13) (14) Constructing a global representation of in-batch samples The global similarity matrix is ​​used to construct a second symmetric contrastive loss using symmetric InfoNCE. : (15) in and These represent image-to-text comparison loss and text-to-image comparison loss based on the global similarity matrix, respectively; total loss. As shown in the following formula: (16) in The weighting coefficients are used to balance fine-grained alignment and global alignment.

8. A computer storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the steps of the method as described in any one of claims 1 to 7.

9. An electronic device, characterized in that, It includes a memory and one or more processors, the memory being used to store one or more programs; when the one or more programs are executed by the one or more processors, they implement the steps of the method as described in any one of claims 1 to 7.