Multimodal large model alignment method and system based on language perception and feature fusion

Abstract

The application relates to the technical field of multi-modal semantic alignment, and discloses a multi-modal large model alignment method and system based on language perception and feature fusion, and the construction and training process of the alignment model comprises the following steps: calculating the saliency score of each image block based on image block features, and generating a differentiable binary decision matrix; weighting and aggregating the important image blocks screened out through a learned aggregation weight matrix to obtain aggregated image block features; calculating an image block-text word similarity matrix based on the aggregated image block features and text word features, and calculating the overall matching score of the image and the text; and using a total loss function combining a bidirectional triple loss and a ratio constraint loss to perform end-to-end optimization on the alignment model. The application aims to use language information as a supervision signal, intelligently filter out redundant visual features, and refine fuzzy average semantic expressions into accurate semantics for specific image-text pairs.

Description

Technical Field

[0001] This invention relates to the field of multimodal semantic alignment technology, specifically to a multimodal large model alignment method and system based on language perception and feature fusion. Background Technology

[0002] Multimodal semantic alignment is a key technology in the field of large multimodal models. It is not only a core part of model training, but also provides a fundamental guarantee for the performance of downstream tasks such as cross-modal retrieval, image captioning, and visual question answering. To improve alignment accuracy, existing methods generally adopt a fine-grained alignment mechanism that combines global and local features. While achieving overall matching of cross-modal data, this mechanism delves into the fine-grained semantic relationships between image regions and text words, aiming to achieve more accurate alignment results.

[0003] However, current fine-grained alignment methods serving large multimodal models still face two significant bottlenecks. First, in the visual feature extraction process, mainstream methods typically use structures such as Vision Transformer to divide the image into multiple image patches and encode them as local visual features. While this method effectively improves the expressive power of visual features, it also introduces a large amount of redundancy and noise. A single image often yields hundreds of image patches after processing, most of which do not contain effective semantic information and lack actual correlation with the text description. A large number of irrelevant image patches can obscure a few important regions during alignment, leading to high error accumulation in local alignment and severely affecting the accuracy and stability of semantic alignment. Second, visual features and text features have an inherent inconsistency in semantic granularity. A single image patch usually corresponds to a local segment in the image, with incomplete semantic expression and a lack of contextual information, while text features often have clear semantic boundaries and a well-defined semantic structure. This inconsistency in granularity not only makes it difficult to establish a unified semantic space for cross-modal alignment but also causes visual features to exhibit an averaging and blurring trend during training, ultimately affecting the model's semantic expression and alignment capabilities at the fine-grained level. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a multimodal large model alignment method and system based on language perception and feature fusion. It aims to utilize language information as a supervisory signal to intelligently filter redundant visual features, while simultaneously calibrating and fusing the semantic and structural information of key regions, thereby refining ambiguous, averaged semantic expressions into precise semantics specific to particular image-text pairs. Specifically, this invention first uses a language-aware image patch selection module to effectively estimate the semantic meaning of visual features to select significant visual features; next, a semantic-spatial image patch aggregation module aggregates the semantic and structural information of important visual features to obtain a higher-level semantic information expression; finally, a sparse image patch-text word matching module is used to promote fine-grained alignment between visual region features and text word features, achieving semantic consistency and computational efficiency in fine-grained semantic alignment.

[0005] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0006] Firstly, this invention provides a multimodal large model alignment method based on language perception and feature fusion. The construction and training process of the alignment model includes:

[0007] Extract image patch features from images and word features from text;

[0008] The saliency score of each image patch is calculated based on the image patch features, and a differentiable binary decision matrix is ​​generated based on the Gumbel-Softmax strategy to filter out important image patches that are related to the text semantics.

[0009] The selected important image patches are weighted and aggregated using a learned aggregation weight matrix to obtain the aggregated image patch features;

[0010] The image patch-text word similarity matrix is ​​calculated based on the aggregated image patch features and text word features, and the overall matching score between the image and text is calculated through bidirectional maximum alignment interaction to achieve image-text alignment;

[0011] The alignment model is optimized end-to-end using a total loss function that combines bidirectional triplet loss and ratio constraint loss.

[0012] In one embodiment, the extraction of image patch features and text word features specifically includes:

[0013] The image is encoded using the Vision Transformer model to obtain a set of image patch features. , It is the number of encoded image blocks. This represents the feature of the Nth image patch; the text is encoded using a Transformer encoder to obtain the word feature set of the text. , Represents the feature of the Mth word in the text, where This represents the number of words in the text.

[0014] In one embodiment, calculating the saliency score for each image patch specifically includes:

[0015] Spatial information from the image is incorporated into image patch features, and a location-guided score prediction network is used to learn the salient scores of image patches. The score prediction network consists of a multilayer perceptron (MLP) and a sigmoid function. composition:

[0016] ;

[0017] in, It is the first The saliency score of each image patch Indicates the first Image patch features;

[0018] Calculate the attention score between image patches and text words, including the text-related attention score derived by calculating the cross-attention between image patch features and text word features. And image salient attention scores derived by calculating self-attention within image patches. ;

[0019] Will , and Combine the results to obtain the final significance score. :

[0020] ;

[0021] These are the weight parameters.

[0022] In one embodiment, calculating the attention score between image patches and text words includes obtaining a text-related attention score by calculating the cross-attention between image patch features and text word features. And image salient attention scores derived by calculating self-attention within image patches. Specifically, it includes:

[0023] ;

[0024] in, The standardization process representing attention; For visual global embedding, it is obtained by averaging the features of image patches; The text is globally embedded by averaging the word features of the text. Indicates transpose; The dimension representing the features of an image patch.

[0025] In one embodiment, the step of generating a differentiable binary decision matrix based on the Gumbel-Softmax strategy to filter out important image patches related to text semantics specifically includes:

[0026] Introducing the Gumbel-Softmax strategy, the Gumbel-Softmax matrix elements in for:

[0027] ;

[0028] It is the number of decision categories. , Indicates the first The image patch in the first The probability of each decision category Indicates the first The image patch in the first Gumbel noise over each decision category This is a parameter that controls the smoothness of the matrix;

[0029] from Mid-sampling and using the argmax operation to obtain the binary decision matrix Through a binary decision matrix The values ​​of each element in the table are used to filter the important image blocks.

[0030] In one embodiment, the step of weighting and aggregating the selected important image patches using a learned aggregation weight matrix to obtain aggregated image patch features specifically includes:

[0031] Filtered The set of features of important image patches is Then, an aggregation network is used to learn multiple sets of aggregation weights to aggregate features of important image patches:

[0032] ;

[0033] in, This represents the features of the j-th aggregated image patch. For the first Features of an important image patch , , This represents the number of image patches after aggregation. It is an element of an aggregate weight matrix composed of multiple aggregate weights.

[0034] In one embodiment, the step of calculating the image patch-text word similarity matrix based on the aggregated image patch features and text word features, and calculating the overall matching score between the image and text through bidirectional maximum alignment interaction, specifically includes:

[0035] Calculate the similarity between each aggregated image patch and each word of the text to generate an image patch-text word similarity matrix. , This represents the number of image patches after aggregation. The number of words in the text. elements in , indicating the first The aggregated image patch and text Similarity scores between words; , , This represents the number of image patches after aggregation. The number of words in the text; Indicates transpose. This represents the features of the j-th aggregated image patch. The first character representing the text Each word's characteristics;

[0036] The maximum correspondence interaction is used to aggregate alignment: the word with the highest alignment is extracted for each aggregated image patch, and the aggregated image patch with the highest alignment is extracted for each word. The average alignment score is calculated to represent the overall matching score between the image and the text. .

[0037] In one embodiment, the step involves extracting the word with the highest alignment for each aggregated image patch, extracting the aggregated image patch with the highest alignment for each word, and calculating the average alignment score to represent the overall matching score between the image and the text. Specifically, it includes:

[0038] .

[0039] In one embodiment, the end-to-end optimization of the alignment model using a total loss function that combines bidirectional triplet loss and ratio constraint loss specifically includes:

[0040] Using two-way triplet loss And hard sample mining strategies:

[0041] ;

[0042] , Indicates the spacing parameter. This represents a matched image-text pair. This refers to the image in a matched image-text pair. This represents the text in the matched image-text pair. and These represent the text and image of the most difficult non-matching image-text pair, respectively; within a batch, the most difficult non-matching image-text pair is the non-matching image-text pair with the highest similarity. This represents the overall matching score between the image and the text.

[0043] Ratio-constrained loss based on mean squared error loss To supervise the process, the ratio of the selected image patches is constrained to a predefined value for stable training. :

[0044] ;

[0045] This represents the selection result for the i-th image patch. This indicates that the i-th image patch is selected as the important image patch. This indicates that the i-th image patch should be discarded; the bidirectional triplet loss should be applied. With ratio constraint loss Combining these, we obtain the total loss function. :

[0046] .

[0047] In a second aspect, the present invention provides a computer system including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method of any embodiment of the first aspect.

[0048] Compared with the prior art, the beneficial technical effects of the present invention are:

[0049] First, this invention introduces a language-aware image patch selection mechanism to effectively locate and filter key visual regions highly relevant to text semantics. This not only improves the model's accuracy in multimodal semantic alignment but also enhances its ability to perceive fine-grained semantic differences, thus laying the foundation for more refined and semantically consistent cross-modal retrieval. Furthermore, the semantic-spatial image patch aggregation module proposed in this invention can adaptively aggregate selected key image patches, significantly reducing the interference of redundant visual information while fully preserving semantic information, thereby improving the compactness and effectiveness of feature representation. Finally, the sparse image patch-text word matching module of this invention, combined with bidirectional triplet loss and hard sample mining, effectively optimizes the model's ability to discriminate difficult examples and improves overall alignment performance. Attached Figure Description

[0050] Figure 1 This is a flowchart of the method in an embodiment of the present invention;

[0051] Figure 2 This is a schematic diagram of the framework in an embodiment of the present invention. Detailed Implementation

[0052] A preferred embodiment of the present invention will now be described in detail with reference to the accompanying drawings.

[0053] like Figure 1 As shown, the multimodal large model alignment method based on language perception and feature fusion in this invention includes the following steps:

[0054] S1, extract image patch features from the image and word features from the text;

[0055] S2, calculate the saliency score of each image patch based on the image patch features, and generate a differentiable binary decision matrix based on the Gumbel-Softmax strategy to filter out important image patches related to text semantics;

[0056] S3, the selected important image patches are weighted and aggregated using the learned aggregation weight matrix to obtain the aggregated image patch features;

[0057] S4. Based on the aggregated image patch features and text word features, calculate the image patch-text word similarity matrix, and calculate the overall matching score between the image and text through bidirectional maximum alignment interaction to achieve image-text alignment;

[0058] S5, perform end-to-end optimization of the alignment model using a total loss function that combines bidirectional triplet loss and ratio constraint loss.

[0059] Cross-modal semantic alignment task definition: Within the framework of a multimodal large model, given a set of images and a set of text The goal of cross-modal semantic alignment is to learn a function. Used to calculate images and text Similarity score between them. Function The goal is to calculate a higher similarity score for semantically related image-text pairs and a lower similarity score for semantically unrelated image-text pairs.

[0060] The alignment method proposed in this invention is as follows: Figure 2 As shown, it includes the following three parts: (1) a language-aware image patch selection module, which removes information redundancy and image patches that are not related to the semantics of the text; (2) a semantic-spatial image patch aggregation module, which aggregates the remaining important image patches for semantic diversity; and (3) a sparse image patch-text word matching module, which performs cross-modal semantic sparsity alignment.

[0061] This invention first extracts lexical features from multimodal data. For example, for image modality, this invention uses the Vision Transformer model to encode the image, obtaining a set of image patch features. ,in It is the number of image patches in the image. This refers to the dimension of the feature. Similarly, for the text modality, this invention uses a Transformer encoder to obtain a set of text word features. ,in This represents the number of words in the text.

[0062] 1. Language-aware image patch selection module:

[0063] This invention treats image patch selection as a discriminative task, which estimates the saliency score of each image patch and then determines the selection based on the saliency score. This invention first incorporates spatial information from the image into the image patch features and uses a location-guided score prediction network to learn the saliency score. The prediction network consists of a multilayer perceptron (MLP) and a sigmoid function.

[0064] .

[0065] in It is the first Each image patch has a saliency score, with higher values ​​indicating greater importance. However, without text supervision, relying solely on location-based modeling prediction networks to predict important image patches is insufficient for cross-modal alignment. Therefore, this invention calculates attention scores between image patches and text words to incorporate textual context information. This invention proposes two different attention scores: First, it calculates the cross-attention between image patch features and text word features to derive a text-related attention score. Secondly, this invention calculates self-attention within image patches to obtain an image salient attention score. .

[0066] .

[0067] Norm represents the attention standardization process, which makes the output value between 0 and 1, ensuring consistency with the saliency score. For visual global embedding, it is obtained by averaging the features of image patches; The text is globally embedded and obtained by averaging the word features of the text. This invention combines these scores to derive the final significance score. ,in As a weighting parameter.

[0068] .

[0069] The challenge in selecting important image patches lies in arranging the saliency score array Convert to a binary decision matrix The sampling process determines whether to retain the corresponding image patch. The most naive sampling method, such as directly selecting the top K image patches with the highest scores, is a non-differentiable process and hinders end-to-end model optimization. To overcome this problem, this invention introduces the Gumbel-Softmax strategy, providing a smooth and differentiable sampling process. Generally, the Gumbel-Softmax matrix... elements It can be written as:

[0070] .

[0071] in , This refers to the number of decision categories. In the binary decision-making method of this invention, , . It is the Gumbel distribution that represents the distribution. yes In the The values ​​that can be taken on each decision category, among which It is a uniform distribution of values ​​between 0 and 1. The smoothness of the control matrix. Finally, this invention... Mid-sampling and using the argmax operation to obtain a differentiable decision matrix. .

[0072] ;

[0073] in, This is the first column of the sampled matrix, which is a one-hot matrix obtained after the argmax operation. Therefore, the decision matrix... The result of image patch selection is represented by a value of 1, indicating that the image patch is selected, and a value of 0, indicating that the image patch is discarded. During the training phase, gradients can be fed back to the preceding neural network through such a decision matrix, enabling end-to-end training and optimization of the model.

[0074] 2. Semantic-spatial image patch aggregation module:

[0075] After selecting important image patches under language supervision, the filtered... The set of features of important image patches is , The number of important image patches is then used, and an aggregation network is used to learn multiple sets of aggregation weights to aggregate these important image patches.

[0076] ; ;

[0077] in, These are the elements of the aggregated weight matrix, which has a size of [missing information]. The algorithm is learned by a multilayer perceptron and a softmax function. The aggregation network composed of the multilayer perceptron and the softmax function can adaptively aggregate semantically similar image patches and ensure end-to-end training. This is the number of image patches after aggregation. .

[0078] 3. Sparse image patch-text word matching module:

[0079] Finally, fine-grained matching of image patches and text words is performed. This invention first calculates the similarity between each image patch and a word, generating an image patch-text word similarity matrix. , , indicating the first The first image block and text The similarity score between words is calculated using cosine similarity. Next, the invention uses maximum correspondence interaction to aggregate alignments: First, the invention extracts the text word (or view image block) with the highest alignment for each image patch (or each word). Then, the invention calculates the average of these alignment scores to represent the overall matching score between the image and the sentence.

[0080] ;

[0081] Following the previous approach, this invention uses a bidirectional triplet loss function and a hard sample mining strategy. Indicates the spacing parameter. , This represents a matched image-text pair, while and The most difficult non-matching image-text pair is the one with the highest similarity in a batch. This invention focuses on optimizing difficult sample pairs.

[0082] .

[0083] Furthermore, this invention uses mean squared error loss to supervise the process, constraining the ratio of the selected image patches to a predefined value for stable training. Finally, this invention addresses the bidirectional triplet loss. With ratio constraint loss Combining these, we obtain the total loss function. :

[0084] ;

[0085] .

[0086] During the testing phase, to improve the model's running efficiency, this invention does not use the Gumbel-Softmax strategy, but instead directly selects the model based on the magnitude of the saliency score. This approach uses key image patches, thus avoiding the sampling process and allowing the model to process subsequent aggregation steps in parallel.

[0087] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0088] It should be understood that although the steps in the flowcharts of the accompanying drawings are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the flowcharts of the accompanying drawings may include multiple steps or stages, which are not necessarily completed at the same time, but may be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but may be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0089] In one embodiment, a computer system is provided, which may be a server. The computer system includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data used in the methods described above. The network interface communicates with external terminals via a network connection. The computer program is executed by the processor to implement the methods described above.

[0090] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0091] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention, and no reference numerals in the claims should be construed as limiting the scope of the claims.

[0092] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A multi-modal large model alignment method based on language perception and feature fusion, characterized in that, The construction and training process of the alignment model used includes: extracting image block features and text word features; The saliency score of each image patch is calculated based on its features: spatial information from the image is incorporated into the patch features, and a location-guided score prediction network is used to learn the saliency score of each patch. The score prediction network consists of a multilayer perceptron (MLP) and a sigmoid function. composition: ; It is the first The saliency score of each image patch It is the first Each image patch feature is used to calculate the attention score between the image patch and the text word, including the text-related attention score obtained by calculating the cross-attention between the image patch features and the text word features. And image salient attention scores derived by calculating self-attention within image patches. ;Will , and Combine the results to obtain the final significance score. ; The weight parameters are used; a differentiable binary decision matrix is ​​generated based on the Gumbel-Softmax strategy to filter out important image patches related to text semantics; weighting and aggregating the screened important image blocks through the learned aggregation weight matrix to obtain aggregated image block features; calculating the image block-text word similarity matrix based on the aggregated image block features and text word features, and calculating the overall matching score of the image and the text through bidirectional maximum alignment interaction to realize the alignment of the image and the text; using a total loss function combining bidirectional triplet loss and ratio constraint loss to end-to-end optimize the alignment model.

2. The method of claim 1, wherein the language perception and feature fusion based multi-modal large model alignment method is characterized by, The extraction of image block features and text word features specifically includes: The image is encoded using a Vision Transformer model to obtain an image block feature set , is the number of encoded image blocks, represents the Nth image block feature; the text is encoded using a Transformer encoder to obtain a text word feature set , represents the Mth word feature of the text, wherein is the number of words in the text.

3. The method of claim 1, wherein the language perception and feature fusion based multi-modal large model alignment method is characterized by, calculating attention scores between the image patches and the text words, including text-related attention scores derived by calculating cross-attention between image patch features and word features of the text and image salient attention scores derived by calculating self-attention within the image patches , specifically comprising: ； wherein, denotes a standardization process of attention; denotes a visual global embedding obtained by averaging operation on image patch features; denotes a textual global embedding obtained by averaging operation on text word features; denotes a transpose; denotes a dimension of image patch features.

4. The method of claim 1, wherein the language perception and feature fusion based multi-modal large model alignment method is characterized by, The Gumbel-Softmax strategy is used to generate a differentiable binary decision matrix to screen important image blocks related to text semantics, specifically including: The Gumbel-Softmax strategy is introduced, and the elements in the Gumbel-Softmax matrix are: ​ ； is the number of decision classes, , denotes the probability of the th image patch to be in the th decision class, denotes the Gumbel noise for the th image patch in the th decision class, is a parameter controlling the smoothness of the control matrix; From sampling and using argmax operation to obtain a binary decision matrix , filtering the important image blocks through the values of each element in the binary decision matrix .

5. The method of claim 1, wherein the language perception and feature fusion based multi-modal large model alignment method is characterized by, The weighting and aggregation of the screened important image blocks through the learned aggregation weight matrix to obtain aggregated image block features specifically includes: Filtered The set of features of important image patches is Then, an aggregation network is used to learn multiple sets of aggregation weights to aggregate features of important image patches: ； wherein, denotes the j-th aggregated image patch feature, is the feature of the i-th important image patch, is the number of aggregated image patches, is an element of the aggregated weight matrix composed of multiple groups of aggregated weights.​​​ 6. The method of claim 1, wherein the language perception and feature fusion based multi-modal large model alignment method is characterized by, The calculation of the image block-text word similarity matrix based on the aggregated image block features and text word features, and the calculation of the overall matching score of the image and the text through bidirectional maximum alignment interaction specifically includes: Calculate the similarity between each aggregated image patch and each word of the text to generate an image patch-text word similarity matrix. , This represents the number of image patches after aggregation. The number of words in the text. elements in , indicating the first The aggregated image patch and text Similarity scores between words; , , This represents the number of image patches after aggregation. The number of words in the text; Indicates transpose. This represents the features of the j-th aggregated image patch. The first character of the text Each word's characteristics; Aggregating alignment using maximum correspondence interaction: extracting the word with the highest degree of alignment for each aggregated image block, extracting the aggregated image block with the highest degree of alignment for each word, and calculating the average value of the degree of alignment to represent the overall matching score between the image and the text .

7. The method of claim 6, wherein the language perception and feature fusion based multi-modal large model alignment method is characterized by, The highest alignment degree word is extracted for each aggregated image block, the highest alignment degree aggregated image block is extracted for each word, and the average value of the alignment degrees is calculated to represent the overall matching score between the image and the text , and specifically comprises: 。 8. The method of claim 1, wherein the language perception and feature fusion based multi-modal large model alignment method is characterized by, The end-to-end optimization of the alignment model using a total loss function combining bidirectional triplet loss and ratio constraint loss specifically includes: Using bidirectional triple loss and hard sample mining strategies: ； , denotes a distance parameter, denotes a matching image-text pair, denotes an image of a matching image-text pair, denotes a text of a matching image-text pair, and denote the text and image, respectively, of the most difficult non-matching image-text pair; in a batch, the most difficult non-matching image-text pair is the one with the highest similarity; denotes the overall matching score between image and text; Using a ratio constraint loss based on mean squared error loss to supervise the process, constraining the ratio of the selected image patches to a predefined value for stable training : ； represents the selection result of the i-th image block, represents selecting the i-th image block as an important image block, represents discarding the i-th image block; combining the bidirectional triplet loss with the ratio constraint loss to obtain a total loss function : 。 9. A computer system comprising a memory and a processor, said memory storing a computer program, characterized in that, The processor executes the computer program to realize the steps of the method in any one of claims 1 to 8.

Images