Image text alignment method based on multi-modal large language model

Abstract

The invention relates to the technical field of image texts, in particular to an image text alignment method based on a multi-modal large language model, which comprises the following steps of: segmenting an image into local areas and encoding the local areas into visual feature vectors by adopting a pre-trained visual encoder; visual and text features are mapped to the same dimension space through a learning projection layer, an InfoNCE loss function is adopted to maximize the similarity of a positive sample pair, dynamic interaction between text query and an image area is achieved through a cross attention mechanism, language module parameters are frozen to pre-train a visual encoder, an overall module is combined to be fine-tuned, and text query is completed. An error case is analyzed, and a verified and improved alignment module is obtained through a data enhancement optimization module. The method solves the problems that a traditional image text alignment method is incomplete in manual feature extraction and low in precision, an existing deep learning model is limited in cross-modal semantic understanding ability, and the generalization ability is poor when facing long-tail data.

Description

Technical Field

[0001] This invention belongs to the field of image and text technology, specifically relating to an image and text alignment method based on a multimodal large language model. Background Technology

[0002] In the fields of artificial intelligence and multimedia information processing, image-text alignment is a crucial and challenging task. With the rapid development of the internet, image and text data have exploded. How to enable computers to accurately understand the semantic relationships between images and text, and achieve efficient information retrieval, question-answering systems, intelligent recommendations, and other applications, has become a current research hotspot. Traditional image-text alignment methods are mostly based on manually designed feature extraction methods. These methods may be effective when processing simple, regular images and text, but their limitations are becoming increasingly apparent for complex and diverse real-world scenarios. Manually designed features cannot fully capture the rich semantic information of images and text, resulting in low alignment accuracy and failing to meet the needs of practical applications. In recent years, the rise of deep learning technology has brought new opportunities for image-text alignment. Methods based on deep neural networks can automatically learn feature representations of images and text, improving alignment performance to some extent. However, most existing deep learning models focus on single-modal feature extraction, with limited ability to understand cross-modal semantics. The semantic gap between different modalities still exists, causing models to perform poorly when handling complex semantic relationships.

[0003] Existing technologies have the following shortcomings: traditional methods manually design features, making it difficult to fully capture semantics and resulting in low alignment accuracy; most deep learning models focus on a single modality, with limited cross-modal semantic understanding; and they have poor generalization ability when facing long-tail data, making it difficult to accurately align rare objects and complex scenes. Summary of the Invention

[0004] To address the aforementioned issues, this invention provides an image-text alignment method based on a multimodal large language model. This method solves the problems of incomplete and low-precision manual feature extraction in traditional image-text alignment methods, limited cross-modal semantic understanding capabilities of existing deep learning models, and poor generalization ability and difficulty in accurately capturing fine-grained semantic relationships between objects and attributes when dealing with long-tailed data. To achieve the above objectives, this invention adopts the following technical solution: The image-text alignment method based on a multimodal large language model includes the following steps: A pre-trained visual encoder is used to segment the image into local regions and encode them as visual feature vectors. The text is converted into a word vector sequence through a language module, and normalization is applied to obtain a unified input representation for each modality. A projection layer is learned to map visual and text features to the same dimensional space. A visual embedding table is used to convert image patches into structured representations, aligning the two modalities at the semantic level. The InfoNCE loss function is used to maximize the similarity of positive sample pairs and minimize the similarity of negative sample pairs. The module is trained using large-scale image-text pair data to obtain a preliminary aligned visual-text joint module. A cross-attention mechanism is used to achieve dynamic interaction between text queries and image regions. A gating unit is used to control the feature fusion ratio, capturing fine-grained semantic associations between objects and attributes, resulting in a multimodal encoder with fine-grained alignment capabilities. The language module parameters are frozen to pre-train the visual encoder. A fine-tuning module is then used in conjunction with the module, and instruction learning is used to enhance the processing capabilities for complex tasks, resulting in a high-performance multimodal module. Alignment accuracy and semantic consistency are tested on general and specialized benchmarks. Error cases are analyzed, and the module is optimized through data augmentation to obtain a validated and improved alignment module.

[0005] Furthermore, the method employs a pre-trained visual encoder to segment the image into local regions and encode them into visual feature vectors. A language module then converts the text into a sequence of word vectors. Through normalization, a modally unified input representation is obtained. This process includes the following steps: Using a visual encoder, the image is segmented into multiple local regions. The visual encoder includes a Vision Transformer architecture, which extracts visual features through multi-layer Transformer encoding to obtain visual feature vectors. A pre-trained language module converts the input text into a sequence of word vectors, capturing contextual dependencies and extracting text features. Image and text features are then normalized separately using L2 normalization to ensure that their numerical ranges in the shared space are consistent, resulting in a modally unified input representation.

[0006] Furthermore, the step of learning a projection layer to map visual and text features to the same dimensional space and using a visual embedding table to convert image patches into structured representations, thereby aligning the two modalities at the semantic level, includes the following steps: First, a learnable projection layer is added after the visual and text encoders. The extracted image and text features are mapped to the same dimensional space using a projection matrix, which includes a linear layer. Second, feature vectors of a unified dimension are obtained. Third, a visual embedding table is used to map the segmented image patches into probability labels, further converting them into a representation consistent with the text embedding structure. Fourth, by explicitly modeling the visual semantic hierarchy, the inherent consistency between the visual and text modalities is strengthened, resulting in semantically aligned representations of the two modalities, visual and text.

[0007] Furthermore, the step of maximizing the similarity of positive sample pairs and minimizing the similarity of negative sample pairs using the InfoNCE loss function, and training the module with large-scale image-text pair data to obtain a preliminary aligned visual-text joint module, includes the following steps: using the InfoNCE loss function as the core optimization objective of contrastive learning, maximizing the similarity of matched positive image-text sample pairs by calculating the similarity between image embeddings and text embeddings; minimizing the similarity of mismatched negative sample pairs, and introducing the temperature parameter τ into the similarity calculation during the training process to adjust the sharpness of the sample distribution; and iteratively training on a large-scale dataset containing hundreds of millions of image-text pairs, utilizing massive amounts of data to cover diverse scenes and long-tail distributions, gradually optimizing the module parameters to obtain a preliminary aligned visual-text joint module.

[0008] Furthermore, the method of achieving dynamic interaction between text queries and image regions through a cross-attention mechanism, and using gating units to control the feature fusion ratio to capture fine-grained semantic associations between objects and attributes, resulting in a multimodal encoder with fine-grained alignment capabilities, includes the following steps: Inserting a cross-attention layer into the Transformer architecture to dynamically focus on image regions, achieving deep interaction between text words and image regions, and extracting potential associations between them; using a cross-attention mechanism to capture fine-grained semantic information between objects and attributes, including color and shape features; adding gating units to control the fusion ratio of text features and image features, avoiding information overload and redundancy, and integrating the features fused through dynamic interaction and gating to obtain a multimodal encoder with fine-grained alignment capabilities.

[0009] Furthermore, the frozen language module parameters are used to pre-train the visual encoder, and the overall module is jointly fine-tuned. Through instruction learning, the ability to handle complex tasks is enhanced, resulting in a high-performance multimodal module. This process includes the following steps: freezing the language module parameters, training only on the visual encoder and projection layer, and performing visual module-to-module text alignment pre-training using an image description dataset to initially extract and establish intermodal association features; unfreezing the language module parameters, and using the visual description dataset to jointly fine-tune the overall module, enhancing cross-modal understanding capabilities and obtaining more accurate semantic mapping; and performing instruction learning on a multimodal instruction dataset, enabling the multimodal module to generate aligned text output based on complex instructions, thus obtaining a high-performance multimodal module.

[0010] Furthermore, the process of testing alignment accuracy and semantic consistency on general and specialized benchmarks, analyzing error cases, and optimizing the alignment module through data augmentation to obtain a validated and improved alignment module includes the following steps: testing the performance of the multimodal alignment module on general multimodal benchmarks and specialized task benchmarks, measuring alignment accuracy and semantic consistency indicators, and extracting the quantitative results of the multimodal alignment module in the tests; identifying performance bottlenecks and clarifying improvement directions by analyzing alignment failure cases of the multimodal alignment module on long-tailed samples; generating targeted training samples using data augmentation techniques and adjusting the structure, including increasing the visual embedding table capacity and optimizing the multimodal alignment module's ability to handle complex scenes and rare objects; and verifying the optimization effect using benchmark testing to obtain a high-quality alignment module that has undergone rigorous evaluation and multiple rounds of iterative improvement.

[0011] Furthermore, the process of converting input text into a word vector sequence, capturing contextual dependencies, and extracting text features through a pre-trained language module includes the following steps: using a pre-trained language module as the core tool for text feature extraction, the input text sequence is converted into corresponding word vectors word by word through its internal multi-layer structure and self-attention mechanism; the self-attention mechanism automatically captures the dependencies between each word and its context, assigns different weights to words in different positions, and extracts text features containing rich semantic information; and by sequentially combining these into a complete word vector sequence, a feature representation that can characterize the overall semantics of the text is obtained, resulting in text features processed by the pre-trained language module and possessing context-aware capabilities.

[0012] Furthermore, the InfoNCE loss function is: ;in, Image and text embedding respectively. For temperature parameters, This represents the number of negative samples.

[0013] In the technical solution provided by this invention, a pre-trained visual encoder is used to segment the image into local regions and encode them into visual feature vectors. The text is converted into a word vector sequence through a language module, and normalization is performed to obtain a modally unified input representation. A projection layer is learned to map visual and text features to the same dimensional space, and a visual embedding table is used to convert image patches into structured representations, aligning the two modalities at the semantic level. The InfoNCE loss function is used to maximize the similarity of positive sample pairs and minimize the similarity of negative sample pairs. The module is trained on large-scale image-text pair data to obtain a preliminary aligned visual-text joint module. A cross-attention mechanism is used to realize the dynamic interaction between text queries and image regions. A gating unit is used to control the feature fusion ratio to capture fine-grained semantic associations between objects and attributes, resulting in a multimodal encoder with fine-grained alignment capabilities. The parameters of the language module are frozen to pre-train the visual encoder. The module is then combined with a fine-tuning module and instruction learning is used to enhance the ability to handle complex tasks, resulting in a high-performance multimodal module. The alignment accuracy and semantic consistency are tested on general and specialized benchmarks. Error cases are analyzed, and the module is optimized through data augmentation to obtain a validated and improved alignment module. This invention addresses the problems of incomplete and low-precision manual feature extraction in traditional image-text alignment methods, the limited cross-modal semantic understanding capabilities of existing deep learning models, and the poor generalization ability and difficulty in accurately capturing fine-grained semantic relationships between objects and attributes when dealing with long-tailed data. Attached Figure Description

[0014] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention.

[0015] Figure 1 This is a schematic diagram of the first embodiment of the image-text alignment method based on a multimodal large language model in this invention.

[0016] Figure 2 This is a schematic diagram of a second embodiment of the image-text alignment method based on a multimodal large language model in this invention.

[0017] Figure 3 This is a schematic diagram of the third embodiment of the image-text alignment method based on a multimodal large language model in this invention.

[0018] Figure 4 This is a schematic diagram of the fourth embodiment of the image-text alignment method based on a multimodal large language model in this invention.

[0019] Figure 5 This is a schematic diagram of the fifth embodiment of the image-text alignment method based on a multimodal large language model in this invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0021] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this specification means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0022] Image-text alignment methods based on multimodal large language models, such as Figure 1 As shown, the process includes the following steps: A pre-trained visual encoder is used to segment the image into local regions and encode them as visual feature vectors. A language module converts the text into a sequence of word vectors, and normalization is applied to obtain a modally unified input representation. A projection layer is learned to map visual and text features to the same dimensional space. A visual embedding table is used to convert image patches into structured representations, aligning the two modalities semantically. The InfoNCE loss function is used to maximize the similarity of positive sample pairs and minimize the similarity of negative sample pairs. The module is trained on large-scale image-text pair data to obtain a pre-aligned visual-text joint module. A cross-attention mechanism is used to achieve dynamic interaction between text queries and image regions. A gating unit is used to control the feature fusion ratio, capturing fine-grained semantic associations between objects and attributes, resulting in a multimodal encoder with fine-grained alignment capabilities. The language module parameters are frozen to pre-train the visual encoder. A fine-tuning module is then used in conjunction with this module. Instruction learning is used to enhance the processing capabilities for complex tasks, resulting in a high-performance multimodal module. Alignment accuracy and semantic consistency are tested on general and specialized benchmarks. Error cases are analyzed, and the module is optimized through data augmentation to obtain a validated and improved alignment module.

[0023] like Figure 2 As shown, in this embodiment, a visual encoder is used to segment the image into multiple local regions. The visual encoder includes a Vision Transformer architecture, which extracts visual features through multi-layer Transformer encoding to obtain visual feature vectors; a pre-trained language module converts the input text into a sequence of word vectors to capture contextual dependencies and extract text features; the image and text features are normalized separately using L2 normalization to ensure that their numerical ranges in the shared space are consistent, resulting in a modally unified input representation.

[0024] A visual encoder employing the Vision Transformer architecture segments images into local regions and extracts visual feature vectors, fully leveraging both local and global information to enhance understanding of image content. A pre-trained language module converts text into word vector sequences and captures contextual dependencies, accurately extracting text features and improving semantic expressiveness. L2 normalization is applied to image and text features to ensure consistency in their shared spatial numerical range, effectively avoiding alignment biases caused by modal scale differences. This lays a solid foundation for high-quality inter-modal alignment, contributing to improved accuracy and stability of the overall image-text alignment task.

[0025] L2 normalization is a commonly used data standardization method. Its core idea is to divide each element of a vector by its L2 norm (the square root of the sum of the squares of its elements), making the vector's magnitude 1. In image-text alignment tasks, image feature vectors and text feature vectors often fall into different numerical ranges, which may cause the model to favor features with larger values ​​during training. L2 normalization maps feature vectors from different modalities to a uniform magnitude scale, eliminating the influence of scale differences between modalities. This allows the model to process image and text information more fairly, thereby improving the accuracy and stability of the alignment.

[0026] like Figure 3 As shown, in this embodiment, a learnable projection layer is added after the visual and text encoders. The extracted image and text features are mapped to the same dimensional space through the projection matrix. The projection matrix includes a linear layer. A feature vector of the same dimension is obtained. A visual embedding table is used to map the segmented image blocks into probability labels, which are then further converted into a representation consistent with the text embedding structure. By explicitly modeling the visual semantic hierarchy, the inherent consistency between the visual and text modalities is strengthened, resulting in visual and text modal representations that are aligned at the semantic level.

[0027] A learnable projection layer is added, utilizing linear layers in the projection matrix to map image and text features to the same dimensional space. This resolves the issue of inconsistent feature dimensions across different modalities, creating conditions for subsequent fusion processing. A visual embedding table is employed to convert image patches into a representation consistent with the text embedding structure, explicitly modeling the visual semantic hierarchy and enabling in-depth exploration of the inherent semantic connections between visual and textual elements. This strengthens the inherent consistency between visual and textual modalities, effectively enhancing the model's ability to understand cross-modal information, and ultimately yielding semantically aligned representations of the two modalities.

[0028] like Figure 4As shown, in this embodiment, the InfoNCE loss function is used as the core optimization objective of contrastive learning. By calculating the similarity between image embeddings and text embeddings, the similarity of matched positive image-text sample pairs is maximized, while the similarity of mismatched negative sample pairs is minimized. During the training process, the temperature parameter τ is introduced into the similarity calculation to adjust the sharpness of the sample distribution. Through iterative training on a large-scale dataset containing hundreds of millions of image-text pairs, the module parameters are gradually optimized by utilizing massive amounts of data to cover diverse scenes and long-tail distributions, resulting in a preliminarily aligned visual-text joint module.

[0029] With the InfoNCE loss function as the core optimization objective, this approach accurately distinguishes between matching positive samples and mismatched negative samples, effectively narrowing the distance between positive samples and widening the distance between negative samples, thus improving the accuracy of modality alignment. A temperature parameter τ is introduced to adjust the sharpness of the sample distribution, preventing the model from overemphasizing simple negative samples and enhancing its learning ability for difficult samples. Iterative training on a large-scale dataset, covering diverse scenes and long-tailed distributions, exposes the model to rich semantic information, gradually optimizing parameters and obtaining a preliminary visual-text joint module for alignment. This lays a solid foundation for achieving more accurate and comprehensive image-text alignment tasks in the future.

[0030] like Figure 5 As shown, in this embodiment, a cross-attention layer is inserted into the Transformer architecture. By allowing text queries to dynamically focus on image regions, deep interaction between text words and image regions is achieved, and the potential correlation between the two is extracted. The cross-attention mechanism is used to capture fine-grained semantic information between objects and attributes. The fine-grained semantic information includes color and shape features. By adding a gating unit, the fusion ratio of text features and image features is controlled to avoid information overload and redundancy. The features that have undergone dynamic interaction and gating fusion are integrated to obtain a multimodal encoder with fine-grained alignment capabilities.

[0031] Inserting a cross-attention layer into the Transformer architecture enables deep interaction between text queries and image regions. This accurately uncovers potential relationships between text words and image regions, effectively capturing fine-grained semantic information such as object color and shape, significantly improving the model's ability to understand image-text details. Adding gating units to control the feature fusion ratio avoids information overload and redundancy, ensuring the quality of the fused features. After dynamically interacting and gating-fused features, the resulting multimodal encoder possesses powerful fine-grained alignment capabilities, enabling more accurate processing of complex image-text tasks.

[0032] In this embodiment, the language module parameters are frozen, and training is performed only on the visual encoder and projection layer. Visual module-to-module text alignment pre-training is performed using an image description dataset to initially extract and establish intermodal correlation features. The language module parameters are then unfrozen, and the entire module is jointly fine-tuned using the visual description dataset to enhance cross-modal understanding capabilities and obtain more accurate semantic mapping. Instruction learning is performed on a multimodal instruction dataset, enabling the multimodal module to generate aligned text output based on complex instructions, resulting in a high-performance multimodal module.

[0033] First, the language module parameters are frozen to train the visually relevant parts. Pre-training with an image description dataset allows for the initial extraction and establishment of intermodal features, laying a solid foundation for subsequent learning. After unfreezing the language module, joint fine-tuning is performed, utilizing the visual description dataset to enhance cross-modal understanding, obtain more accurate semantic mapping, and improve the model's ability to grasp the semantics of image and text. Finally, instruction learning is conducted on a multimodal instruction dataset, enabling the model to generate aligned text output based on complex instructions, thus enhancing its task processing capabilities. Through this multi-stage training, a high-performance multimodal module is finally obtained.

[0034] In this embodiment, a general multimodal benchmark and a specialized task benchmark are used to test the performance, alignment accuracy, and semantic consistency indicators of the multimodal alignment module, and to extract the quantitative results of the multimodal alignment module in the test. By analyzing alignment failure cases of the multimodal alignment module on long-tailed samples, performance bottlenecks are identified and improvement directions are clarified. Targeted training samples are generated using data augmentation techniques, and the structure is adjusted, including increasing the capacity of the visual embedding table and optimizing the multimodal alignment module's ability to handle complex scenes and rare objects. Benchmark testing is used to verify the optimization effect, resulting in a high-quality alignment module that has undergone rigorous evaluation and multiple rounds of iterative improvement.

[0035] By conducting benchmark tests on general and specialized tasks, the module's performance is quantified using alignment accuracy and semantic consistency metrics, providing a clear understanding of its strengths and weaknesses. Analyzing long-tail sample failure cases pinpoints performance bottlenecks, pointing the way for improvement. Data augmentation is used to generate targeted samples, and structural adjustments such as increasing the visual embedding table capacity effectively enhance the module's ability to handle complex scenes and rare objects. Finally, benchmark testing verifies the optimization results. After multiple rounds of iterative improvements, the resulting high-quality alignment module exhibits more stable and superior performance on multimodal alignment tasks, better meeting the needs of practical applications.

[0036] In this embodiment, a pre-trained language module is used as the core tool for text feature extraction. Through its internal multi-layer structure and self-attention mechanism, the input text sequence is converted into corresponding word vectors word by word. The self-attention mechanism automatically captures the dependency relationship between each word and the context, assigns different weights to words in different positions, and extracts text features containing rich semantic information. By sequentially combining them into a complete word vector sequence, a feature representation that can characterize the overall semantics of the text is obtained, resulting in text features that have been processed by the pre-trained language module and have context awareness capabilities.

[0037] Employing a pre-trained language module as the core tool for text feature extraction, its multi-layered structure and self-attention mechanism offer significant advantages. It can convert text sequences word by word into word vectors, automatically capturing word-contextual dependencies using the self-attention mechanism, assigning appropriate weights to words at different positions, and accurately extracting text features rich in semantics. By sequentially combining word vectors into a complete sequence, the resulting overall semantic feature representation of the text possesses context-aware capabilities. This lays a solid foundation for high-quality alignment with image features, effectively enhancing the model's ability to understand and express text semantics.

[0038] In this embodiment, the InfoNCE loss function is: ;in, Image and text embedding respectively. For temperature parameters, This represents the number of negative samples.

[0039] The InfoNCE loss function calculates the similarity between image embeddings fi and text embeddings ft, and incorporates a temperature parameter τ for adjustment, effectively measuring the matching degree of positive sample pairs. During the calculation, the numerator highlights matched image-text positive sample pairs, while the denominator comprehensively considers the influence of multiple negative samples. By maximizing the proportion of positive sample similarity and minimizing negative sample interference, the model can accurately distinguish between positive and negative samples, thereby continuously narrowing the distance between positive samples and widening the distance between negative samples during training, effectively improving the semantic alignment between images and text.

[0040] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. An image-text alignment method based on a multi-modal large language model, characterized in that, The image-text alignment method based on the multimodal large language model comprises the following steps: An image is segmented into local regions and encoded into a visual feature vector by using a pre-trained visual encoder, a text is converted into a word vector sequence by using a language module, and a modal unified input representation is obtained through normalization processing; Visual and text features are mapped to the same dimensional space by learning a projection layer, and an image block is converted into a structured representation by using a visual embedding table, so that the two types of modalities are aligned at the semantic level; An InfoNCE loss function is used to maximize the similarity of positive samples and minimize the similarity of negative samples, and a large-scale image-text pair data training module is used to obtain a preliminary aligned visual-text joint module; A cross-attention mechanism is used to realize dynamic interaction between text queries and image regions, a gating unit is used to control the feature fusion ratio, object-attribute fine-grained semantic associations are captured, and a multimodal encoder with fine-grained alignment capability is obtained; The language module parameters are frozen, the visual encoder is pre-trained, and the joint fine-tuning module is used to strengthen the complex task processing capability through instruction learning, thereby obtaining a high-performance multimodal module; The alignment accuracy and semantic consistency are tested on general and specialized benchmarks, error cases are analyzed, and a verified and improved alignment module is obtained through a data augmentation optimization module.

2. The image-text alignment method based on a multi-modal large language model according to claim 1, characterized in that, The image is segmented into local regions and encoded into a visual feature vector by using a pre-trained visual encoder, a text is converted into a word vector sequence by using a language module, and a modal unified input representation is obtained through normalization processing, comprising the following steps: An image is segmented into multiple local regions by using a visual encoder, the visual encoder comprises a VisionTransformer architecture, visual features are extracted by using a multi-layer Transformer encoder, and a visual feature vector is obtained; An input text is converted into a word vector sequence by using a pre-trained language module, context dependency relationships are captured, and text features are extracted; The image and text features are normalized respectively, an L2 normalization method is used to ensure that the numerical ranges of the two in the shared space are consistent, and a modal unified input representation is obtained.

3. The multi-modal large language model-based image-text alignment method according to claim 1, characterized in that, The image and text features are mapped to the same dimensional space by learning a projection layer, an image block is converted into a structured representation by using a visual embedding table, and the two types of modalities are aligned at the semantic level, comprising the following steps: A learnable projection layer is added after the visual and text encoders, the extracted image and text features are mapped to the same dimensional space by using a projection matrix, and the projection matrix comprises a linear layer; A unified dimensional feature vector is obtained, a visual embedding table is used to map the segmented image block into a probability label, and the probability label is further converted into a representation form consistent with the text embedding structure; The intrinsic consistency between the visual and text modalities is strengthened by explicitly modeling the visual semantic hierarchy, and the visual and text two types of modal representations are aligned at the semantic level.

4. The multi-modal large language model-based image-text alignment method according to claim 1, characterized in that, The InfoNCE loss function is used to maximize the similarity of positive sample pairs and minimize the similarity of negative sample pairs. The preliminary aligned visual-text joint module is obtained through large-scale image-text pair data training module, including the following steps: The InfoNCE loss function is used as the core optimization target of contrast learning. The similarity between image embedding and text embedding is calculated to maximize the similarity of matching image-text positive sample pairs; The temperature parameter τ is introduced into the similarity calculation during the training process to adjust the sharpness of the sample distribution; Through iterative training on a large-scale dataset containing hundreds of millions of image-text pairs, the module parameters are gradually optimized using massive data to cover diverse scenarios and long-tail distribution, and a preliminary aligned visual-text joint module is obtained.

5. The multi-modal large language model-based image-text alignment method according to claim 1, characterized in that, The cross-attention mechanism is used to realize dynamic interaction between text queries and image regions. The gating unit is used to control the feature fusion ratio to capture object-attribute fine-grained semantic association, and a multi-modal encoder with fine-grained alignment capability is obtained, including the following steps: The cross-attention mechanism is used to capture fine-grained semantic information between objects and attributes, including color and shape features. By adding a gating unit, the fusion ratio of text features and image features is controlled to avoid information overload and redundancy. The features after dynamic interaction and gating fusion are integrated to obtain a multi-modal encoder with fine-grained alignment capability. The language module parameters are frozen, and only the visual encoder and projection layer are trained. The visual module-text alignment pre-training is performed on the image description dataset to preliminarily extract and establish the inter-modal correlation features.

6. The multi-modal large language model-based image-text alignment method according to claim 1, characterized in that, The language module parameters are unfrozen, and the overall module is fine-tuned using the visual description dataset to strengthen the cross-modal understanding ability and obtain more accurate semantic mapping. Instruction learning is performed on the multi-modal instruction dataset to enable the multi-modal module to generate aligned text output based on complex instructions, and a high-performance multi-modal module is obtained. The general and specialized benchmarks are used to test alignment accuracy and semantic consistency, and the data augmentation module is used to optimize the alignment module, including the following steps: The general multi-modal benchmark and specialized task benchmark are used to test the performance of the multi-modal alignment module, including alignment accuracy and semantic consistency indicators. The quantitative results of the multi-modal alignment module in the test are extracted.

7. The multi-modal large language model-based image-text alignment method according to claim 1, characterized in that, By analyzing the alignment failure cases of the multi-modal alignment module on long-tail samples, the performance bottleneck is located and the improvement direction is clarified. ​ ​ The data augmentation method is used to generate targeted training samples, and the structure is adjusted, including increasing the capacity of the visual embedding table and optimizing the processing ability of the multimodal alignment module for complex scenes and rare objects. The benchmark test is used to verify the optimization effect, and the high-quality alignment module is obtained through strict evaluation and multiple rounds of iterative improvement.

8. The multi-modal large language model-based image-text alignment method according to claim 2, characterized in that, The pre-trained language module is used to convert the input text into a word vector sequence, capture context-dependent relationships, and extract text features, including the following steps: The pre-trained language module is used as the core tool for text feature extraction, which converts the input text sequence into corresponding word vectors word by word through internal multi-layer structure and self-attention mechanism. The self-attention mechanism automatically captures the dependency between each word and the context, assigns different weights to words in different positions, and extracts text features containing rich semantic information. Through sequential combination into a complete word vector sequence, the feature representation that can represent the overall semantics of the text is obtained, and the text features with context awareness processed by the pre-trained language module are obtained.

9. The multi-modal large language model-based image-text alignment method according to claim 4, characterized in that, The InfoNCE loss function is: wherein, are image and text embeddings, respectively, is a temperature parameter, is the number of negative samples.

Images