Image-text alignment method and system based on multi-modal hybrid expert model

Abstract

The invention discloses an image-text alignment method and system based on a multi-modal hybrid expert model, and the method comprises the steps: constructing the multi-modal hybrid expert model which comprises a heterogeneous expert cooperation module, a cosine similarity dynamic routing module and a difference perception comparison learning module; the heterogeneous expert cooperation module uses a plurality of pre-trained visual model experts and a plurality of pre-trained text model experts to extract features of images and texts respectively, and maps the features to a semantic space of a unified dimension through a projection layer; a cosine similarity dynamic routing module calculates importance scores of different experts based on cosine similarity to realize adaptive fusion of features; a difference perception contrast learning module calculates a similarity matrix between image-text features, and performs multi-modal hybrid expert model training in combination with basic contrast loss and dynamically weighted hard negative example loss; and carrying out image-to-text or text-to-image bidirectional retrieval based on the multi-modal hybrid expert model. According to the method, the cross-modal retrieval performance is greatly improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-modal image-text retrieval, and particularly relates to an image-text alignment method and system based on a multi-modal hybrid expert model. BACKGROUND

[0002] With the rapid development of the Internet and artificial intelligence technology, multi-modal data such as images and texts are growing explosively. According to statistics, more than 3 billion digital images are generated globally every day in 2023, and more than 5000 TB of text content is generated on social media platforms every day. How to accurately understand and match the semantic association between different modalities to achieve high-quality image-text retrieval has become an important research topic in the field of artificial intelligence. In many application scenarios such as e-commerce search, intelligent media, and content recommendation, users often need to retrieve related images through text descriptions or find corresponding text descriptions based on images. The industry's mainstream search engines and recommendation systems are actively exploring more accurate image-text alignment technology to improve user experience and business value.

[0003] The core goal of image-text retrieval is to semantically match the input query (which can be an image or text) with the content in the target library and output the most relevant results. To achieve this goal, two key challenges need to be addressed: 1) how to effectively extract semantic features of different modal data, and 2) how to accurately measure the relevance between cross-modal content. The development of deep learning provides new ideas and methods to solve these problems. Existing image-text retrieval methods can be mainly divided into the following categories: 1) Single-modal feature-based methods Early image-text retrieval mainly used single models to process image and text features respectively. For example, CNN models such as ResNet were used to extract visual features of images, and pre-trained language models such as BERT were used to extract semantic features of texts, and then simple cosine similarity or other measurement methods were used for matching. Hong et al. proposed an image feature extraction method based on deep convolutional neural networks, and used Word2Vec model to process text features. Li et al. proposed TranSearch, which uses transfer learning to enhance the expressiveness of visual features. Wang et al. explored text feature extraction methods based on attention mechanisms. This type of method is simple to implement, but its retrieval accuracy is limited due to the neglect of semantic association between modalities. In addition, the feature extraction capability of single models is often limited by their architecture design, making it difficult to fully capture complex semantic information.

[0004] 2) Dual-stream architecture-based methods To better model cross-modal semantic relationships, researchers have proposed dual-stream architectures that use two separate encoders to process images and text. The most representative is the CLIP model proposed by OpenAI, which learns a unified feature space by pre-training on large-scale image-text pairs. Subsequently, Lu et al. proposed ViLBERT, which enhances modality interaction through a bidirectional Transformer. Chen et al. proposed UNITER, which introduced multi-task learning to improve feature representation. Radford et al. designed a larger-scale image-text pre-training model. These methods have achieved good results in specific scenarios, but still have the following limitations: 1) Each modality relies only on a single model for feature extraction, limiting the expressiveness. 2) The pre-training process relies on a large amount of labeled data, limiting the transferability. 3) The model structure is fixed, making it difficult to adapt to different retrieval scenarios.

[0005] 3) Multi-modal fusion-based methods To fully utilize multi-modal information, researchers have begun to explore more complex feature fusion mechanisms. Mainly including: 1) Attention-based fusion, such as the cross-modal attention network proposed by Yu et al., and the hierarchical attention model designed by Kim et al. 2) Graph neural network-based fusion, such as the heterogeneous graph network proposed by Li et al., and the relationship-enhanced graph network developed by Zhang et al. 3) Knowledge-enhanced fusion, such as the introduction of a knowledge graph to assist feature alignment by Wang et al., and the combination of common sense reasoning to improve semantic understanding by Chen et al. However, most of these methods use static fusion strategies, lacking dynamic adaptability to different input characteristics. At the same time, complex fusion mechanisms also bring problems such as large computational overhead and difficult training. SUMMARY

[0006] The present application is directed to the problems of single feature expression, static fusion strategy, and insufficient ability to distinguish difficult examples in contrast learning in existing cross-modal image-text retrieval technology, and proposes an image-text alignment method and system based on a multi-modal hybrid expert model.

[0007] To achieve the above purpose, the technical scheme adopted by the present application is as follows: The present application proposes an image-text alignment method based on a multi-modal hybrid expert model, comprising: A multi-modal hybrid expert model is constructed, which includes a heterogeneous expert cooperation module, a cosine similarity dynamic routing module, and a difference perception contrast learning module; the heterogeneous expert cooperation module is used to extract features of images and texts using a plurality of pre-trained visual model experts and a plurality of pre-trained text model experts respectively, and map the features to a unified dimensional semantic space through a projection layer; the cosine similarity dynamic routing module is used to calculate the importance score of different experts based on the cosine similarity, and realize adaptive fusion of the features; the difference perception contrast learning module is used to calculate a similarity matrix between the image and text features, and combine a basic contrast loss and a dynamically weighted hard negative example loss to train the multi-modal hybrid expert model. Based on the multi-modal hybrid expert model, bidirectional retrieval of image to text or text to image is performed.

[0008] Further, the visual model experts include DINO-ViT-B / 8 (good at capturing global semantic structure), ConvNeXtV2-Tiny-1k-224 (good at extracting local detail features), and SigLIP-base-patch16-256 (as a benchmark expert to guide cross-modal semantic anchoring) visual encoders; the text model experts include E5-base-v2 (optimized for text retrieval) and SigLIP (capable of good cooperation with visual features) text encoders.

[0009] Further, in the heterogeneous expert cooperation module, the features extracted by different experts are mapped to a unified feature space through an unbiased linear projection layer and normalized to ensure that the features of different experts have consistent scales in the unified semantic space.

[0010] Further, in the cosine similarity dynamic routing module, the importance score of different experts is calculated in the following manner: The cosine similarity between the expert features is calculated, and a temperature parameter is used to adjust the distribution to obtain a similarity matrix; The importance score of each expert is calculated based on the similarity matrix: wherein represents the importance score of the i-th expert; i is a batch size; represents the average similarity between the feature corresponding to the i-th expert and the features corresponding to all the experts in the j-th batch of samples. b i N

[0011] Further, in the cosine similarity dynamic routing module, after obtaining the importance score, it further includes: ​​​​pass The function converts the importance score into a routing probability: in Indicates the first i The routing probability of an expert exp Indicates taking the exponent; Based on the routing probabilities, the one with the highest probability is selected. One expert, regarding the above The final feature representation is obtained by weighting and combining the features of each expert.

[0012] The cosine similarity-based dynamic routing module can adaptively select the most relevant experts according to the characteristics of the input content, which significantly improves the model's adaptability to different inputs.

[0013] Furthermore, the difference-aware contrastive learning module first calculates the similarity matrix between image and text features, and then calculates a bidirectional cross-entropy loss as the basic contrastive loss. Simultaneously, by selecting the most difficult negative sample (the negative sample with the highest similarity to the positive sample), the similarity difference between positive and negative samples is calculated, and the weight of the hard negative sample loss is dynamically adjusted based on this. When the positive and negative samples are clearly distinguishable (large similarity difference), the weight of the hard negative sample loss is reduced; when the positive and negative samples are difficult to distinguish (small similarity difference), the weight of the hard negative sample loss is increased, strengthening the learning of difficult samples.

[0014] Furthermore, the basic contrastive loss is a bidirectional cross-entropy loss: In the formula, in Based on the comparison loss, and These represent the losses in the image-to-text and text-to-image directions, respectively. This represents the function for calculating cross-entropy. This represents the similarity matrix between image and text features. The labels represent the data in this batch. For learnable temperature parameters, For image feature matrix, This is the text feature matrix.

[0015] Furthermore, the hard negative example loss is: in For hard negative examples of loss, It is a hyperparameter used to control the minimum difference between positive and negative samples. is a similarity matrix from image to text, is a similarity matrix from text to image, is a similarity of positive samples, denotes the gap between the similarity of the hardest negative example from image to text and the similarity of positive samples, denotes finding the maximum negative example similarity for each image in the similarity matrix from image to text, denotes the gap between the similarity of the hardest negative example from text to image and the similarity of positive samples, wherein denotes finding the maximum negative example similarity for each image in the similarity matrix from image to text.

[0016] Further, in the difference-aware contrast learning module, the dynamic weight of the hard negative loss is: In the formula, wherein is the dynamic weight of the hard negative loss, denotes function, is the initial weight of the hard negative loss, and respectively denote the mean of the positive example similarity and the negative example similarity.

[0017] Further, in actual retrieval matching use, the fused features are used to calculate a graph-text similarity matrix, supporting bidirectional retrieval of image searching text and text searching image, and returning the most matched result according to the similarity ranking. The system realizes the whole process automation and intelligent processing from multi-modal input to retrieval result, greatly improves the accuracy and robustness of cross-modal retrieval, and provides strong technical support for graph-text content retrieval in complex scenarios.

[0018] Another aspect of the present application provides a graph-text alignment system based on a multi-modal hybrid expert model, comprising: The multi-modal hybrid expert model construction unit is configured to construct a multi-modal hybrid expert model, which comprises a heterogeneous expert cooperation module, a cosine similarity dynamic routing module, and a difference perception contrast learning module; the heterogeneous expert cooperation module is configured to extract features of images and texts using a plurality of pre-trained visual model experts and a plurality of pre-trained text model experts respectively, and map the features to a unified dimensional semantic space through a projection layer; the cosine similarity dynamic routing module is configured to calculate importance scores of different experts based on cosine similarity to realize adaptive fusion of the features; and the difference perception contrast learning module is configured to calculate a similarity matrix between the image and text features, and combine a basic contrast loss and a dynamically weighted hard negative example loss to train the multi-modal hybrid expert model. The retrieval matching unit is configured to perform bidirectional retrieval of images to texts or texts to images based on the multi-modal hybrid expert model.

[0019] Compared with the prior art, the present application has the following beneficial effects: 1. The present application realizes adaptive fusion of cross-modal features by constructing a dynamic routing multi-modal expert system combined with a difference perception contrast learning technology. The collaborative work of multiple visual and text experts can capture complex semantic information from different angles, effectively solving the problem of limited expression ability of a single model and greatly improving the cross-modal retrieval performance.

[0020] 2. The present application designs a dynamic routing mechanism based on feature consistency, which adaptively adjusts the expert selection strategy through a learnable temperature parameter. This dynamic mechanism is more flexible than the traditional fixed structure method, can select the most suitable expert combination according to the characteristics of the input content, and significantly improves the adaptability of the model to different types of input.

[0021] 3. The innovative difference perception contrast learning strategy dynamically adjusts the difficulty weight of the example, enhancing the model's ability to distinguish similar samples. This method remedies the defect of traditional contrast learning that ignores the difficulty difference of samples, and performs excellently especially when dealing with samples with subtle semantic differences, providing a new solution for cross-modal retrieval technology. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 The schematic diagram of the architecture of the multi-modal hybrid expert model provided by the embodiment of the present application is shown in the figure; Figure 2 The schematic diagram of the structure of the heterogeneous expert cooperation module provided by the embodiment of the present application is shown in the figure; Figure 3 The schematic diagram of the structure of the cosine similarity dynamic routing module provided by the embodiment of the present application is shown in the figure; Figure 4 The flowchart of the image-text alignment method based on the multi-modal hybrid expert model provided by the embodiment of the present application is shown in the figure. Detailed Implementation

[0023] The present invention will be further explained below with reference to the accompanying drawings and specific embodiments: The present invention provides a text-image alignment method based on a multimodal hybrid expert model, comprising: A multimodal hybrid expert (image-text alignment retrieval) model is constructed, comprising a heterogeneous expert collaboration module, a cosine similarity dynamic routing module, and a difference-aware contrastive learning module. The heterogeneous expert collaboration module extracts features from images and text using multiple pre-trained visual model experts and multiple pre-trained text model experts, respectively, and maps these features to a unified semantic space through a projection layer. The cosine similarity dynamic routing module calculates importance scores for different experts based on cosine similarity, achieving adaptive feature fusion. The difference-aware contrastive learning module calculates the similarity matrix between image and text features, and trains the multimodal hybrid expert model by combining basic contrastive loss and dynamically weighted hard negative example loss. Bidirectional retrieval of images to text or text to images is performed based on a multimodal hybrid expert model.

[0024] The multimodal hybrid expert model constructed in this invention, such as Figure 1 As shown, it mainly consists of three core modules: a heterogeneous expert collaboration module, a cosine similarity dynamic routing module, and a difference-aware contrastive learning module.

[0025] 1. Heterogeneous Expert Collaboration Module like Figure 2 As shown, this heterogeneous expert collaboration module adopts a dual-tower architecture. The image (encoder) expert set includes three pre-trained visual models: DINO-ViT-B / 8, ConvNeXtV2-Tiny-1k-224, and SigLIP-base-patch16-256. DINO-ViT-B / 8 uses a Vision Transformer architecture, extracting the feature vector corresponding to the first position [CLS] marker in the input sequence as the global representation of the image, and outputting a 768-dimensional feature vector, which excels at capturing the global semantic structure of the image. The ConvNeXtV2-Tiny-1k-224 model is based on an improved convolutional neural network design, extracting the global average pooling features of the last hidden state, also outputting a 768-dimensional feature vector, and has advantages in extracting local detail features. The SigLIP visual encoder is based on a 16×16 image patch VisionTransformer architecture, outputting a 768-dimensional feature vector, and is introduced into the system as a baseline expert, using its cross-modal alignment characteristics obtained through large-scale pre-training to guide the feature projection learning of other experts.

[0026] The text (encoder) expert ensemble comprises two pre-trained language models: E5-base-v2 and the SigLIP text encoder. E5-base-v2 is a language model optimized for text retrieval, possessing powerful unimodal semantic understanding capabilities. The SigLIP text encoder, through cross-modal pre-training, exhibits excellent synergy with visual features. These expert models with distinct characteristics collectively constitute a heterogeneous expert ensemble capable of comprehensively capturing multi-granular semantic representations.

[0027] To ensure effective fusion of the output features from different experts, this invention performs unified projection processing on the features of each expert. Specifically, for the first... The features extracted by each expert model are mapped to a unified feature space through a linear transformation layer with an unbiased term: in For a trainable projection matrix, For the first i Features extracted by an expert model, ultimately representing the expert's features. The feature output dimension is uniformly set to 1024. This unbiased term design reduces the risk of overfitting while ensuring the linearity of the feature mapping. The projected features are then further processed... Normalization ensures that different expert features have a consistent scale in a unified semantic space.

[0028] 2. Cosine Similarity Dynamic Routing Module like Figure 3 As shown, this module achieves dynamic selection and fusion of experts based on feature similarity. The projected features of all experts are stacked along the expert dimension to form a feature matrix. ,in For batch size, For the number of experts, The feature dimension is 1024. The dot product similarity (i.e., cosine similarity) between features is calculated, and the distribution is adjusted using a temperature parameter. in The temperature parameter is set to a learnable value, with an initial value of 0.1. This is a similarity matrix. Temperature parameters. The introduction of this feature can adjust the smoothness of the similarity distribution and prevent the routing weights from becoming too concentrated. Simultaneously, an importance score for each expert is calculated based on the similarity matrix. in, Indicates the first i The importance score of each expert For batch size, Indicates the first b In the batch of samples, the first i The characteristics corresponding to each expert and all N The average similarity of features corresponding to each expert.

[0029] That is, take the average of the similarity matrix along the third dimension, and then... The function converts importance scores into routing probabilities: in The exponentiation is taken, resulting in a probability distribution that sums to 1, representing the routing probability (i.e., the probability of being selected) of each expert. This dynamic routing mechanism based on cosine similarity can adaptively select the most relevant expert according to the characteristics of the input content, significantly improving the model's adaptability to different inputs.

[0030] Based on the calculated route probabilities, the route with the highest probability is selected. One expert (K=2 in this implementation). Regarding this... The final feature representation is obtained by weighting and combining the features of each expert: in For the first The normalized weights of the selected experts are determined, and normalization ensures that the sum of the weights of all selected experts is 1, i.e., it satisfies... Meanwhile This refers to the characteristics of the selected expert.

[0031] The Top-K strategy ensures that only the most relevant experts are selected for fusion, avoiding noise and computational overhead caused by redundant experts.

[0032] The cosine similarity dynamic routing module enhances the model's feature fusion capability for different input contents by dynamically allocating the importance of different experts based on input characteristics through adaptive weight allocation based on cosine similarity, combined with learnable temperature parameters and Top-k selection strategy, thereby further improving the accuracy of cross-modal retrieval.

[0033] 3. Difference Perception and Comparison Learning Module First, calculate the similarity matrix between image and text features. in, For learnable temperature parameters, For image feature matrix, This is the text feature matrix.

[0034] Based on this, the bidirectional cross-entropy loss is calculated: where represents the cross-entropy calculation function, represents the label of the batch data, respectively represent the loss of the graph-to-text and text-to-graph directions.

[0035] Hard negative loss is to select the hardest negative sample and calculate the distance loss between it and the positive sample.

[0036] where is a hyperparameter to control the minimum gap between positive and negative samples. is the similarity matrix of image-to-text. is the similarity matrix of text-to-image. is the similarity of the positive sample, which is usually the diagonal element of the similarity matrix, because the diagonal line represents the matching image-text.

[0037] where the first part is: represents the gap between the similarity of the hardest negative example of image-to-text and the similarity of the positive sample, where represents finding the maximum negative example similarity of each image in the image-to-text similarity matrix.

[0038] And the second part is: represents the gap between the similarity of the hardest negative example of text-to-image and the similarity of the positive sample, where represents finding the maximum negative example similarity of each image in the image-to-text similarity matrix.

[0039] Take the average of the results of the above two parts to get the final hard negative loss. However, we still need to determine the appropriate weight for the hard negative loss after calculating it. Here we use dynamic weight, which adjusts its weight after each calculation. Its weight is not large in the early training to facilitate fast convergence, and its weight increases in the later training to enhance the model's negative mining ability. For dynamic weight, we define it as follows: where respectively represent the mean of the positive similarity and the negative similarity. That is, the dynamic weight is defined as follows: where is a function, is the initial weight of the hard negative loss (hyperparameter, default value is 0.15).

[0040] In summary, the final loss is expressed as follows: The difference perception contrast learning module adjusts the weight dynamically, considers the similarity difference between positive and negative sample pairs, calculates the average similarity difference between the positive sample and the most difficult negative sample, and dynamically adjusts the difficult example learning weight, thereby enhancing the model's ability to distinguish difficult examples and significantly improving the cross-modal matching accuracy.

[0041] Further, for the corresponding retrieval use stage, the single modal information and the to-be-retrieved modal information are both formed into feature vectors after passing through the encoder of the hybrid expert architecture.

[0042] For image query: For text query: wherein, is the query feature, is the feature in the feature library, is a learnable temperature parameter.

[0043] Sort each row of the similarity matrix to obtain the similarity score ranking result of each query. The top results with the highest similarity can be returned.

[0044] On the basis of the above embodiment, the application further provides a graph-text alignment system based on a multi-modal hybrid expert model, comprising: A multi-modal hybrid expert model construction unit is configured to construct a multi-modal hybrid expert model, wherein the multi-modal hybrid expert model comprises a heterogeneous expert cooperation module, a cosine similarity dynamic routing module, and a difference perception contrast learning module; the heterogeneous expert cooperation module is configured to use a plurality of pre-trained visual model experts and a plurality of pre-trained text model experts to extract features of images and texts respectively, and map the features to a unified dimensional semantic space through a projection layer; the cosine similarity dynamic routing module is configured to calculate importance scores of different experts based on cosine similarity to realize adaptive fusion of the features; and the difference perception contrast learning module is configured to calculate a similarity matrix between the image and text features, and train the multi-modal hybrid expert model in combination with a basic contrast loss and a dynamically weighted hard negative loss. A retrieval matching unit is configured to perform bidirectional retrieval of images to text or text to images based on the multi-modal hybrid expert model.

[0045] Based on the above embodiments, as shown in Figure 4 Another multimodal hybrid expert model-based image-text alignment method of the present application includes two main stages: training process and inference retrieval process. Step 1: Input of image-text pair The system receives paired image and text data as input. Each image expert (including DINO-ViT-B / 8, ConvNeXtV2-Tiny-1k-224 and SigLIP visual encoder) is equipped with its own image preprocessor to ensure that the image data is standardized and processed effectively. Similarly, the text expert (including E5-base-v2 and SigLIP text encoder) also normalizes the input text.

[0046] Step 2: Image-text encoding feature extraction Image and text information are processed by multi-expert feature extraction network. The image is processed by multiple visual experts to extract visual features at different levels and angles, and the text is processed by multiple language experts to extract text semantic information at different granularities. The features extracted by each expert are mapped to a unified 1024-dimensional feature space through an unbiased linear projection layer, and normalized to ensure scale consistency.

[0047] The system converts the similarity into expert selection probability by calculating the cosine similarity between the expert features and adjusting the distribution smoothness using a learnable temperature parameter. The Top-K strategy (K=2 in the implementation) is used to select the most important experts, which are combined according to their routing weights to form the final image feature vector and text feature vector.

[0048] Step 3: Contrastive loss and negative example loss calculation The system calculates the similarity matrix between image and text features, which represents the similarity between image and text features, and adjusts the similarity distribution through the temperature parameter. Based on the similarity matrix, the bidirectional cross-entropy loss is calculated, including the loss in the image-to-text direction and the loss in the text-to-image direction, and the average of the two constitutes the basic contrastive loss.

[0049] At the same time, the system introduces a mining strategy for difficult negative samples to find the most difficult negative samples (the most similar negative samples to the positive samples) in the image-to-text direction and the text-to-image direction, respectively, and calculate the gap between their similarity and the corresponding positive samples. Compare the average of the two with the set boundary value to obtain the hard negative mining loss.

[0050] To make more effective use of hard negative example information, the system calculates the positive sample similarity mean (the average of the similarity of all matched image-text pairs) and the most difficult negative sample similarity mean (the average of the most difficult negative sample similarity of all image and text), and based on the difference between the two, the hard negative loss weight is dynamically adjusted by The function dynamically adjusts the weight of the hard negative loss. When the positive and negative samples are clearly distinguished, the weight is small, accelerating the model convergence; when the positive and negative samples are difficult to distinguish, the weight is large, enhancing the learning of difficult samples. The final total loss is the sum of the base contrast loss and the weighted hard negative loss.

[0051] Step 4: Single-modal query input In practical applications, the system receives single-modal query input, such as image query or text query. For image query, an image is input; for text query, a description text is input.

[0052] Step 5: Query feature extraction The query input is extracted by the corresponding expert encoder (image or text). For image query, multiple visual experts are used to extract features and perform dynamic routing fusion; for text query, multiple text experts are used to extract features and perform dynamic routing fusion. This ensures that the query features are consistent with the feature processing in the training phase.

[0053] Step 6: Similarity calculation and sorting The system calculates the similarity matrix between the query features and the features in the to-be-retrieved library. For image query text, the image features are calculated with all text features in the text feature library; for text query image, the text features are calculated with all image features in the image feature library. The calculation method is the dot product of the query features and the feature library features divided by the temperature parameter.

[0054] Step 7: Retrieval result output The system sorts each row of the similarity matrix to obtain the similarity ranking result of each query, and returns the top K results with the highest similarity as the final retrieval result. For image query, the most matching text description is returned; for text query, the most matching image is returned.

[0055] The entire system obtains multi-dimensional feature expression through heterogeneous expert cooperation, realizes adaptive feature fusion through dynamic expert routing, and enhances the model's ability to distinguish difficult samples through difference perception contrast learning. These three are organically combined to build an end-to-end cross-modal retrieval framework.

[0056] The above only shows the preferred embodiments of the present application, and it should be noted that for ordinary skilled persons in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, which should also be considered as the protection scope of the present application.

Claims

1. A method for aligning text and image based on a multi-modal hybrid expert model, characterized in that, include: A multimodal hybrid expert model is constructed, which includes a heterogeneous expert collaboration module, a cosine similarity dynamic routing module, and a difference-aware contrastive learning module. The heterogeneous expert collaboration module is used to extract features from images and text using multiple pre-trained visual model experts and multiple pre-trained text model experts respectively, and to map the features to a semantic space of a unified dimension through a projection layer; the cosine similarity dynamic routing module is used to calculate the importance scores of different experts based on cosine similarity to achieve adaptive feature fusion. The difference-aware contrastive learning module is used to calculate the similarity matrix between image and text features, and to train a multimodal hybrid expert model by combining basic contrastive loss and dynamically weighted hard negative example loss. Bidirectional retrieval of images to text or text to images is performed based on a multimodal hybrid expert model.

2. The method of claim 1, wherein, The visual model experts include DINO-ViT-B / 8, ConvNeXtV2-Tiny-1k-224, and SigLIP-base-patch16-256 visual encoders; the text model experts include E5-base-v2 and SigLIP text encoders.

3. The method of claim 1, wherein, In the heterogeneous expert collaboration module, the features extracted by different experts are mapped to a unified feature space and normalized through an unbiased linear projection layer.

4. The method of claim 1, wherein, In the cosine similarity dynamic routing module, the importance scores of different experts are calculated in the following manner: Calculate the cosine similarity between expert features and adjust the distribution using a temperature parameter to obtain the similarity matrix; Calculate the importance score for each expert based on the similarity matrix: wherein represents the importance score of the i th expert; is the batch size; represents the average similarity of the b th feature corresponding to the i th expert in the samples of the N th batch to the features corresponding to all experts.

5. The method of claim 4, wherein, The cosine similarity dynamic routing module, after obtaining the importance score, further includes: By The importance score is converted to a routing probability by a function: wherein represents the routing probability of the i expert, exp represents taking the exponential; Based on the routing probability, select the expert with the highest probability Weighted combination of the features of the experts to obtain a final feature representation.

6. The method of claim 1, wherein, The basic contrastive loss is a bidirectional cross-entropy loss: In the formula, wherein is the base contrastive loss, and denote the loss in the image-to-text and text-to-image directions, respectively, denotes the cross-entropy computation function, denotes the similarity matrix between image and text features, denotes the labels of the batch of data, is a learnable temperature parameter, is the image feature matrix, is the text feature matrix.

7. The method of claim 1, wherein, The hard negative example loss is: wherein is a hard negative loss, is a hyper-parameter to control the minimum gap between positive and negative samples, is a similarity matrix from image to text, is a similarity matrix from text to image, is a similarity of positive samples, denotes the gap between the similarity of the hardest negative example from image to text and the similarity of positive samples, denotes finding the maximum negative similarity for each image in the similarity matrix from image to text, denotes the gap between the similarity of the hardest negative example from text to image and the similarity of positive samples, wherein denotes finding the maximum negative similarity for each image in the similarity matrix from image to text.

8. The method of claim 7, wherein, In the difference-aware contrastive learning module, the dynamic weights of the hard negative example loss are: In the formula, wherein is a dynamic weight for hard-negative loss, denotes a function, is an initial weight for hard-negative loss, and denotes the mean of positive similarity and negative similarity, respectively.

9. A system for aligning text and image based on a multi-modal hybrid expert model, the system comprising: include: A multimodal hybrid expert model construction unit is used to construct a multimodal hybrid expert model, which includes a heterogeneous expert collaboration module, a cosine similarity dynamic routing module, and a difference-aware contrastive learning module. The heterogeneous expert collaboration module is used to extract features from images and text using multiple pre-trained visual model experts and multiple pre-trained text model experts, respectively, and to map the features to a semantic space of a unified dimension through a projection layer. The cosine similarity dynamic routing module is used to calculate the importance scores of different experts based on cosine similarity to achieve adaptive feature fusion. The difference-aware contrastive learning module is used to calculate the similarity matrix between image and text features, and to train a multimodal hybrid expert model by combining basic contrastive loss and dynamically weighted hard negative example loss. The retrieval matching unit is used for bidirectional retrieval of images to text or text to images based on a multimodal hybrid expert model.

Images