A Cross-Modal Image-Text Retrieval Method Based on an Embedded Sparse Gate Expert Hybrid Model

Abstract

This invention belongs to the field of cross-modal image and text retrieval technology, and discloses a cross-modal image and text retrieval method based on an embedded sparse gate expert hybrid model, including the following steps: Cross-modal multi-scale modeling: Multi-scale image features are extracted using the Spatial Pyramid Pooling Module (ASPP), and text features are dynamically weighted using multi-scale activation factors to achieve cross-modal multi-scale semantic alignment of images and text; Multi-scale cross-modal feature fusion: A multi-scale cross-modal router is designed to fuse image and text features through cross-attention, and cross-modal joint features are extracted at different scales using an expert network; Dual-path triplet loss calculation: A dual-path triplet loss function is used, combined with intra-scale and cross-scale constraints to optimize the feature space, achieving joint optimization at the multi-scale and cross-modal levels, and obtaining the final cross-modal retrieval result. This invention improves the accuracy and efficiency of cross-modal image and text retrieval.

Description

Technical Field

[0001] This invention belongs to the field of cross-modal image and text retrieval technology, and specifically relates to a cross-modal image and text retrieval method based on an embedded sparse gate expert hybrid model. Background Technology

[0002] Cross-modal retrieval of marine remote sensing data is a technique for associating and matching data from different modalities, aiming to achieve efficient cross-referencing of multi-source heterogeneous data. Among them, cross-modal image-text retrieval of marine remote sensing data has attracted widespread attention due to its advantages of flexible input and efficient query. Cross-modal retrieval of remote sensing images mainly includes three key processes: first, multimodal feature extraction, including visual feature extraction and text feature modeling; second, cross-modal alignment, including multimodal feature fusion and cross-modal feature alignment; and third, similarity measurement, which measures the distance between the image and text in high-dimensional space, and selects the image features and text features with the highest similarity as the best retrieval matching pair.

[0003] Currently, the mainstream approach to cross-modal image text retrieval methods is to utilize the large-scale trained CLIP model to enhance feature representation. However, this method suffers from the following drawbacks when applied to marine remote sensing images:

[0004] Firstly, the multi-scale semantic alignment capability is insufficient. Marine scenes exhibit high intra-class similarity, and existing methods rely on dilated convolution to extract multi-scale features from images. However, they lack multi-scale modeling of text modalities, leading to the loss of scale information during cross-modal alignment and causing mismatches. Furthermore, existing technologies have not established a cross-modal multi-scale alignment mechanism, making it difficult to meet fine-grained retrieval requirements and affecting retrieval accuracy in complex scenes.

[0005] Secondly, cross-modal feature fusion efficiency is low. In traditional MoE (Mixture of Experts) models, the router performs classification and assignment based solely on single-modal features, ignoring the multi-scale interaction relationships between images and text. This prevents the dynamic fusion of cross-modal local-global semantic associations, resulting in a lack of contextual consistency in the features received by the expert network. Furthermore, traditional methods lack collaborative learning mechanisms for multi-scale features, making it difficult for the model to capture implicit associations between heterogeneous data in cross-modal representation spaces.

[0006] Thirdly, the fine-grained semantic gap has not been effectively bridged. Existing pre-trained models employ coarse-grained semantic alignment, which cannot adapt to the spectral-texture-spatial multi-scale characteristics of remote sensing images. The lack of spatial relationship modeling for occluded areas widens the semantic gap, causing retrieval results to deviate from actual needs. Summary of the Invention

[0007] To address the shortcomings of existing technologies, this invention provides a cross-modal image-text retrieval method based on an embedded sparse gate expert hybrid model. It designs a cross-modal scale activation factor mechanism to model multi-scale semantic information in the text modality, achieving alignment and fusion of cross-modal multi-scale features. A multi-scale cross-modal router is constructed to ensure contextual consistency of features received by the expert network, thereby improving retrieval robustness in complex scenarios. A collaborative learning strategy using shared experts is introduced, constraining semantic consistency between multi-scale features through a triplet loss function, and combining this with global optimization by shared experts to effectively bridge the cross-modal semantic gap and improve fine-grained retrieval accuracy.

[0008] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0009] A cross-modal image-text retrieval method based on an embedded sparse gate expert hybrid model, combining multi-scale feature alignment and a sparse gate expert hybrid model, includes the following steps:

[0010] Step 1: Multimodal Data Input and Feature Extraction: A pre-trained model is used to extract initial modal features from both the image and text, resulting in the original image features E. k and original text features F k ;

[0011] Step 2, Cross-modal multi-scale modeling: Based on the original image features E obtained in Step 1 k and original text features F k The system utilizes the Spatial Pyramid Pooling (ASPP) module to extract multi-scale image features, generates multi-scale activation factors from these features, and dynamically weights the original text features F using these multi-scale activation factors. k To achieve cross-modal, multi-scale semantic alignment of text and images;

[0012] Step 3: Multi-scale cross-modal feature fusion: Design a multi-scale cross-modal router, which fuses image features at the same scale and weighted text features through cross-attention, and extracts multi-scale cross-modal joint features with multi-level semantic associations through multiple expert networks at different scales;

[0013] Step 4, Dual-path triplet loss calculation and optimization: The dual-path triplet loss function is adopted, and the feature space is optimized by combining intra-scale and cross-scale constraints to achieve joint optimization at multiple scales and cross-modal levels, thus completing the training of the model.

[0014] Step 5, Cross-modal retrieval and matching: After the model is trained, multi-scale cross-modal joint features are generated for the image or text to be retrieved. The cosine similarity between the features and the heteromodal features in the candidate set is calculated, and the top-k retrieval results are output in order of similarity.

[0015] Furthermore, step two specifically includes image multi-scale feature extraction, multi-scale activation factor generation, and text multi-scale weighted alignment, as follows:

[0016] (1) Multi-scale image feature extraction: The original image features are input into the hollow spatial pyramid pooling module ASPP, and local-global image features under multi-scale receptive field are obtained through hollow convolution with different hollow rates;

[0017] (2) Generation of multi-scale activation factors: The multi-scale image features output by ASPP are mapped to a weight matrix through learnable parameters, which is the multi-scale activation factor A;

[0018] (3) Multi-scale weighted alignment of text: The original text features are dynamically weighted by the multi-scale activation factor A, enabling multi-scale semantic modeling of text modality synchronous modeling and image matching, and finally realizing cross-modal multi-scale collaborative modeling of text and image.

[0019] Furthermore, step three, the multi-scale cross-modal feature fusion process, includes steps of cross-attention interaction, sparse gate expert assignment, and multi-scale feature extraction and aggregation, as detailed below:

[0020] (1) Cross-attention interaction: For each scale, the interaction relationship between image features and weighted text features at that scale is calculated through the cross-attention mechanism to generate scale-aware cross-modal interaction features;

[0021] (2) Sparse gate expert allocation: The multi-scale cross-modal router activates only the expert network that best matches the current scale feature based on the semantic attributes of the interaction features through the top-k sparse gating mechanism, thereby achieving accurate matching between features and experts.

[0022] (3) Scale-wise feature extraction and aggregation: After concatenating the interaction features with the original modal features, the corresponding activated expert network is input for fine-grained alignment, and the single-scale cross-modal alignment features are output. The above process is repeated to cover all scales, and finally the outputs of all expert networks are aggregated to obtain multi-scale cross-modal joint features with multi-level semantic associations.

[0023] Furthermore, the dual-path triplet loss function includes an internal loss (intra-scale constraint) and an external loss (cross-scale constraint). Joint optimization achieves multi-scale and cross-modal feature alignment. The specific technical solution is as follows:

[0024] (1) Internal loss: For cross-modal alignment features at each scale, a triplet loss is constructed to constrain the local semantic consistency of image and text features at the same scale;

[0025] (2) External loss: Input the multi-scale joint features into the shared expert network to generate shared features, and constrain the global semantic consistency across scales through triple loss;

[0026] (3) Loss fusion and model optimization: By balancing the internal and external losses through weight coefficients, the model parameters are jointly optimized, the feature alignment accuracy is iteratively improved, and the model is trained.

[0027] Furthermore, the calculation of internal losses is as follows:

[0028] For each scale t, the FFN layer of the expert network outputs the alignment features between the image and the text, which are used as input to the triplet loss function. The specific calculation formula is as follows:

[0029] ;

[0030] Where α is the boundary value, p is the positive sample, n is the negative sample, and CosSim(⋅,⋅) is the cosine similarity. For anchor point features at scale t, To and Corresponding positive sample features To and Mismatched negative sample features.

[0031] Furthermore, the calculation of external losses is as follows:

[0032] The external loss utilizes a shared expert network to globally optimize shared expert image features and shared expert text features across all scales. Image features are processed by shared experts to obtain shared expert image features, and text features are processed by shared experts to obtain shared expert text features. This generates a shared feature Z that constrains cross-scale semantic consistency. The shared feature Z is obtained by aggregating alignment features from multiple scales, as shown in the formula:

[0033] ;

[0034] Where Concat() represents the concatenation function, For anchor point features at scale t0, This represents the set of all anchor point features from the 1st scale to the tth scale;

[0035] Therefore, the formula for calculating external losses is as follows:

[0036] ;

[0037] in, For the shared features of anchor points, For anchor point Corresponding positive sample features For anchor point Mismatched negative sample features, where P is the total number of shared features;

[0038] Finally, loss fusion is performed, and the specific formula for the total loss is as follows:

[0039] ;

[0040] Where λ1 and λ2 are weighting coefficients, balancing intra-scale and cross-scale constraints.

[0041] Compared with the prior art, the advantages of this invention are:

[0042] First, it improves the accuracy of fine-grained feature extraction. The expert-led module introduces a multi-expert collaborative mechanism, decomposing text semantics and image features into sub-tasks, which are then processed separately by dedicated expert networks. For example, some experts focus on hierarchical parsing of text semantics, while others model local textures or spatial relationships in the image, achieving accurate capture of cross-modal fine-grained features. Compared to the single Transformer structure of the traditional CLIP model, this division of labor effectively alleviates the semantic confusion caused by high intra-class similarity in marine remote sensing data, significantly improving feature discrimination capabilities, especially in target recognition and semantic association under complex sea conditions.

[0043] Second, it reduces computational complexity and accelerates training. The dynamic routing module activates only a small number of experts that best match the input features through group scheduling and capacity constraints, avoiding redundant computation of all experts. For example, tokens are grouped and processed in parallel, and capacity constraints ensure load balancing, allowing the model to reduce computation by about 70% while maintaining high performance. This design directly solves the problem of high computational overhead caused by the large number of parameters in the CLIP model, especially when processing high-resolution remote sensing images, significantly shortening inference time and training cycle, and improving the feasibility of deploying the model in resource-constrained scenarios.

[0044] Third, it enhances interpretability and stability. The expert domain localization module explicitly binds each expert to a specific domain (such as texture and color) through a post-training label assignment and test-time weight fusion strategy, solving the problems of expert expertise drift and black-box selection in the MoE architecture. For example, during testing, it combines dynamic selection with a gating network and domain constraints to forcibly retain the output weights of key domain experts (such as spatial relationship experts), avoiding random fluctuations in expert capabilities and improving the interpretability of model decisions. This makes the model's response to fine-grained features more stable in marine remote sensing retrieval, while reducing the risk of performance degradation due to expert drift. Attached Figure Description

[0045] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 This is a schematic diagram of the method flow of the present invention;

[0047] Figure 2 This is a schematic diagram of the system structure of the present invention. Detailed Implementation

[0048] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0049] Example 1

[0050] Combination Figure 1 and Figure 2 As shown, the image-text cross-modal retrieval method of the present invention based on an embedded sparse gate expert hybrid model follows the classic image-text cross-modal retrieval architecture. First, it extracts multimodal features of images and text through an image input layer and a text input layer. For image data... and text data ( Figure 1 Text 1, Text 2, ..., Text 5, etc. (these are just examples), introduces text prompts. and image prompts In this embodiment, the VisionTransformer pre-trained in CLIP with k layers is used to process images, while the Transformer is used to process text, obtaining text features F. k and image features E k .

[0051] Combining multi-scale feature alignment and a sparse gate expert hybrid model, this invention includes the following steps:

[0052] Step 1: Multimodal data input and feature extraction:

[0053] Based on the feature extraction module, a pre-trained model is used to extract the initial modal features of the image and text respectively, resulting in the original image features E. k and original text features F k .

[0054] (1) Image side: The input image is encoded by a pre-trained Vision Transformer to output the original image features.

[0055] (2) Text side: The input text is encoded by a pre-trained Transformer to output the original text features, providing basic feature support for subsequent multi-scale modeling and cross-modal fusion.

[0056] Step 2: Cross-modal multi-scale modeling:

[0057] Based on the original image features E obtained in step one k and original text features F k The system utilizes the Spatial Pyramid Pooling (ASPP) module to extract multi-scale image features, generates multi-scale activation factors from these features, and dynamically weights the original text features F using these multi-scale activation factors. k This enables cross-modal, multi-scale semantic alignment of text and images.

[0058] Specifically, the steps include: multi-scale image feature extraction, multi-scale activation factor generation, and multi-scale weighted alignment of text, as follows:

[0059] (1) Multi-scale image feature extraction: The original image features are input into the hollow spatial pyramid pooling module ASPP, and local-global image features under multi-scale receptive field are obtained through hollow convolution with different hollow rates;

[0060] (2) Generation of multi-scale activation factors: The multi-scale image features output by ASPP are mapped to a weight matrix through learnable parameters, which is the multi-scale activation factor A;

[0061] (3) Multi-scale weighted alignment of text: using multi-scale activation factor A to align the original text features F k By performing scale-aware dynamic weighting, text modality synchronous modeling and image matching with multi-scale semantics are achieved, ultimately realizing cross-modal multi-scale collaborative modeling of text and images.

[0062] Step 3: Multi-scale cross-modal feature fusion:

[0063] A multi-scale cross-modal router is designed, which fuses image features at the same scale and weighted text features through cross-attention, and extracts multi-scale cross-modal joint features with multi-level semantic associations through multiple expert networks. .

[0064] Step three, multi-scale cross-modal feature fusion, includes steps such as cross-attention interaction, sparse gate expert assignment, and multi-scale feature extraction and aggregation, as detailed below:

[0065] (1) Cross-attention interaction: For each scale, the interaction relationship between image features and weighted text features at that scale is calculated through the cross-attention mechanism to generate scale-aware cross-modal interaction features;

[0066] (2) Sparse gate expert allocation: The multi-scale cross-modal router activates only the expert network that best matches the current scale feature based on the semantic attributes of the interaction features through the top-k sparse gating mechanism, thereby achieving accurate matching between features and experts.

[0067] (3) Scale-wise feature extraction and aggregation: After concatenating the interaction features with the original modal features, the corresponding activated expert network is input for fine-grained alignment, and the single-scale cross-modal alignment features are output. The above process is repeated to cover all scales, and finally the outputs of all expert networks are aggregated to obtain multi-scale cross-modal joint features with multi-level semantic associations. .

[0068] Step 4: Calculation and optimization of dual-path triplet loss:

[0069] By employing a dual-path triplet loss function and combining intra-scale and cross-scale constraint optimization of the feature space, joint optimization at multiple scales and cross-modal levels is achieved to complete the model training.

[0070] The dual-path triplet loss function includes an internal loss (intra-scale constraint) and an external loss (cross-scale constraint). Joint optimization achieves multi-scale and cross-modal feature alignment. The specific technical solution is as follows:

[0071] (1) Internal loss of dual-path loss (i.e., scale constraint): For cross-modal alignment features at each scale, construct triplet loss to constrain the local semantic consistency of image and text features at the same scale.

[0072] (2) External loss of dual-path loss (i.e. cross-scale constraint): Input multi-scale joint features into a shared expert network to generate shared features, and constrain global semantic consistency across scales through triple loss;

[0073] (3) Loss fusion and model optimization: The internal and external losses are balanced by weight coefficients (λ1, λ2), the model parameters are jointly optimized, the feature alignment accuracy is iteratively improved, and the model is trained.

[0074] Step 5: Cross-modal retrieval and matching:

[0075] After the model is trained, multi-scale cross-modal joint features are generated for the image or text to be retrieved. The cosine similarity between the joint features and the heteromodal features in the candidate set is calculated. The top-k retrieval results are then sorted by similarity and output.

[0076] The details are as follows.

[0077] (1) Cross-modal multi-scale modeling:

[0078] To address the multi-scale problem inherent in remote sensing data, existing methods can only extract multi-scale information from the image level, but cannot model the multi-scale semantics of the text modality, nor do they possess cross-modal scale alignment capabilities. Therefore, this invention proposes a cross-modal multi-scale modeling mechanism for images and text based on cross-modal scale activation factors.

[0079] First, multi-scale feature extraction is performed on the image side. This is done using the ASPP (Atrous Spatial Pyramid Pooling) module to extract image features E. k Perform multi-scale pooling operations to obtain local-global features under different receptive fields, represented as multi-scale image features E. ASPP ,Right now Figure 1 Features 1, feature 2, and feature t in the model.

[0080] Multiscale image features E ASPP The formula is as follows:

[0081] ;

[0082] Where r1, r2, ..., r t The dilation rate of the 1st, 2nd, ..., tth dilated convolutions is used to control the size of the receptive field. Different dilation rates correspond to different scales. t is the total number of scales, which is the number of different dilation rates used. Conv represents dilated convolution, and Concat represents the concatenation function, which concatenates feature maps of multiple different scales along the channel dimension.

[0083] Next is the generation of multi-scale activation factors, which converts the multi-scale image features output by ASPP. Multiscale activators are generated through specific transformations. As shown in the following formula:

[0084] ;

[0085] Where A represents the multiscale activator, E ASPP W represents the multi-scale image features output by ASPP. a and b a Here, M represents the total number of image features, and t represents the total number of scales. These are learnable parameters. The dimension representing the image features.

[0086] Let the importance weight of the i-th image feature at the t0-th scale be denoted as... ,in Indicates feature index, Represents the scale index (the total number of image features is N, and t is the total number of scales), all Composition of weight matrix multi-scale activation factors .

[0087] Finally, cross-modal multi-scale alignment is performed, utilizing the generated activation factor A on text features. Scale-aware weighting is applied to enable simultaneous modeling of multi-scale semantics in the text modality, ultimately achieving collaborative modeling of images and text at multiple scale levels. The specific formula is as follows:

[0088] ;

[0089] The total number of text features is N. Dimensions representing text features For multi-scale features, The weights of all features at scale t0 are... For product, For a multilayer perceptron at the t0 scale.

[0090] Traditional MoE (Mixture of Experts) models rely solely on single-modal features for classification and assignment, failing to integrate multi-scale interactive information from images and text. Therefore, this invention designs a multi-scale cross-modal router.

[0091] As a preferred implementation, the steps of multi-scale cross-modal feature fusion are as follows:

[0092] First, cross-attention computation: At one scale, the interaction relationship between image and text features is dynamically calculated through a cross-attention mechanism to generate a scale-aware cross-attention matrix. This process is actually a cross-modal interaction modeling of image and text features, and the output is the fused cross-modal feature (i.e., the interaction result of image and text features under the attention mechanism).

[0093] Taking scale t as an example, calculate the image features respectively. Text features Cross attention matrix :

[0094] ;

[0095] ;

[0096] in, , , d is the weight parameter. k This is the scaling factor.

[0097] Then, multi-scale feature extraction: the fused cross-modal interaction features (i.e., the cross-attention matrix) obtained in the previous step are extracted. The features are concatenated with the original image and text features and input into an expert network of the corresponding scale for fine-grained feature alignment. The output is a single-scale cross-modal alignment feature, which is the image and text alignment feature at the current scale.

[0098] The specific formula is as follows:

[0099] ;

[0100] ;

[0101] in, Let be the feature vector input to the t-th expert network, and let Concat be the concatenation function. This represents the output feature vector of the t-th expert network. For the mapping function of the expert network, , For expert network weight parameters, and is the bias vector of the expert network.

[0102] Finally, multi-scale feature aggregation: the above steps are repeated for other scales, and image-text alignment features for each scale are extracted through multiple expert networks. The outputs of all expert networks at all scales are aggregated to achieve deep fusion of multi-scale and cross-modal features, resulting in multi-scale joint features with multi-level semantic associations. The specific formula is as follows:

[0103] ;

[0104] Among them, O1, O2, O t For image and text alignment features at all scales 1, 2, ..., t.

[0105] Features are assigned to expert networks based on attention weights. Each expert handles cross-modal features within a specific scale range. Sparse gating expert processing is employed, using a MoE architecture containing m expert networks (FFN1-FFN). m Each expert focuses on processing features of a specific scale combination, optimizing computational efficiency through a top-k gating mechanism. FFN1-FFN m We obtain the scale 1 features, scale 2 features, ..., scale m features respectively.

[0106] To optimize multi-scale and cross-modal alignment, this invention designs a dual-path triplet loss function.

[0107] As a preferred implementation method, the loss is calculated as follows:

[0108] The dual-path triplet loss function designed in this invention includes an internal loss (intra-scale loss) and an external loss (cross-scale loss), achieving multi-scale and cross-modal feature alignment through joint optimization. The specific technical solution is as follows:

[0109] (1) Internal loss

[0110] The internal loss is used to constrain the local semantic consistency of image and text features at each scale. For each scale t, the FFN layer outputs image and text alignment features, which are used as input to the triplet loss function. The specific formula for calculating the internal loss is as follows:

[0111] ;

[0112] Where α is the boundary value, p is the positive sample, n is the negative sample, and CosSim(⋅,⋅) is the cosine similarity. For anchor point features at scale t, To and Corresponding positive sample features To and Mismatched negative sample features.

[0113] (2) External losses

[0114] The external loss, namely the triplet loss of the shared expert network, is used to globally optimize the shared expert image features and shared expert text features at all scales (image features are transformed into shared expert image features through shared experts, and text features are transformed into shared expert text features through shared experts) to generate shared features. To ensure cross-scale semantic consistency, shared features are obtained by aggregating aligned features across multiple scales. The formula for shared features is:

[0115] ;

[0116] Where Concat() represents the concatenation function, For anchor point features at scale t0, This represents the set of all anchor point features from the first scale (t0=1) to the t-th scale (all scales).

[0117] Therefore, the formula for calculating external losses is as follows:

[0118] ;

[0119] in, For the shared features of anchor points, For anchor point Corresponding positive sample features For anchor point Mismatched negative sample features, where P is the total number of shared features;

[0120] Finally, loss fusion is performed, and the specific formula for the total loss is as follows:

[0121] ;

[0122] Where λ1 and λ2 are weighting coefficients, balancing intra-scale and cross-scale constraints.

[0123] By combining the two types of losses according to their weights, the model can be effectively learned at both multi-scale and cross-modal levels.

[0124] This invention improves the model's accuracy in extracting features from marine remote sensing data by decomposing semantics into sub-tasks and having different expert networks work together to process them. The dynamic routing module achieves optimal matching of semantic tasks through an intelligent allocation mechanism, reducing computational complexity, significantly reducing computational load, and improving model training speed.

[0125] The present invention also includes a retrieval matching step: after the model training is completed, for the image or text to be retrieved, the image-text multi-scale joint feature similarity is calculated, and a hierarchical retrieval strategy is adopted: coarse-grained screening (top-k candidates) and fine-grained re-ranking (multi-scale feature matching) to output cross-modal retrieval results.

[0126] To verify the effectiveness of the algorithm modification, a comparative experiment was conducted on the large-scale trained model CLIP with the embedded sparse gate expert hybrid model proposed in this patent and the basic CLIP model on the public dataset RSITMD. The specific results are shown in Table 1 below.

[0127] Table 1 Comparative Experimental Results

[0128]

[0129] The experiment used R@K (K=1, 5, 10) as the evaluation index. This index, by calculating the proportion of correct answers in the top K search results, can more scientifically reflect the overall performance of the model. Data shows that the improved model proposed in this patent achieves a significant performance leap compared to the basic CLIP:

[0130] In the text retrieval image task, the improved model significantly outperformed other models across all metrics. Specifically, R@1 increased from 10.63 to 14.90, a 40.16% increase; R@5 increased from 24.78 to 40.91, a remarkable 65.09% increase; and R@10 increased from 41.81 to 57.14, a 36.67% improvement. These results demonstrate that when locating target images based on text descriptions, the improved model not only significantly increases the probability of first-hit accuracy but also maintains a higher correct answer coverage even when expanding to the top 5 or top 10 results, validating its superiority in text-image cross-modal matching.

[0131] In the image-to-text retrieval task, the improved model also demonstrated targeted advantages: the R@1 metric increased from 11.51 to 17.46, an improvement of 51.69%, meaning that the model's initial detection accuracy in "image-to-text retrieval" was significantly optimized; the R@5 metric increased slightly from 34.69 to 37.02, maintaining a stable performance comparable to the basic model; although the R@10 metric decreased slightly, combined with the overall data, it can be seen that the improved model tends to rank the correct text results higher, which can better meet users' needs for "fast and accurate retrieval" in practical applications.

[0132] Further analysis reveals that the significant improvement in the R@1 metric is the core advantage of this patented algorithm. This metric directly reflects the model's ability to hit the correct result in the first retrieval and is closely related to the user experience in practical applications. The improved model achieves an R@1 improvement of over 40% in both retrieval tasks, indicating that its embedded sparse gate expert hybrid model structure effectively enhances the alignment accuracy of cross-modal features, making the model more robust when handling complex semantic matching.

[0133] In summary, the improved algorithm proposed in this patent outperforms the basic CLIP model across the entire RSITMD dataset, especially in text retrieval image tasks, achieving an accuracy improvement of 36.67%-65.09%. This fully demonstrates the effectiveness of the sparse gate expert hybrid model in large-scale training scenarios and provides a practical and feasible technical path for performance optimization in the field of cross-modal retrieval.

[0134] Example 2

[0135] like Figure 2 As shown, this invention also provides a cross-modal image-text retrieval system based on an embedded sparse gate expert hybrid model, used to implement the cross-modal image-text retrieval method based on an embedded sparse gate expert hybrid model as described above. The system includes:

[0136] The feature extraction module includes an image input layer and a text input layer, used to extract multimodal features from images and text;

[0137] The cross-modal multi-scale modeling module is used to generate multi-scale activation factors and weight text features through the cross-modal multi-scale activation factor mechanism;

[0138] The multi-scale cross-modal router module is used to distribute multi-scale features to the expert network through the multi-scale cross-modal router, and extract cross-modal joint features by scale through the expert network;

[0139] The loss optimization module is used to calculate the dual-path triplet loss and update the model parameters.

[0140] The cross-modal multi-scale modeling module adopts the ASPP structure, and the multi-scale cross-modal router module is implemented based on the cross-attention mechanism.

[0141] This invention utilizes the Spatial Pyramid Pooling Module (ASPP) to extract multi-scale features from image features, generating multi-scale image features; based on the multi-scale image features, multi-scale activation factors are generated and dynamically weighted text features to achieve multi-scale semantic modeling of text modalities.

[0142] By using a multi-scale cross-modal router, multi-scale features of images and text are fused and distributed to an expert network for cross-modal joint feature learning.

[0143] A dual-path triplet loss function is adopted, which combines intra-scale and cross-scale constraints to optimize the feature space, thereby achieving joint optimization at multiple scales and cross-modal levels.

[0144] After the model is trained, multi-scale cross-modal joint features are generated for the image or text to be retrieved. The cosine similarity between the joint features and the heteromodal features in the candidate set is calculated. The top-k retrieval results are then sorted by similarity and output.

[0145] The functions and data processing procedures of each module can be found in the detailed steps described in Example 1 above, and will not be repeated here.

[0146] Example 3

[0147] As another embodiment of the present invention, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, is used to implement the method described in the previous embodiment 1.

[0148] Computer-readable storage media can be non-volatile computer-readable storage media. Examples include, but are not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0149] In summary, to address the challenge of simultaneously modeling multi-scale features in marine remote sensing images with text modalities, this invention designs a cross-modal scale activation factor mechanism. By introducing a dilated convolutional pyramid pooling (ASPP) module, multi-scale local-global features of the image are extracted, and specific parameter settings are used to convert these multi-scale image features into activation factors. These activation factors quantify the importance of image features at different scales and further dynamically weight text features, enabling the text modality to simultaneously capture multi-scale semantics. This mechanism achieves, for the first time, collaborative alignment of images and text at multiple scales, effectively alleviating the mismatch problem caused by scale information loss in traditional methods and significantly improving fine-grained retrieval accuracy under complex sea conditions.

[0150] To address the shortcomings of traditional MoE routers, which rely solely on single-modal feature allocation and fail to capture cross-modal semantic relationships, this invention proposes a multi-scale cross-modal router. By calculating the interaction between image and text features, scale-specific features are generated and input into an expert network to extract image-text alignment features at the current scale. This process is repeated for all scales, ultimately aggregating multi-scale joint features through multiple expert networks. This design achieves dynamic fusion of cross-modal multi-scale features for the first time, ensuring contextual consistency of features received by the expert network and significantly improving retrieval robustness in complex scenarios.

[0151] To address the challenge of traditional triplet loss functions failing to constrain multi-scale and cross-scale semantic consistency, this invention designs a dual-path triplet loss function. First, a triplet loss function is constructed for each scale feature output by the multi-scale router to constrain intra-scale semantic consistency. Second, image and text features output by the multi-scale activation factor mechanism are input into a shared expert network to further constrain cross-scale semantic consistency. Finally, by adjusting the weight coefficients to balance the two losses, joint optimization at the multi-scale and cross-modal levels is achieved. This loss function design, for the first time, unifies intra-scale and cross-scale constraints in modeling, effectively bridging the cross-modal semantic gap in marine remote sensing data and significantly improving the accuracy and recall of fine-grained retrieval.

[0152] Of course, the above description is not intended to limit the present invention, and the present invention is not limited to the examples given above. Any changes, modifications, additions or substitutions made by those skilled in the art within the scope of the present invention should be protected by the present invention.

Claims

1. A method for image-text cross-modal retrieval based on an embedded sparse gate expert mixture model, characterized in that, In combination with the multi-scale feature alignment and sparse gate expert hybrid model, the following steps are included: Step one, multi-modal data input and feature extraction: adopt pre-training model to extract initial modal features of image and text respectively, get original image features E k and original text features F k ; Step two, cross-modal multi-scale modeling: based on step one to obtain the original image feature E k and the original text feature F k , use the empty space pyramid pooling module ASPP to extract the multi-scale image feature of the image, use the multi-scale image feature to generate a multi-scale activation factor, and use the multi-scale activation factor to dynamically weight the original text feature F k , realize cross-modal multi-scale semantic alignment of image and text; Step two specifically includes the steps of image multi-scale feature extraction, multi-scale activation factor generation, and text multi-scale weighted alignment, which are as follows: (1) Image multi-scale feature extraction: input the original image features into the Atrous Spatial Pyramid Pooling module ASPP, and obtain local-global image features under multi-scale receptive fields through different atrous convolution; (2) Multi-scale activation factor generation: map the multi-scale image features output by ASPP to a weight matrix through learnable parameters, which is the multi-scale activation factor A; (3) Text multi-scale weighted alignment: use the multi-scale activation factor A to perform dynamic weighting on the original text features for scale-aware, so as to synchronize the text modal with the multi-scale semantic of image matching, and finally realize the multi-scale collaborative modeling of cross-modal image and text; Step three, multi-scale cross-modal feature fusion: design a multi-scale cross-modal router, which fuses the same scale image features and weighted text features through cross attention, and extracts multi-scale cross-modal joint features with multi-level semantic association through multiple expert networks; The steps of multi-scale cross-modal feature fusion in step three include the steps of cross attention interaction, sparse gate expert distribution, and scale-specific feature extraction and aggregation, which are as follows: (1) Cross attention interaction: for each scale, the cross attention mechanism is used to calculate the interaction between the image features and the weighted text features at this scale, and scale-aware cross-modal interaction features are generated; (2) Sparse gate expert distribution: according to the semantic attributes of the interaction features, the multi-scale cross-modal router uses the top-k sparse gating mechanism to activate only the expert network that best matches the current scale feature, thereby realizing the precise matching of features and experts; (3) Scale-specific feature extraction and aggregation: after concatenating the interaction features and the original modal features, the corresponding activated expert network is input to perform fine-grained alignment, and single-scale cross-modal alignment features are output; repeat the above process to cover all scales, and finally aggregate the outputs of all expert networks to obtain multi-scale cross-modal joint features with multi-level semantic association; Step four, double-path triplet loss calculation and optimization: a double-path triplet loss function is used to optimize the feature space in combination with intra-scale and cross-scale constraints, realize joint optimization in multi-scale and cross-modal aspects, and complete the training of the model; Step five, cross-modal retrieval matching: after the model is trained, generate multi-scale cross-modal joint features for the image or text to be retrieved, calculate the cosine similarity between the features and the candidate features in the candidate set, and output the top-k retrieval results in order of similarity.

2. The method of claim 1, wherein the method is based on an embedded sparse gate expert mixture model. The double-path triplet loss function includes internal loss, i.e., intra-scale constraint, and external loss, i.e., cross-scale constraint, which realizes multi-scale and cross-modal feature alignment through joint optimization. The specific technical solutions are as follows: (1) Internal loss: for the cross-modal alignment features at each scale, construct a triplet loss to constrain the local semantic consistency of the image and text features at the same scale; (2) External loss: input the multi-scale joint features into the shared expert network to generate shared features, and constrain the global semantic consistency across scales through a triplet loss; (3) Loss fusion and model optimization: balance internal loss and external loss through weight coefficient, jointly optimize model parameters, iteratively improve feature alignment accuracy, and train the model.

3. The method of claim 2, wherein the method is based on an embedded sparse gate expert mixture model. The calculation of internal loss is as follows: The FFN layer of the expert network outputs the alignment features of the image and the text for each scale t, which are input into the triplet loss function as follows: ； wherein a is a boundary value, p is a positive sample, n is a negative sample, CosSim(·, ·) is a cosine similarity, is an anchor feature at scale t, is a positive sample feature corresponding to is an anchor feature at scale t, is a positive sample feature corresponding to is a negative sample feature that does not match.

4. The method of claim 3, wherein the method is based on an embedded sparse gate expert mixture model. The calculation of external loss is as follows: The external loss uses the shared expert network to globally optimize the shared expert image features and the shared expert text features of all scales. The image features are obtained through the shared expert to get the shared expert image features, and the text features are obtained through the shared expert to get the shared expert text features. The generation of shared features Z constrains the cross-scale semantic consistency. The shared features Z are obtained by aggregating multi-scale alignment features, and the formula is as follows: ； where Concat() denotes a concatenation function, is an anchor feature at scale t0, denotes a set of all anchor features from the 1st scale to the tth scale. Therefore, the external loss calculation formula is as follows: ； wherein, a shared feature of the anchor point, a positive sample feature corresponding to the anchor point a shared feature of the anchor point, a positive sample feature corresponding to the anchor point a negative sample feature not matching the anchor point, P is a total number of shared features; Finally, the loss fusion, the total loss formula is as follows: ； Where λ1, λ2 are weight coefficients, balancing the scale and cross-scale constraints.

Images