A cross-modal conditional routing multi-expert low-rank adaptation remote sensing image-text retrieval method

Abstract

The application provides a kind of cross-modal conditional routing multi-expert low rank adaptation remote sensing image-text retrieval method, it is related to computer vision and artificial intelligence technical field.The method includes: obtaining remote sensing image-text retrieval dataset, constructing retrieval model containing visual encoder, text encoder and router network, and injecting multi-expert low rank adaptation module in the linear layer of the encoder;Extract image global feature and text global feature to construct cross-modal conditional vector, get the selection weight of each expert through the router network, and modulate the output of linear layer after weighted aggregation;The total loss update module and router parameters including global contrast loss, overall uncertainty loss and overall symmetric distillation matching loss are used to freeze the main parameters of the encoder;The image to be retrieved and the candidate text are input into the trained model, and the matched text description or remote sensing image is output.In this way, the fine-grained alignment capability, retrieval robustness and generalization ability to complex scenes and long-tail semantics are effectively improved.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and artificial intelligence, and in particular to a remote sensing image retrieval method with cross-modal conditional routing and multi-expert low-rank adaptation. Background Technology

[0002] Remote sensing image-text retrieval is a crucial task in the field of intelligent remote sensing understanding. Its goal is to retrieve matching remote sensing images based on text descriptions, or to retrieve corresponding text based on remote sensing images. Compared to natural images, remote sensing images are characterized by diverse shooting perspectives, large scale variations, complex backgrounds, small differences between categories, and abstract semantic representations of targets, making cross-modal semantic alignment of images and text more challenging. Furthermore, remote sensing image-text retrieval commonly suffers from one-to-many relationships, semantic ambiguity, and annotation noise, placing higher demands on the robustness and generalization ability of retrieval models.

[0003] Currently, methods typically rely on pre-trained visual-language models and low-rank adaptation for efficient parameter fine-tuning. This involves freezing the core parameters of the pre-trained image and text encoders, injecting fixed-structure low-rank adaptation modules only into certain linear mapping layers, and finally achieving image-text feature alignment through contrastive learning at the global embedding layer. However, most existing low-rank adaptation methods treat the adapter as an independent intra-modal module, lacking a conditional mechanism to explicitly guide adapter selection and parameter aggregation using complementary image-text information. Consequently, they cannot leverage cross-modal information's conditional effect at the adaptation level to improve fine-grained alignment capabilities, resulting in poor fine-grained alignment performance. Furthermore, remote sensing image and text data often suffer from issues such as multiple reasonable descriptions for a single image, text descriptions covering only local targets, and annotation noise. Existing methods struggle to maintain stable gradient update directions when facing highly ambiguous samples during training, leading to inaccurate estimation of matching confidence for difficult samples after model convergence, poor consistency of retrieval results, and compromised retrieval robustness. Furthermore, existing methods only perform cross-modal alignment at the final embedding layer, lacking bidirectional constraints on the semantic distribution of intermediate layers. This causes the local discriminative signal to attenuate during network transmission, making it difficult to support stable retrieval of fine-grained semantics in complex scenarios and resulting in insufficient generalization ability for complex scenarios and long-tail semantics. Summary of the Invention

[0004] The purpose of this invention is to provide a remote sensing image and text retrieval method with cross-modal conditional routing and multi-expert low-rank adaptation, which solves the problems of poor fine-grained alignment capability, which affects retrieval robustness, and insufficient generalization capability for complex scenes and long-tail semantics in existing technologies.

[0005] To address the aforementioned technical problems, the embodiments of the present invention provide the following technical solutions: The first aspect of this invention provides a remote sensing image and text retrieval method with cross-modal conditional routing and multi-expert low-rank adaptation, the method comprising: Obtain the remote sensing image and text retrieval dataset, which includes image data and text data; A remote sensing image retrieval model is constructed, which includes a visual encoder, a text encoder, and a router network. A multi-expert low-rank adaptation module is injected into the linear layers of both the visual encoder and the text encoder. The multi-expert low-rank adaptation module includes multiple parallel low-rank adaptation experts, each of whom is used to generate a low-rank correction amount for the corresponding linear layer. Image and text data are input into a visual encoder and a text encoder to extract global features of the image and text, respectively. A cross-modal conditional vector is constructed based on the global features of the image and text. The cross-modal conditional vector is input into a router network to obtain the selection weights of each low-rank adaptation expert. The low-rank corrections of each low-rank adaptation expert are weighted and aggregated according to the selection weights to obtain the aggregated increment, which is used to modulate the output of the linear layer. The parameters of the multi-expert low-rank adaptation module and the router network are updated through backpropagation using the total loss, while keeping the main parameters of the visual encoder and text encoder frozen, to obtain a well-trained remote sensing image and text retrieval model; the total loss includes global contrast loss, overall uncertainty loss and overall symmetric distillation matching loss. The remote sensing image to be retrieved and the candidate text are input into the trained remote sensing image-text retrieval model to output a text description that matches the remote sensing image to be retrieved, or a remote sensing image that matches the candidate text to be retrieved.

[0006] A second aspect of the present invention provides a remote sensing image retrieval device with cross-modal conditional routing and multi-expert low-rank adaptation, the device comprising: The acquisition module is used to acquire the remote sensing image and text retrieval dataset, which includes image data and text data. The building module is used to build a remote sensing image and text retrieval model. The remote sensing image and text retrieval model includes a visual encoder, a text encoder, and a router network. A multi-expert low-rank adaptation module is injected into the linear layers of both the visual encoder and the text encoder. The multi-expert low-rank adaptation module includes multiple parallel low-rank adaptation experts. Each low-rank adaptation expert is used to generate a low-rank correction amount for the corresponding linear layer. The extraction and weighted aggregation module is used to input image data and text data into the visual encoder and text encoder to extract global features of the image and global features of the text, respectively; construct cross-modal conditional vectors based on the global features of the image and global features of the text; input the cross-modal conditional vectors into the router network to obtain the selection weights of each low-rank adaptation expert; and perform weighted aggregation on the low-rank corrections of each low-rank adaptation expert based on the selection weights to obtain the aggregation increment, which is used to modulate the output of the linear layer. The update module is used to update the parameters of the multi-expert low-rank adaptation module and the router network through backpropagation using the total loss, while keeping the main parameters of the visual encoder and text encoder frozen, to obtain the trained remote sensing image and text retrieval model; the total loss includes global contrast loss, overall uncertainty loss and overall symmetric distillation matching loss; The image and text retrieval module is used to input the remote sensing image to be retrieved and candidate text into the trained remote sensing image and text retrieval model, so as to output the text description that matches the remote sensing image to be retrieved, or output the remote sensing image that matches the candidate text to be retrieved.

[0007] A third aspect of this application provides a computer-readable storage medium including a stored program, wherein, when the program is running, it controls the device where the storage medium is located to execute the remote sensing image retrieval method of cross-modal conditional routing and multi-expert low-rank adaptation according to the first aspect or any optional embodiment of the first aspect.

[0008] Compared to existing technologies, the remote sensing image and text retrieval method based on cross-modal conditional routing and multi-expert low-rank adaptation provided by this invention acquires a remote sensing image and text retrieval dataset, which includes image data and text data; constructs a remote sensing image and text retrieval model, which includes a visual encoder, a text encoder, and a router network; injects a multi-expert low-rank adaptation module into the linear layers of both the visual encoder and the text encoder, and the multi-expert low-rank adaptation module includes multiple parallel low-rank adaptation experts, each of whom generates a low-rank correction value for the corresponding linear layer; inputs the image data and text data into the visual encoder and text encoder to extract global features of the image and text, respectively; and constructs a cross-modal conditional routing method based on the global features of the image and text. The model dynamically assigns selection weights for each low-rank adaptation expert to the router network using cross-modal conditional vectors. The low-rank corrections of each expert are weighted and aggregated based on these weights to obtain an aggregated increment, which is used to modulate the output of the linear layer. The parameters of the multi-expert low-rank adaptation module and the router network are updated via backpropagation using the total loss, while keeping the core parameters of the visual encoder and text encoder frozen, resulting in a trained remote sensing image-text retrieval model. The total loss includes global contrast loss, overall uncertainty loss, and overall symmetric distillation matching loss. The remote sensing image to be retrieved and candidate text are input into the trained model to output either a text description matching the image to be retrieved or a remote sensing image matching the candidate text. By driving the router network with cross-modal conditional vectors to dynamically assign selection weights for each low-rank adaptation expert to different image-text samples, and using the weighted aggregated increment to modulate the output of the linear layer, the remote sensing image-text retrieval model can adaptively select parameters and modulate features based on the cross-modal semantic differences of image-text samples, thereby effectively improving fine-grained alignment capabilities. By adaptively reducing the weight of highly uncertain samples in the total loss through overall uncertainty loss, the interference of ambiguous and noisy samples on gradient updates is suppressed, thereby effectively improving retrieval robustness. Furthermore, by applying a global symmetric distillation matching loss to the consistency constraint of the bidirectional semantic distribution of intermediate layer features in both the visual and text encoders, local discriminative signals are preserved during network transmission, thus effectively improving the generalization ability for complex scenes and long-tail semantics. Attached Figure Description

[0009] The above and other objects, features, and advantages of exemplary embodiments of the present invention will become readily apparent upon reading the following detailed description with reference to the accompanying drawings. In the drawings, several embodiments of the invention are illustrated by way of example and not limitation, with the same or corresponding reference numerals denoteing the same or corresponding parts, wherein: Figure 1 A flowchart illustrating a remote sensing image retrieval method based on multi-expert low-rank adaptation for cross-modal conditional routing is shown. Figure 2 The algorithmic framework of a remote sensing image retrieval method based on multi-expert low-rank adaptation for cross-modal conditional routing is illustrated schematically. Figure 3 A schematic diagram of a remote sensing image retrieval device with cross-modal conditional routing and multi-expert low-rank adaptation is shown. Detailed Implementation

[0010] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the invention and to fully convey the scope of the invention to those skilled in the art.

[0011] It should be noted that, unless otherwise stated, the technical or scientific terms used in this invention should have the ordinary meaning as understood by one of ordinary skill in the art.

[0012] The methods described in the embodiments of the present invention will be explained in detail below.

[0013] Figure 1 A flowchart illustrating a cross-modal conditional routing multi-expert low-rank adaptation remote sensing image retrieval method according to an embodiment of the present invention is shown. See [link to relevant documentation]. Figure 1 As shown, this cross-modal conditional routing multi-expert low-rank adaptation remote sensing image retrieval method may include: S101. Obtain the remote sensing image and text retrieval dataset.

[0014] The remote sensing image and text retrieval dataset includes both image data and text data.

[0015] The remote sensing image retrieval datasets include the University of California, Merced Caption Dataset (UCM Caption), the RemoteSensing Image Captioning Dataset (RSICD), and the RemoteSensing Image Text Matching Dataset (RSITMD). The UCM Caption dataset contains approximately 2100 high-resolution remote sensing images, covering 21 scene categories, with each image accompanied by 5 text descriptions. The RSICD dataset contains approximately 10921 images, covering 30 scene categories, with each image accompanied by 5 text descriptions. The RSITMD dataset is a fine-grained dataset containing approximately 4743 images, covering 32 scene categories, with each image also accompanied by 5 manually annotated texts. The UCM Caption, RSICD, and RSITMD datasets are all used for subsequent training, validation, and testing of the remote sensing image retrieval model.

[0016] After obtaining the remote sensing image-text retrieval dataset, image preprocessing and text preprocessing are required to ensure consistent input format and improve the stability of subsequent learning. Image preprocessing includes random cropping, scaling, horizontal flipping, and random enhancement operations. Text preprocessing includes lowercase conversion, punctuation normalization, word segmentation, and word count limiting operations.

[0017] Specifically, after obtaining the remote sensing image and text retrieval dataset, the method also includes: Step A1: Perform random cropping on the image data to obtain the cropping result.

[0018] Step A2: Scale the cropped result to the target size, and then horizontally flip the scaled image with a set probability to obtain the flipped image.

[0019] Step A3: Perform random enhancement on the flipped image to obtain the preprocessed image.

[0020] First, the image data is randomly cropped with a cropping scale between 0.5 and 1.0, and the aspect ratio is randomly changed during the cropping process to form local image sub-regions. Then, the cropped result is scaled to 224×224 pixels. On this basis, a horizontal flip is performed with a probability of 0.5, and further random enhancement operations are applied. The random enhancement operations can be randomly selected from operations such as automatic contrast, equalization, brightness adjustment, color dithering, and rotation transformation to enhance the model's adaptability to remote sensing images.

[0021] Step A4: Perform lowercase conversion and punctuation normalization on the text data to remove redundant characters and merge consecutive spaces to obtain a normalized text sequence.

[0022] Step A5: Segment the standardized text sequence into words and limit the number of words to a preset maximum length to generate a fixed-length text input sequence, and use the fixed-length text input sequence as the preprocessed text.

[0023] First, the original text data is converted to lowercase. Then, punctuation, spaces, and placeholders are normalized to remove redundant characters and merge consecutive spaces, resulting in a normalized text sequence. This normalized text sequence is then segmented into words, with the word count limited to a preset maximum length of 30 words, forming a fixed-length text input sequence. The preprocessed image and preprocessed text are then input into the visual encoder and text encoder, respectively, for feature extraction.

[0024] S102. Construct a remote sensing image and text retrieval model.

[0025] The remote sensing image retrieval model includes a visual encoder, a text encoder, and a router network. Multi-expert low-rank adaptation modules are injected into the linear layers of both the visual encoder and the text encoder. The multi-expert low-rank adaptation module includes multiple parallel low-rank adaptation experts, each of whom is used to generate low-rank corrections for the corresponding linear layer.

[0026] Each low-rank adaptation expert includes a dimensionality reduction matrix and an dimensionality increase matrix. The low-rank correction is determined by the dimensionality reduction matrix, the dimensionality increase matrix, and a preset scaling factor. The router network is a multilayer perceptron. The linear layer includes an attention projection layer and a fully connected multilayer perceptron (MLP) layer.

[0027] Specifically, the remote sensing image retrieval model also includes a backbone network, with the image patch size of the contrastive language-image pre-training vision transformer base model (CLIP ViT-B / 32) set at 32×32 pixels.

[0028] Without altering the overall structure of the backbone network, a multi-expert low-rank adaptation module (i.e., low-rank correction) is injected into the linear layers of the visual encoder and text encoder. Samples are then dynamically allocated among multiple experts through conditional routing, thereby enhancing cross-modal representation capabilities. The visual encoder extracts land cover categories, spatial structures, target relationships, and scene layout information from remote sensing images, while the text encoder extracts target attributes, spatial locations, object relationships, and scene semantic information from text descriptions.

[0029] The visual encoder adopts a visual Transformer architecture, and the text encoder adopts a text Transformer architecture. Using CLIP ViT-B / 32 as the backbone network, multi-expert low-rank adaptation (LoRA) modules are introduced into the attention projection layers of the visual and text Transformers, as well as the fully connected layers of the MLP. For any linear mapping, the output of the linear layer modulated by the LoRA module is represented as: ; in, For the output of the linear layer, These are the original weights of the linear layer. For the input features of the linear layer, This is a bias term.

[0030] After adding the low-rank increment, i.e., the low-rank correction, the original weights are updated as follows: ; in, For the updated weights, These are the original weights of the linear layer. This is a low-rank correction. It is a low rank value. Scaling factor For a dimension reduction matrix, For an upgraded matrix, , For the output dimension of the linear layer, For the input dimension of the linear layer, The output dimension of the linear layer With low rank value The matrix space, Low rank value Input dimension of linear layer The matrix space. The output of the updated linear layer can then be expressed as: ; in, For the output of the updated linear layer, For the updated weights, Scaling factor For a dimension reduction matrix, For an upgraded matrix, For the input features of the linear layer, This is a bias term.

[0031] Unlike a single LoRA module, this invention employs a multi-path LoRA module with low-rank adaptation, extending one adapter into multiple parallel expert branches. Let the total number of these multiple parallel low-rank adapting experts be... Each expert corresponds to an independent set of low-rank parameter pairs. Its low-rank correction is defined as: ; in, For the first The low-rank correction amount of a low-rank fitting expert. For the first Scaling factor for a low-rank adaptation expert For the first A dimensionality reduction matrix for a low-rank fitting expert For the first An upgraded matrix of low-rank fitting experts For a dimension reduction matrix, For an upgraded matrix, , The output dimension of the linear layer With low rank value The matrix space, Low rank value Input dimension of linear layer The matrix space. To ensure that the output of the backbone network is not disturbed during the initial training phase, the third expert among all experts... An upgraded matrix of low-rank adaptation experts All are initialized with zero.

[0032] S103. Input image data and text data into the visual encoder and text encoder to extract global image features and global text features respectively; construct cross-modal conditional vectors based on the global image features and global text features; input the cross-modal conditional vectors into the router network to obtain the selection weights of each low-rank adaptation expert; perform weighted aggregation of the low-rank corrections of each low-rank adaptation expert based on the selection weights to obtain the aggregation increment, which is used to modulate the output of the linear layer.

[0033] To construct a unified semantic scale modal representation, a dual-branch Transformer is used as a deterministic feature mapper for images and text. The preprocessed input images and text are projected, positionally encoded, and transformed through multiple layers of self-attention in their respective branches. The final output is a linearly transformed and normalized global vector, which serves as a stable benchmark for subsequent uncertainty handling and contrastive learning.

[0034] Specifically, image data and text data are input into the visual encoder and text encoder to extract global features of the image and global features of the text, respectively, including: The preprocessed image is divided into multiple image patches, and each patch is linearly mapped to an image patch token. A category token and a positional encoding are added to generate an image token sequence. This image token sequence is then transformed through a multi-layer Transformer by a visual encoder. The output at the position of the category token is taken, and after linear projection and normalization, the global image features are obtained. Similarly, the preprocessed text is converted into a text token sequence by a word segmenter, and a positional encoding is added to the text token sequence to generate a text token sequence. This text token sequence is then transformed through a multi-layer Transformer by a text encoder. The output at the position of the end token is taken, and after linear projection and normalization, the global text features are obtained.

[0035] The visual encoder uses a 12-layer visual Transformer CLIP ViT-B / 32, with the preprocessed image as input: ; in, The image after preprocessing. For batch size The four-dimensional tensor space of a preprocessed image with 3 channels, an image height of 224, and an image width of 224. Here, 3 represents the batch size, 224 represents the image width and height, respectively. The batch size is 32×32, and each image is divided into 7×7=49 image blocks. Each image block is linearly mapped to a 768-dimensional image block token, denoted as . , For the first Each image block token is combined with a category token and a location code to generate a sequence of 50 768-dimensional image tokens: ; in, The image token sequence is, For category tokens, The position code for the 0th position. The position code for the first position. The position code for the 49th position. For the first The location code of each position. For the first image block token, For the 49th image block token, It is a matrix space with 50 tokens and a feature dimension of 768.

[0036] After the image token sequence is transformed by a 12-layer Transformer from a visual encoder, the following is obtained: : ; in, For the first Image token sequence encoded by the Transformer layer. It is a multilayer perceptron. For linear layers, For the attention of the bulls, For the first The image token sequence after Transformer encoding. The output at the position of the category token is the final feature of the category token. After linear projection to 512 dimensions and normalization, a global representation of a single image is obtained. The final characteristic of category tokens, For the first Image features encoded by the Transformer layer It is a 768-dimensional vector space.

[0037] The text encoder uses a 12-layer Transformer CLIP text encoder with a vocabulary of approximately 49,408 words and a maximum sequence length of 77. The preprocessed text is then converted into an initial text tag sequence by a tokenizer. This text tag sequence contains start tags, end tags, and padding tags, with a sequence length of 77 and an embedding dimension of 512. The expression for the text tag sequence is: ; in, For the initial text tag sequence, Embed the text token for the first position. Embed the text token for the second position. Embed the text token at position 77. The position code for the first position. The position code for the 77th position. It is a two-dimensional matrix space containing 77 tokens, each with a feature dimension of 512. The initial text token sequence contains start / end / padding characters.

[0038] Position encoding is added to the text token sequence to generate a text token sequence; the text token sequence is then transformed through a 12-layer Transformer layer of a text encoder, and the output at the position of the end token is taken. After linear projection to 512 dimensions and normalization, a global representation of the text is obtained.

[0039] Specifically, the cross-modal condition vector is input into the router network to obtain the selection weights of each low-rank adaptation expert, including: Step B1: Input the cross-modal condition vector into the router network to output the original routing scores of each low-rank adaptation expert based on the cross-modal condition vector.

[0040] Step B2: Score the original routes and normalize them using the Softmax function under temperature coefficient control to obtain the selection weights of each low-rank adaptation expert.

[0041] For each input sample, construct a cross-modal condition vector. Cross-modal conditional vectors can be composed of global image features, global text features, and hierarchical uncertainty information, or they can be obtained from a joint image-text representation through a linear mapping. The cross-modal conditional vectors... Input a lightweight router network to obtain the original route scores for each low-rank adaptation expert. The original route scores reflect the degree of matching between the current sample and each low-rank adaptation expert. Specifically, let the first... A low-rank adaptation expert's routing score : ; in, For cross-modal conditional vectors The router network used as input can be a single-layer or two-layer MLP. These are the parameters of the router network. The original route scores are normalized using the softmax function to obtain the selection weights of each low-rank adaptation expert. ; in, For the first The selection weights of low-rank fitting experts For the first Router scoring by a low-rank adaptation expert. The total number of multiple parallel low-rank adaptors. For multiple parallel low-rank adaptors, the first Routing scores from a low-rank adaptation expert index. It is a low rank value. This is the routing temperature coefficient, used to control the smoothness of the expert selection distribution. When the temperature coefficient is large, the distribution of experts is relatively even, and multiple experts can participate together; route temperature coefficient When the number of experts is small, the distribution of experts tends to be sparse, and the model tends to select a small number of highly relevant experts.

[0042] In a preferred embodiment, a Top-k routing strategy can also be adopted, which retains only the experts corresponding to the top k weights with the largest weight values, and renormalizes the retained top k weights to form a sparse expert selection mechanism.

[0043] Specifically, the low-rank correction values ​​of each low-rank fitting expert are weighted and aggregated according to the selection weights to obtain the aggregated increment, including: Step C1: Normalize the selection weights of each low-rank fitting expert to obtain the normalized expert selection weights.

[0044] Step C2: Select the top k weights with the largest values ​​from the normalized expert selection weights, reset the unselected expert weights to zero, and re-normalize the top k weights to obtain the sparsified expert selection weights.

[0045] The expression for the sparsified expert selection weights is: ; in, For the first The sparsified expert selection weights of a low-rank fitting expert. For the first Normalized expert selection weights for low-rank fitting experts For multiple parallel low-rank adaptors, the first The selection weights of a low-rank adaptive expert index. To select experts after normalization, the set of experts corresponding to the top k weights with the largest values ​​is chosen. To reset the rights of unselected experts to zero.

[0046] Step C3: Based on the sparsified expert selection weights, the low-rank corrections of each low-rank fitting expert are weighted and summed to obtain the aggregated increment.

[0047] The expression for the aggregation increment is: ; in, For aggregated increments, The total number of multiple parallel low-rank adaptors. For the first The selection weights of low-rank fitting experts For the first The low-rank correction amount of a low-rank fitting expert.

[0048] The aggregated increment is applied to the original weights of the linear layer. ,get: ; in, For the output of the linear layer, These are the original weights of the linear layer. For the input features of the linear layer, For aggregated increments, This is a bias term.

[0049] In this way, the routing weights determine the contribution ratio of each expert to the current sample, enabling the model to dynamically call different experts to complete the adaptation based on cross-modal content differences and semantic granularity differences.

[0050] S104. Using the total loss, update the parameters of the multi-expert low-rank adaptation module and the router network through backpropagation, while keeping the main parameters of the visual encoder and text encoder frozen, to obtain the trained remote sensing image and text retrieval model.

[0051] The total loss includes global comparison loss, overall uncertainty loss, and overall symmetric distillation matching loss.

[0052] The uncertainty modeling in step S104, which drives the adaptive scaling of the training loss with uncertainty metric, is not simply adding noise for data augmentation. This reduces the overfitting penalty for inherently ambiguous samples and enhances overall robustness in the retrieval task.

[0053] Specifically, the formation process of overall uncertainty loss includes: Step D1: Extract visual features from several pre-selected intermediate layers in the visual encoder and text encoder.

[0054] Step D2: Perform global pooling on the visual features of each intermediate layer to obtain the pooled representation, and calculate the batch-level mean and standard deviation of the pooled representation. Use the batch-level mean and standard deviation as the corresponding uncertainty statistics.

[0055] Step D3: Based on the uncertainty statistics, sample the Gaussian factor and offset, and use the Gaussian factor and offset to perturb the pooling representation to construct enhanced features.

[0056] Several intermediate layers pre-selected for visual and text networks Each is represented by its in-batch pooling: ; in, For the first Pooling representation of layers, For the first The complete characteristics of the layer For global average pooling, For batch size The feature degree of each sample is The dimensional matrix space.

[0057] Computational pooling representation of batch-level mean and standard deviation This serves as a measure of the current semantic consistency and sample ambiguity at this layer. This uncertainty statistic directly reflects the consistency of samples at the same level within a batch: standard deviation. A higher Gaussian factor indicates a more dispersed semantic distribution or fuzzy alignment within that layer. Based on the aforementioned uncertainty statistics, a controllable Gaussian factor is used for sampling. With offset Constructing Enhanced Features : ; in, For the first Enhanced features of the layer Gaussian factor, This is the offset. For the first Pooling representation of layers.

[0058] The perturbation is dynamically generated layer by layer and batch to simulate semantic uncertainty rather than blind perturbation, ensuring that the perturbation amplitude naturally amplifies or shrinks with hierarchical statistics.

[0059] Step D4: Calculate the uncertainty loss corresponding to each intermediate layer based on the standard deviation of the enhanced features.

[0060] Among them, the contribution of uncertainty loss to the total loss decreases as the standard deviation of the enhanced feature increases.

[0061] Step D5: Combine the uncertainty losses corresponding to several intermediate layers to obtain the overall uncertainty loss.

[0062] For the The layer-specific uncertainty loss is defined as follows: ; in, For the first Uncertainty loss of the intermediate layer. For the first Enhanced features of the layer For the first The standard deviation of the enhancement features of the layer For the first Text features of the layer The expected value of the standard deviation. To contrast with cross-entropy (without temperature), this design has a dual effect by enhancing the standard deviation of the post-features: when sample pairs are significantly dispersed at this layer (high uncertainty), the factor... The weight of this layer in the total loss is automatically reduced to avoid forced one-to-one constraints on noisy or ambiguous samples; when samples are significantly consistent in this layer (low uncertainty), the scaling factor enhances the contrast constraint to refine the alignment. A second regularization term prevents the standard deviation from being amplified to circumvent constraints. Finally, the pre-selected set of intermediate layers is processed. (like The average of these values ​​yields the overall uncertainty loss. : ; in, For overall uncertainty loss, For the first Uncertainty loss at the layer.

[0063] The overall uncertainty loss, together with the global contrast loss and the symmetric distillation matching loss, constitutes the optimization objective. It works in conjunction with the phased low-rank adaptive training to gradually improve the model's matching robustness at different semantic granularities.

[0064] The cross-modal conditional vector-driven routing mechanism and the uncertainty loss work synergistically at the training objective level. The routing mechanism selects appropriate low-ranking expert combinations for different image and text samples based on the cross-modal conditional vectors to adapt parameter fitting capabilities under different semantic scenarios. The uncertainty loss reduces the weight of highly uncertain samples in the total loss, i.e., it suppresses the negative gradient of ambiguous samples through loss reweighting, thereby reducing the interference of noisy samples on expert assignment from an optimization perspective. Since both share the same backpropagation process, the router's expert assignment strategy, constrained by the uncertainty loss, gradually tends to select more explicit expert combinations for stable samples and retain smoother fitting paths for highly uncertain samples, thus avoiding overfitting to a small number of outlier samples.

[0065] Specifically, the formation process of overall symmetric distillation matching loss includes: Step E1: Extract visual and text features from several pre-selected intermediate layers in the visual encoder and text encoder.

[0066] Step E2: Normalize the visual and text features of each intermediate layer to obtain normalized visual and text features.

[0067] Step E3: Project the normalized visual features onto the semantic space of the text features through a linear mapping to obtain the projected visual features.

[0068] Step E4: Based on the projected visual features and the normalized text features, calculate the similarity matrices from image to text and from text to image, respectively.

[0069] Step E5: Normalize the similarity matrix to obtain the bidirectional probability distributions in both the image-to-text and text-to-image directions.

[0070] Step E6: Calculate the symmetric divergence between the two-way probability distributions, and use the symmetric divergence as the symmetric distillation matching loss for each intermediate layer.

[0071] Step E7: Combine the symmetric distillation matching losses of several intermediate layers to obtain the overall symmetric distillation matching loss.

[0072] To ensure consistency in semantic representation between the visual encoder and the text encoder across different network depths, a symmetric distillation matching loss is applied to constrain the intermediate layer features. Unlike alignment performed only on the final global embedding, this step extends the alignment mechanism to several preset layers, enabling the model to simultaneously learn local semantic correspondences and global semantic consistency, thereby improving the stability of cross-modal retrieval in both fine-grained and coarse-grained scenarios.

[0073] Extract visual and text features from several pre-selected intermediate layers of the visual encoder and text encoder. Let the visual encoder and text encoder be at the th... The features output by the layers (intermediate layers) are denoted as follows: , For the first The visual features output by the layer. For the first The text features output by the layer, where This indicates the number of layers involved in the alignment. First, L2 normalization is performed on the visual and textual features of each intermediate layer. When the features are three-dimensional tensors, mean normalization is also performed on the batch dimension to eliminate amplitude differences caused by different sample sizes. Then, the normalized visual features are mapped using a linear mapping matrix. Projecting onto a semantic space consistent with text features yields: ; in, For the first Visual features of the layer after projection For the first The normalized visual features of the layers For the first Text features of the layer For the first Normalized text features of the layer Let be the dimension of the linear mapping matrix.

[0074] Based on this, similarity matrices are constructed in both the image-to-text and text-to-image directions: ; in, For the first The image-to-text similarity matrix of the layer. For the first Visual features of the layer after projection For the first Text features of the layer For the first The text-to-image similarity matrix of the layer. This is a temperature coefficient used to adjust the smoothness of the distribution. Then, a softmax operation is performed row-by-row on the similarity matrix to obtain the bidirectional probability distribution: ; in, For the first The probability distribution of the image-to-text direction in the layer. For the first The image-to-text similarity matrix of the layer. For the first The probability distribution of the text-to-image direction in the layer. For the first The text-to-image similarity matrix of the layer. It is a normalized exponential function.

[0075] Furthermore, to enhance the bidirectional consistency constraint, the symmetric Kullback-Leibler (KL) divergence definition is adopted. Distillation matching loss of layers: ; ; ; in, For the first The KL divergence from the image to the text direction of the layer. Used as the separator for KL divergence. For the first The probability distribution of the image-to-text direction in the layer. For the first The probability distribution of the text-to-image direction in the layer. For batch size, For sample index, For the first The first layer The probability value of an image to text for a given sample. For the first The first layer The probability value of text to image for each sample. It is the natural logarithm. For the first KL divergence from text to image direction in a layer. For the first Symmetrical distillation matching loss of the layers Text-to-image. Image-to-text.

[0076] After averaging the symmetric distillation matching losses of each intermediate layer, the overall symmetric distillation matching (SDM) loss is obtained: ; in, For overall symmetrical distillation matching loss, The number of layers for a pre-selected set of intermediate layers, For the first Layer-specific symmetric distillation matching loss. Global symmetric distillation matching loss. The ultimate goal of synthesizing the main contrast loss is to promote consistency between the visual / text encoder at the local level.

[0077] After obtaining the normalized image global embedding and text global embedding, a cross-modal contrastive learning objective is constructed at the batch level to constrain the relative positional relationships of image-text paired samples in the shared embedding space. Let the image global embedding and text global embedding constitute the image feature matrix and text feature matrix, respectively: ; in, For image feature matrix, This is the global batch size. The embedding dimension of the feature vector. Let be a two-dimensional matrix space containing the global batch size and the embedding dimension of the feature vectors. This is a text feature matrix. It requires that the row indices of the image features and their corresponding text features in the matrix be consistent to establish correct positive sample matching relationships.

[0078] Based on this, a cross-modal similarity matrix is ​​constructed by multiplying the image feature matrix by the transpose of the text feature matrix: ; in, This is a cross-modal similarity matrix, specifically the similarity matrix between the image-to-text direction and the text-to-image direction. This is the transpose of the text feature matrix. For learnable or configurable temperature parameters, Let the matrix be a two-dimensional matrix with rows and columns equal to the global batch size. Then, the cross-entropy loss is calculated in both the image-to-text and text-to-image directions: ; ; in, The image-to-text cross-entropy loss is the loss in the image-to-text direction. Equivalent to the first The image and the first Cosine similarity between texts For the first The image and the first Does each text match the actual matching tag? For the first The image and the first Elements of the similarity matrix between texts The cross-entropy loss is calculated in the text-to-image direction. For the first The image and the first Elements of the similarity matrix between texts.

[0079] The mean of the two cross-entropy losses constitutes the final global contrastive loss: ; in, This represents the loss for global comparison.

[0080] Based on this, the uncertainty loss and the overall symmetric distillation matching loss are combined and summed to form the total loss for joint optimization, resulting in the expression for the total loss: ; in, For the total loss, For global comparison loss, For overall uncertainty loss, This represents the overall symmetric distillation matching loss.

[0081] S105. Input the remote sensing image to be retrieved and the candidate text into the trained remote sensing image-text retrieval model to output a text description that matches the remote sensing image to be retrieved, or output a remote sensing image that matches the candidate text to be retrieved.

[0082] Figure 2The schematic diagram illustrates the algorithmic framework of a cross-modal conditional routing multi-expert low-rank adaptation remote sensing image retrieval method. (See attached diagram.) Figure 2 As shown, the algorithm framework includes a dual-encoding feature extraction stage, a conditional routing low-rank adaptation stage, and a joint training and retrieval output stage. In the dual-encoding feature extraction stage, the input remote sensing image, after image normalization and image segmentation, is input to a visual encoder with frozen subject parameters to obtain image features, global features, and intermediate layer features. The input text, after text segmentation, positional encoding, and serialization, is input to a text encoder with frozen subject parameters to obtain text features, global features, and intermediate layer features. In the conditional routing low-rank adaptation stage, a cross-modal conditional vector is constructed based on image and text features. This cross-modal conditional vector includes image-text interaction features and hierarchical uncertainty statistics. The cross-modal conditional vector is input to a lightweight router to generate selection weights for each low-rank adaptation expert. The top few experts are selected from the low-rank expert database, and the low-rank increments of these experts are weighted and aggregated to obtain an aggregated increment. The aggregated increment is used for conditional adaptation on the image and text sides, respectively, modulating the output of the linear layers in the visual encoder and text encoder. In the joint training and retrieval output phase, a hierarchical bidirectional distribution distillation method is used to constrain the consistency of intermediate layer features, calculating the dispersion of these features to constrain consistency in both the image-to-text and text-to-image directions. Global contrastive learning is used to constrain matching samples to be close together and non-matching samples to be far apart. Based on these training constraints, a low-rank expert, router, and mapping parameters are jointly trained. After training, similarity is calculated and ranked based on image embeddings and text embeddings, outputting either image-to-text retrieval results or text-to-image retrieval results.

[0083] This invention first acquires a remote sensing image-text retrieval dataset and performs unified preprocessing. The preprocessed images and text are then input into the CLIP visual encoder and text encoder, respectively, to obtain corresponding cross-modal feature sequences. The core of this method lies in employing a multi-expert conditional routing low-rank adaptation mechanism guided by cross-modal information. While freezing the backbone parameters, a multi-expert LoRA adapter is injected into the key linear layer of the Transformer. Expert weights are dynamically allocated based on the joint semantic information of the image and text through a conditional routing network, achieving sample-level adaptive parameter selection. During training, uncertainty modeling is further introduced, applying controllable perturbations to the intermediate layer visual features and performing adaptive loss weighting to improve the model's robustness to semantic ambiguity and one-to-many matching scenarios. Simultaneously, symmetric distillation matching loss is combined to constrain the bidirectional distribution consistency between image-to-text and text-to-image, and jointly optimized with the global contrastive learning loss. After model convergence, the cosine similarity between image features and text features is calculated and sorted in descending order of similarity. The most relevant text description or image result is returned, thus achieving high-precision and robust remote sensing image-text retrieval.

[0084] Unlike existing low-rank adaptation methods that typically employ a single adapter, this invention introduces a multi-expert parallel LoRA and conditional routing mechanism within a unified training process. This enables the model to adaptively select parameters based on the semantic differences between different image-text samples. Specifically, based on a dual-encoder vision-language pre-trained model, this invention injects LoRA modules into the attention projection layer and key linear layer of the MLP in both the vision and text Transformers. It freezes the backbone network parameters and updates only the low-rank adapter parameters and routing-related parameters, thereby significantly reducing the size of training parameters and storage overhead while maintaining efficient transfer capabilities for cross-modal semantics in remote sensing. By inputting global image features, global text features, and their cross-modal consistency information into the routing network, expert weights are dynamically generated, and multiple LoRA experts are weighted and aggregated. This allows the model to achieve stronger adaptability and discriminative capabilities in remote sensing retrieval tasks with large differences in ground feature scale, inconsistent descriptive granularity, and complex scene semantics.

[0085] This invention further couples uncertainty modeling with cross-modal alignment constraints to improve retrieval robustness and result reliability. During training, a controllable noise perturbation related to the feature distribution is applied to the intermediate layer visual features, and the loss term is adaptively weighted according to the degree of feature fluctuation. This allows the model to avoid over-penalizing uncertain samples when facing semantic ambiguity, one-to-many matching, annotation noise, and sample distribution shifts, thus improving tolerance for difficult and weakly supervised samples. Simultaneously, a bidirectional symmetric distillation matching constraint is introduced to impose consistency constraints on the similarity distribution in both image-to-text and text-to-image directions, promoting collaborative alignment between visual and text representations at both the local and global levels. Therefore, this invention effectively mitigates the risk of overfitting under limited remote sensing data conditions, enhancing the stability, generalization, and semantic alignment accuracy of the cross-modal retrieval system in complex scenarios.

[0086] To verify the applicability and generalization ability of the method of this invention under different data scales and semantic granularities, this embodiment selects three representative publicly available remote sensing image retrieval datasets for training and testing: the UCMCaption dataset, the RSITD dataset, and the RSICD dataset. These datasets cover a variety of application scenarios, from small to large scale and from coarse-grained scene description to fine-grained semantic characterization, and can comprehensively reflect the performance boundaries of the model in remote sensing image retrieval tasks. The UCM Caption dataset contains 2100 high-resolution remote sensing images covering 21 land use and cover scenarios. Each image is accompanied by 5 manually written text descriptions, which mainly include land cover types, spatial layouts, and scene attributes. The RSITD dataset contains 4743 images covering 32 scenarios. Each image is accompanied by 5 text descriptions, focusing on object attributes, state changes, and relationships between objects. It is suitable for testing the model's ability to model fine-grained semantic information. The RSICD dataset contains 10921 images covering 30 scenarios. Each image provides 5 descriptive texts, making it suitable for validating the model's learning ability, generalization ability, and stability under large-scale data conditions.

[0087] On the three datasets mentioned above, this embodiment performs bidirectional evaluation of image-to-text retrieval and text-to-image retrieval to measure the model's bidirectional retrieval performance in cross-modal retrieval. In implementation, a CLIP-based dual encoder framework is used as the overall backbone network. The visual branch adopts a ViT-B / 32 structure, with input images uniformly scaled to 224×224 pixels and encoded in 32×32 patch sizes. The text branch uses the CLIP text encoder to perform word segmentation, positional encoding, and Transformer encoding on the input text sequence to obtain the corresponding semantic representation. To improve the efficiency of parameter fine-tuning, this embodiment introduces a multi-expert low-rank adaptation module in the key linear mapping layers of the encoder, especially the projection and feedforward linear mapping positions in the attention module. The multi-expert low-rank adaptation module contains multiple parallel low-rank adaptation experts and adopts a cross-modal information-guided conditional routing mechanism in its configuration. Sample adaptive parameter adjustment is achieved through multi-expert parallel modeling and conditional routing selection. Specifically, multiple low-rank adaptation experts are set up. The router network dynamically selects experts and assigns selection weights based on cross-modal condition vectors. A Top-k routing strategy is used to sparsify the selection weights, resetting the weights of unselected experts to zero and renormalizing the selection weights of selected experts. The low-rank correction values ​​of each expert are weighted and aggregated according to the sparsified selection weights to obtain the aggregated increment, which is used to modulate the output of the linear layer. To ensure the stability of the backbone network behavior in the early stage of training, the up-dimensionality matrix of each low-rank adaptation expert is initialized with zero, and the router network controls the smoothness of its selection distribution through a temperature coefficient.

[0088] During training and inference, this embodiment inputs samples into the model in batches, with the batch size set to 64 for both the training and testing phases to maintain consistency in gradient estimation and evaluation. The optimizer used is AdamW, with a weight decay coefficient set to 0.04 to suppress overfitting and improve the model's generalization performance. A linear decay strategy is employed for learning rate scheduling, combined with a warmup mechanism. The first 10% of the total training steps are used as a warm-up phase to reduce initial parameter updates and improve training stability. The initial learning rate is set to 4.0 × 10⁻⁶. -4 Simultaneously, a learning rate amplification factor is applied to some parameter groups to enhance the learning speed of specific modules. The total number of training epochs can be set according to the dataset size; for example, 100 batches can be used for the UCM Caption dataset, 50 batches for the RSITD dataset, and 30 batches for the RSICD dataset. The specific training cycle can be further adjusted according to the experimental design. The entire experiment was conducted using the Python and PyTorch deep learning frameworks, and training and inference were performed on a computing server equipped with a GTX4090.

[0089] In the evaluation phase, this embodiment calculates the results of image-to-text retrieval and text-to-image retrieval on the corresponding test set, and uses retrieval recall R@K as a quantitative evaluation index, where K is the number of truncations in the retrieval results, and K takes values ​​of 1, 5, and 10. For image-to-text retrieval tasks, R@K represents the probability that the correctly matched text appears in the returned results corresponding to the first K retrieval results when an image is used as a query; for text-to-image retrieval tasks, R@K represents the probability that the corresponding image appears in the candidate results corresponding to the first K retrieval results when a text is used as a query. Furthermore, to comprehensively reflect the overall performance of the model in different retrieval directions, R@1, R@5, and R@10 can be calculated separately for text-to-image retrieval and image-to-text retrieval, and the average of the two can be taken as the average recall index, which serves as a single-value index of the model's overall retrieval performance. This index can simultaneously characterize the model's exact matching ability, lenient retrieval ability, and top-ranking coverage ability, making it suitable for comprehensive comparison of remote sensing image-text retrieval tasks.

[0090] The advantage of this invention lies in its realization of conditional adaptation and hierarchical feature fusion guided by cross-modal information. This invention first extracts image embeddings in the visual branch and then passes them as cross-modal context to the low-rank adaptation module and attention projection module in the text branch. By modulating the intensity of the adaptation increment with lightweight context signals, it achieves explicit guidance of the text-side adaptation process by image information. This method employs a fixed-rank LoRA structure and combines it with a multi-expert conditional routing mechanism to perform Top-k selection among preset experts. This enhances the modeling ability for multi-scale targets, complex scenes, and cross-modal semantic differences in remote sensing images without changing the main parameters of the basic model.

[0091] The explicit uncertainty regularization of this invention improves robustness and reliability. This invention introduces uncertainty loss for intermediate features at specified levels. By applying noise enhancement to visual or textual intermediate features and combining this with batch-level contrast constraints between enhanced and original samples, feature uncertainty is estimated and utilized. During training, for levels with high uncertainty, this loss enhances the model's tolerance to perturbed samples, semantically ambiguous samples, and one-to-many matching samples; for levels with low uncertainty, normal constraint strength is maintained to stably learn high-confidence semantic representations. This mechanism, jointly optimized with cross-modal contrastive learning, helps improve the model's robustness and generalization ability in remote sensing retrieval scenarios.

[0092] The hierarchical symmetric distillation method of this invention promotes local consistency and global alignment. This invention extracts intermediate features from multiple Transformer layers of the visual and text encoders, and then normalizes, linearly maps, and performs bidirectional distribution matching on these features. By constructing symmetric KL divergence constraints for each participating layer, the similarity distribution between image-to-text and text-to-image is kept consistent, thereby progressively transferring local hierarchical semantic consistency to the final global embedding space. This hierarchical distillation mechanism can reduce the semantic shift caused by relying solely on final layer contrast loss, improving the alignment effect and retrieval accuracy of the adapter at intermediate semantic levels.

[0093] The method of this invention has extremely high parameter efficiency and training stability. Following the principle of efficient parameter fine-tuning, it only updates the injected LoRA and a small number of control parameters, and the total number of trainable parameters is about 2.7M (far less than the 151M of full fine-tuning), which significantly reduces computational overhead, storage requirements and overfitting risk, and facilitates lightweight deployment in resource-constrained environments.

[0094] Based on the above Figure 1As can be seen from the implementation method, the embodiments of the present invention acquire a remote sensing image and text retrieval dataset, which includes image data and text data; construct a remote sensing image and text retrieval model, which includes a visual encoder, a text encoder, and a router network; inject a multi-expert low-rank adaptation module into the linear layers of both the visual encoder and the text encoder, the multi-expert low-rank adaptation module includes multiple parallel low-rank adaptation experts, each of whom is used to generate a low-rank correction amount for the corresponding linear layer; input the image data and text data into the visual encoder and the text encoder to extract global features of the image and global features of the text, respectively; construct a cross-modal conditional vector based on the global features of the image and global features of the text; and input the cross-modal conditional vector into the visual encoder and the text encoder. The selection weights of each low-rank adaptation expert are obtained by inputting the router network. The low-rank correction values ​​of each low-rank adaptation expert are then weighted and aggregated according to the selection weights to obtain the aggregated increment, which is used to modulate the output of the linear layer. The parameters of the multi-expert low-rank adaptation module and the router network are updated through backpropagation using the total loss, while keeping the main parameters of the visual encoder and text encoder frozen, resulting in a trained remote sensing image-text retrieval model. The total loss includes global contrast loss, overall uncertainty loss, and overall symmetric distillation matching loss. The remote sensing image to be retrieved and candidate text are input into the trained remote sensing image-text retrieval model to output either a text description matching the remote sensing image to be retrieved or a remote sensing image matching the candidate text to be retrieved. By driving the router network with cross-modal conditional vectors, selection weights of each low-rank adaptation expert are dynamically assigned to different image and text samples. The weighted aggregated increment is used to modulate the output of the linear layer, enabling the remote sensing image-text retrieval model to adaptively select parameters and modulate features based on the cross-modal semantic differences of image and text samples, thereby effectively improving fine-grained alignment capabilities. By adaptively reducing the weight of highly uncertain samples in the total loss through overall uncertainty loss, the interference of ambiguous and noisy samples on gradient updates is suppressed, thereby effectively improving retrieval robustness. Furthermore, by applying a global symmetric distillation matching loss to the consistency constraint of the bidirectional semantic distribution of intermediate layer features in both the visual and text encoders, local discriminative signals are preserved during network transmission, thus effectively improving the generalization ability for complex scenes and long-tail semantics.

[0095] Based on the same inventive concept, as an implementation of the above-mentioned remote sensing image and text retrieval method with cross-modal conditional routing and multi-expert low-rank adaptation, this embodiment of the invention also provides a remote sensing image and text retrieval device with cross-modal conditional routing and multi-expert low-rank adaptation. Figure 3 This is a structural diagram of the remote sensing image retrieval device with cross-modal conditional routing and multi-expert low-rank adaptation in an embodiment of the present invention. See also... Figure 3 As shown, the device may include: The acquisition module 301 is used to acquire the remote sensing image and text retrieval dataset, which includes image data and text data. Module 302 is used to build a remote sensing image and text retrieval model. The remote sensing image and text retrieval model includes a visual encoder, a text encoder, and a router network. A multi-expert low-rank adaptation module is injected into the linear layers of both the visual encoder and the text encoder. The multi-expert low-rank adaptation module includes multiple parallel low-rank adaptation experts. Each low-rank adaptation expert is used to generate a low-rank correction amount for the corresponding linear layer. The extraction and weighted aggregation module 303 is used to input image data and text data into the visual encoder and text encoder to extract global features of the image and global features of the text, respectively; construct a cross-modal conditional vector based on the global features of the image and global features of the text; input the cross-modal conditional vector into the router network to obtain the selection weights of each low-rank adaptation expert; and perform weighted aggregation on the low-rank corrections of each low-rank adaptation expert based on the selection weights to obtain the aggregation increment, which is used to modulate the output of the linear layer. Update module 304 is used to update the parameters of the multi-expert low-rank adaptation module and the router network through backpropagation using the total loss, while keeping the main parameters of the visual encoder and text encoder frozen, to obtain the trained remote sensing image and text retrieval model; the total loss includes global contrast loss, overall uncertainty loss and overall symmetric distillation matching loss; The image and text retrieval module 305 is used to input the remote sensing image to be retrieved and the candidate text into the trained remote sensing image and text retrieval model, so as to output the text description that matches the remote sensing image to be retrieved, or output the remote sensing image that matches the candidate text to be retrieved.

[0096] The device may further include: a preprocessing module, used to randomly crop the image data after acquiring the remote sensing image retrieval dataset to obtain a cropping result; scale the cropping result to a target size and horizontally flip the scaled image with a set probability to obtain a flipped image; perform random enhancement operations on the flipped image to obtain a preprocessed image; perform lowercase conversion and punctuation normalization on the text data to remove redundant characters and merge consecutive spaces to obtain a normalized text sequence; segment the normalized text sequence into words and limit the number of corresponding words to a preset maximum length to generate a fixed-length text input sequence, and use the fixed-length text input sequence as the preprocessed text.

[0097] In the extraction and weighted aggregation module 303, the cross-modal condition vector is input into the router network to obtain the selection weights of each low-rank adaptation expert. This includes: inputting the cross-modal condition vector into the router network to output the original route scores of each low-rank adaptation expert based on the cross-modal condition vector; and normalizing the original route scores using the Softmax function under temperature coefficient control to obtain the selection weights of each low-rank adaptation expert.

[0098] In the extraction and weighted aggregation module 303, the low-rank correction values ​​of each low-rank fitting expert are weighted and aggregated according to the selection weights to obtain the aggregation increment. This includes: normalizing the selection weights of each low-rank fitting expert to obtain normalized expert selection weights; selecting the top k weights with the largest values ​​among the normalized expert selection weights, resetting the weights of unselected experts to zero, and re-normalizing the top k weights to obtain sparsed expert selection weights; and weighted summing the low-rank correction values ​​of each low-rank fitting expert according to the sparsed expert selection weights to obtain the aggregation increment.

[0099] In the extraction and weighted aggregation module 303, the expression for the aggregation increment is: ; in, For aggregated increments, The total number of multiple parallel low-rank adaptors. For the first The selection weights of low-rank fitting experts For the first The low-rank correction amount of a low-rank fitting expert.

[0100] In update module 304, the formation process of the overall uncertainty loss includes: extracting visual features from several pre-selected intermediate layers in the visual encoder and text encoder; performing global pooling on the visual features of each intermediate layer to obtain a pooled representation, and calculating the batch-level mean and standard deviation of the pooled representation, using the batch-level mean and standard deviation as the corresponding uncertainty statistics; sampling Gaussian factors and offsets based on the uncertainty statistics, and applying perturbation to the pooled representation using Gaussian factors and offsets to construct enhanced features; calculating the uncertainty loss corresponding to each intermediate layer based on the standard deviation of the enhanced features, with the contribution of the uncertainty loss to the total loss decreasing as the standard deviation of the enhanced features increases; and merging the uncertainty losses corresponding to several intermediate layers to obtain the overall uncertainty loss.

[0101] In update module 304, the formation process of the overall symmetric distillation matching loss includes: extracting visual and text features from several pre-selected intermediate layers in the visual encoder and text encoder; normalizing the visual and text features of each intermediate layer to obtain normalized visual and text features; projecting the normalized visual features onto the semantic space of the text features through a linear mapping to obtain projected visual features; calculating the similarity matrices in the image-to-text and text-to-image directions based on the projected visual features and normalized text features; normalizing the similarity matrices to obtain bidirectional probability distributions in the image-to-text and text-to-image directions; calculating the symmetric divergence between the bidirectional probability distributions and using the symmetric divergence as the symmetric distillation matching loss for each intermediate layer; and merging the symmetric distillation matching losses of several intermediate layers to obtain the overall symmetric distillation matching loss.

[0102] In update module 304, the expression for the total loss is: ; in, For the total loss, For global comparison loss, For overall uncertainty loss, This represents the overall symmetric distillation matching loss.

[0103] It should be noted that the above description of the remote sensing image retrieval device embodiment with cross-modal conditional routing and multi-expert low-rank adaptation is similar to the description of the remote sensing image retrieval method embodiment with cross-modal conditional routing and multi-expert low-rank adaptation, and has similar beneficial effects. For any technical details not disclosed in the embodiments of the remote sensing image retrieval device with cross-modal conditional routing and multi-expert low-rank adaptation of the present invention, please refer to the description of the remote sensing image retrieval method embodiment with cross-modal conditional routing and multi-expert low-rank adaptation of the present invention for understanding.

[0104] Based on the same inventive concept, embodiments of the present invention also provide a computer-readable storage medium, the computer-readable storage medium including a stored program, wherein, when the program is running, it controls the device where the storage medium is located to execute the remote sensing image and text retrieval method with cross-modal conditional routing and multi-expert low-rank adaptation described in one or more of the above embodiments.

[0105] It should be noted that the description of the above computer-readable storage medium embodiments is similar to the description of the above embodiments of the remote sensing image and text retrieval method with cross-modal conditional routing and multi-expert low-rank adaptation, and has similar beneficial effects. For technical details not disclosed in the embodiments of the computer-readable storage medium of this invention, please refer to the description of the embodiments of the remote sensing image and text retrieval method with cross-modal conditional routing and multi-expert low-rank adaptation.

[0106] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A remote sensing image retrieval method with cross-modal conditional routing and multi-expert low-rank adaptation, characterized in that, include: Obtain a remote sensing image and text retrieval dataset, which includes image data and text data; A remote sensing image and text retrieval model is constructed, which includes a visual encoder, a text encoder, and a router network. A multi-expert low-rank adaptation module is injected into the linear layers of the visual encoder and the text encoder. The multi-expert low-rank adaptation module includes multiple parallel low-rank adaptation experts, and each low-rank adaptation expert is used to generate a low-rank correction amount for the corresponding linear layer. The image data and the text data are input into the visual encoder and the text encoder, respectively, to extract global features of the image and global features of the text. A cross-modal conditional vector is constructed based on the global features of the image and the global features of the text; the cross-modal conditional vector is input into the router network to obtain the selection weights of each low-rank adaptation expert; the low-rank corrections of each low-rank adaptation expert are weighted and aggregated according to the selection weights to obtain the aggregation increment, which is used to modulate the output of the linear layer; The parameters of the multi-expert low-rank adaptation module and the router network are updated through backpropagation using the total loss, while keeping the main parameters of the visual encoder and the text encoder frozen, to obtain a trained remote sensing image and text retrieval model; the total loss includes global contrast loss, overall uncertainty loss and overall symmetric distillation matching loss; The remote sensing image to be retrieved and the candidate text are input into the trained remote sensing image-text retrieval model to output a text description that matches the remote sensing image to be retrieved, or to output a remote sensing image that matches the candidate text to be retrieved.

2. The remote sensing image retrieval method with cross-modal conditional routing and multi-expert low-rank adaptation according to claim 1, characterized in that, After acquiring the remote sensing image and text retrieval dataset, the method further includes: The image data is randomly cropped to obtain the cropping result; The cropping result is scaled to the target size, and the scaled image is horizontally flipped with a set probability to obtain the flipped image. The flipped image is then subjected to random enhancement to obtain a preprocessed image; The text data is converted to lowercase and punctuation is standardized to remove redundant characters and merge consecutive spaces, resulting in a standardized text sequence. The standardized text sequence is segmented into words, and the number of corresponding words is limited to a preset maximum length to generate a fixed-length text input sequence, which is then used as the preprocessed text.

3. The remote sensing image retrieval method with cross-modal conditional routing and multi-expert low-rank adaptation according to claim 1, characterized in that, The step of inputting the cross-modal condition vector into the router network to obtain the selection weights of each low-rank adaptation expert includes: The cross-modal condition vector is input into the router network to output the original routing scores of each low-rank adaptation expert based on the cross-modal condition vector. The original route scores are normalized using a Softmax function under temperature coefficient control to obtain the selection weights of each low-rank adaptation expert.

4. The remote sensing image retrieval method with cross-modal conditional routing and multi-expert low-rank adaptation according to claim 1, characterized in that, The step of weighted aggregation of the low-rank correction values ​​of each low-rank adaptation expert based on the selection weight to obtain the aggregation increment includes: The selection weights of each low-rank fitting expert are normalized to obtain the normalized expert selection weights. Select the k largest weights from the normalized expert selection weights, reset the unselected expert weights to zero, and re-normalize the k largest weights to obtain the sparsified expert selection weights. Based on the sparsified expert selection weights, the low-rank corrections of each low-rank fitting expert are weighted and summed to obtain the aggregation increment.

5. The remote sensing image retrieval method with cross-modal conditional routing and multi-expert low-rank adaptation according to claim 1, characterized in that, The expression for the aggregation increment is: ； in, For the aggregation increment, The total number of the multiple parallel low-rank adaptation experts. For the first The selection weights of low-rank fitting experts For the first The low-rank correction amount of a low-rank fitting expert.

6. The remote sensing image retrieval method with cross-modal conditional routing and multi-expert low-rank adaptation according to claim 1, characterized in that, The formation process of the overall uncertainty loss includes: Extract visual features from several pre-selected intermediate layers in the visual encoder and the text encoder; Global pooling is performed on the visual features of each intermediate layer to obtain a pooled representation, and the batch-level mean and standard deviation of the pooled representation are calculated. The batch-level mean and standard deviation are used as the corresponding uncertainty statistics. Based on the uncertainty statistics, a Gaussian factor and an offset are sampled, and the pooling representation is perturbed using the Gaussian factor and the offset to construct an enhanced feature; Based on the standard deviation of the enhanced feature, the uncertainty loss corresponding to each intermediate layer is calculated, and the contribution of the uncertainty loss to the total loss decreases as the standard deviation of the enhanced feature increases; The uncertainty losses corresponding to the several intermediate layers are combined to obtain the overall uncertainty loss.

7. The remote sensing image retrieval method with cross-modal conditional routing and multi-expert low-rank adaptation according to claim 1, characterized in that, The formation process of the overall symmetric distillation matching loss includes: Extract visual and text features from several pre-selected intermediate layers in the visual encoder and the text encoder; The visual and text features of each intermediate layer are normalized separately to obtain normalized visual features and normalized text features. The normalized visual features are projected onto the semantic space of the text features through a linear mapping to obtain the projected visual features. Based on the projected visual features and the normalized text features, calculate the similarity matrices from image to text and from text to image, respectively. The similarity matrix is ​​normalized to obtain bidirectional probability distributions in both the image-to-text and text-to-image directions; Calculate the symmetric divergence between the two-way probability distributions, and use the symmetric divergence as the symmetric distillation matching loss corresponding to each intermediate layer; The overall symmetric distillation matching loss is obtained by combining the symmetric distillation matching losses of the intermediate layers.

8. The remote sensing image retrieval method with cross-modal conditional routing and multi-expert low-rank adaptation according to claim 1, characterized in that, The expression for the total loss is: ； in, For the total loss, The global contrast loss is... For the overall uncertainty loss, The overall symmetric distillation matching loss is given.

9. A remote sensing image retrieval device with cross-modal conditional routing and multi-expert low-rank adaptation, characterized in that, include: The acquisition module is used to acquire a remote sensing image and text retrieval dataset, which includes image data and text data; A construction module is used to construct a remote sensing image and text retrieval model. The remote sensing image and text retrieval model includes a visual encoder, a text encoder, and a router network. A multi-expert low-rank adaptation module is injected into the linear layers of the visual encoder and the text encoder. The multi-expert low-rank adaptation module includes multiple parallel low-rank adaptation experts. Each low-rank adaptation expert is used to generate a low-rank correction amount for the corresponding linear layer. An extraction and weighted aggregation module is used to input the image data and the text data into the visual encoder and the text encoder, respectively, to extract global features of the image and global features of the text. A cross-modal conditional vector is constructed based on the global features of the image and the global features of the text; the cross-modal conditional vector is input into the router network to obtain the selection weights of each low-rank adaptation expert; the low-rank corrections of each low-rank adaptation expert are weighted and aggregated according to the selection weights to obtain the aggregation increment, which is used to modulate the output of the linear layer; The update module is used to update the parameters of the multi-expert low-rank adaptation module and the parameters of the router network through backpropagation using the total loss, while keeping the main parameters of the visual encoder and the text encoder frozen, to obtain the trained remote sensing image and text retrieval model; the total loss includes global contrast loss, overall uncertainty loss and overall symmetric distillation matching loss; The image and text retrieval module is used to input the remote sensing image to be retrieved and candidate text into the trained remote sensing image and text retrieval model, so as to output a text description that matches the remote sensing image to be retrieved, or output a remote sensing image that matches the candidate text to be retrieved.

10. A computer-readable storage medium, characterized in that, The storage medium includes a stored program, wherein, when the program is running, it controls the device where the storage medium is located to execute the remote sensing image retrieval method of cross-modal conditional routing and multi-expert low-rank adaptation as described in any one of claims 1 to 8.

Images