Remote sensing cross-modal retrieval method and system based on large model fine tuning

Abstract

The invention discloses a remote sensing cross-modal retrieval method and system based on large model fine tuning. The method comprises the following steps: S1, inputting an obtained remote sensing image and / or description text into a trained remote sensing image text retrieval network for processing to obtain image features and / or text features after cross-modal fusion; s2, acquiring a text and / or an image matched with the remote sensing image and / or the description text based on the image features and / or the text features, and outputting the text and / or the image; the step S1 comprises: extracting initial image features and / or initial text features from the remote sensing image and / or the description text through an image text encoder; performing cross-modal fusion processing on the initial image features and / or the initial text features through a cross-modal asymmetric adapter to obtain image features and / or text features after cross-modal fusion; and carrying out optimization processing on the image features and / or the text features through a dual-task consistency loss function.

Description

Technical Field

[0001] The present application relates to the technical field of remote sensing cross-modal retrieval, and in particular to a remote sensing cross-modal retrieval method and system based on large model fine-tuning. Background Art

[0002] Remote sensing image-text retrieval tasks are of great importance and far-reaching significance in remote sensing data processing and applications. The core of remote sensing image-text retrieval is to achieve efficient matching and retrieval between images and texts. By integrating multimodal data, remote sensing image-text retrieval can not only improve the accuracy and comprehensiveness of remote sensing data analysis, but also provide strong support for diversified application scenarios.

[0003] At present, remote sensing image-text retrieval methods are mainly divided into traditional deep learning methods and methods based on large-scale visual-language pre-trained models. Traditional deep learning methods usually rely on deep learning operators such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), and Transformers to extract features from remote sensing images and text data respectively. Subsequently, these features are mapped to a shared feature space and cross-modal matching is achieved through similarity metrics. However, when dealing with complex and abstract cross-modal relationships between remote sensing images and texts, such methods often find it difficult to achieve accurate feature alignment, which limits the improvement of retrieval performance. To this end, some studies have introduced methods based on large-scale visual-language pre-trained models, which freeze most of the model parameters and only fine-tune a small number of key parameters, while retaining the original learning information while combining domain knowledge. This method not only significantly reduces the training cost, but also improves the cross-modal matching accuracy of the model in remote sensing scenarios. Among the parameter efficient fine-tuning methods, adapters are the most common fine-tuning scheme. However, in remote sensing image-text retrieval tasks, there are the following problems: Since text modality information is more discriminative and easier to optimize, it usually becomes the dominant modality, thereby inhibiting the optimization process of the image modality. More importantly, image and text modalities often use adapters with the same structure, which ignores the potential representation differences between the modalities, resulting in modality imbalance, thus affecting the performance and generalization ability of the overall model. Summary of the invention

[0004] The embodiments of the present application provide a remote sensing cross-modal retrieval method and system based on large model fine-tuning, so as to solve the problem that the fine-tuning method in the above-mentioned prior art fails to fully consider the differences in image and text modal features.

[0005] On the one hand, an embodiment of the present application provides a remote sensing cross-modal retrieval method based on large model fine-tuning, the method comprising: S1. Using the text describing the remote sensing image as the description text, the remote sensing image and / or the description text are input into a trained remote sensing image text retrieval network for processing to obtain image features and text features after cross-modal fusion; S2, obtaining text and / or image matching the remote sensing image and / or description text based on image features and / or text features for output; Wherein, step S1 comprises: S11, extracting initial image features and / or initial text features from the remote sensing image and / or description text by an image text encoder; S12, performing cross-modal fusion processing on the initial image features and / or initial text features through a cross-modal asymmetric adapter to obtain cross-modal fused image features and / or text features; S13. Optimizing image features and / or text features through a dual-task consistency loss function.

[0006] Preferably, the image text encoder includes an image encoder and a text encoder; step S11 includes: Extracting initial image features from remote sensing images through an image encoder; Initial text features are extracted from the description text through a text encoder.

[0007] Preferably, step S12 includes: S121, performing dimensionality reduction processing on initial image features and initial text features; S122, using a nonlinear activation function GELU to process the initial image features and initial text features after the dimensionality reduction process; S123, performing feature expression enhancement processing on the initial image features and the initial text features processed by the nonlinear activation function respectively; S124, inputting the initial image features and initial text features processed based on feature expression enhancement into a shared layer for cross-modal fusion processing to obtain cross-modal fused image features and / or text features.

[0008] Preferably, the cross-modal asymmetric adapter includes: a visual enhancement adapter and a text semantic adapter; the visual enhancement adapter and the text semantic adapter respectively include a linear transformation and a nonlinear activation function.

[0009] Preferably, in step S123, the visual enhancement adapter performs feature expression enhancement processing on the initial image features through a differential attention mechanism, including: Map the query, key vector, and value vector to output; Calculate the attention score based on the query and key vectors, and then weighted sum it with the value vector; Eliminate attention score noise based on the softmax function.

[0010] Preferably, in step S123, the text semantic adapter performs feature expression enhancement processing on the initial text features through a hierarchical attention mechanism, including: Obtain word-level hidden representations from word-level annotations through a multi-layer perceptron; Calculate the importance weight of a word based on the word-level hidden representation and the word-level context vector; The sentence vector is obtained by weighted summing of word-level annotations based on importance weights.

[0011] Preferably, the dual-task consistency loss function includes the following loss terms: Consistency constraints, based on the exponential moving average mechanism, use the text modality as a teacher model to guide the image modality learning; Cross-modal constraints, measuring the global alignment of images and texts through cosine similarity; Classification constraints, which calculate the difference between the predicted probability distribution and the true category distribution; Based on an adaptive weight adjustment strategy, the consistency constraint loss term, the cross-modal constraint loss term and the classification constraint loss term are dynamically weighted to optimize image features and / or text features.

[0012] On the other hand, the present application also provides a remote sensing cross-modal retrieval system based on large model fine-tuning, the system comprising: An image text input module is used to use the text describing the remote sensing image as the description text, and output the remote sensing image and / or the description text; A network composition and processing module, used for processing the remote sensing image and / or description text input by the image text input module to obtain image features and / or text features after cross-modal fusion; An image text output module, which outputs text and / or images matching the remote sensing image and / or the description text based on the image features and / or the text features; Among them, the network composition and processing modules include: An image-text encoder that extracts initial image features and / or initial text features from a remote sensing image and / or a description text; Performing cross-modal fusion processing on initial image features and / or initial text features to obtain a cross-modal asymmetric adapter of cross-modal fused image features and / or text features; A dual-task consistency loss function that optimizes image features and / or text features.

[0013] On the other hand, the present application also provides a terminal device, including: a memory and a processor, the memory is used to store a remote sensing cross-modal retrieval program based on large model fine-tuning, and when the remote sensing cross-modal retrieval program based on large model fine-tuning is executed by the processor, the steps of the remote sensing cross-modal retrieval method based on large model fine-tuning as above are implemented.

[0014] On the other hand, the present application also provides a computer-readable storage medium, wherein the storage medium stores a computer program, and when the computer program is executed, it implements the above-mentioned remote sensing cross-modal retrieval method based on large model fine-tuning.

[0015] The remote sensing cross-modal retrieval method and system based on large model fine-tuning in this application has the following advantages: The solution provided in this application effectively addresses the differences in image and text modal features through a cross-modal asymmetric adapter, combined with the collaborative work of a visual enhancement adapter and a text semantic adapter, and achieves better alignment and optimization of image and text modal features during fine-tuning. At the same time, by expanding the single-task model to a multi-task model and introducing a dual-task consistency loss function, the consistency learning between modalities is strengthened, thereby improving the retrieval performance. This method has demonstrated excellent adaptability and significant performance improvement in remote sensing image-text retrieval tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 A schematic diagram of a remote sensing cross-modal retrieval method based on large model fine-tuning provided in an embodiment of the present application; Figure 2 A schematic diagram of a process for processing remote sensing images and / or description texts by a remote sensing image text retrieval network provided in an embodiment of the present application; Figure 3 A schematic diagram of a process for processing initial image features and / or initial text features by a cross-modal asymmetric adapter provided in an embodiment of the present application; Figure 4 A schematic diagram of the structure of a remote sensing cross-modal retrieval system based on large model fine-tuning provided in an embodiment of the present application; Figure 5 A schematic diagram of the training process of a remote sensing image text retrieval network provided in an embodiment of the present application; Figure 6 An image-text retrieval analysis comparison table provided for the embodiments of the present application; Figure 7 A text-image retrieval analysis comparison table provided for an embodiment of the present application. DETAILED DESCRIPTION

[0018] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0019] refer to Figure 1 As shown, a flow chart of a remote sensing cross-modal retrieval method based on large model fine-tuning provided in an embodiment of the present application is provided. An embodiment of the present application provides a remote sensing cross-modal retrieval method based on large model fine-tuning, the method comprising: S1. Using the text describing the remote sensing image as the description text, the remote sensing image and / or the description text are input into a trained remote sensing image text retrieval network for processing to obtain image features and / or text features after cross-modal fusion; S2. Based on the cross-modal fusion features, text and / or image matching the remote sensing image and / or description text is obtained for output.

[0020] In an embodiment of the present application, in the process of performing a remote sensing image-text retrieval task, the remote sensing image or description text is obtained and input into a remote sensing image text retrieval network obtained by deep learning training in advance through big data for processing, and image features and text features after cross-modal fusion are obtained. Furthermore, text information matching the image features or image information matching the text features is obtained through the image features and text features, and the matched text information or image information is output as text or image matching the remote sensing image or description text.

[0021] In one embodiment, when performing a remote sensing image-text retrieval task, when only a remote sensing image is obtained, the remote sensing image is processed through a remote sensing image text retrieval network to obtain image features after cross-modal fusion of the remote sensing image and the auxiliary text features. For example, in the fused features, the regular arrangement of the roof of the factory building (visual) is semantically combined with "intensive industrial facilities" (text), and the interference of "residential area" is excluded during retrieval; the outline of the lake (visual) is associated with "lake surrounded by mountains" (text) to avoid mismatching as "plain reservoir". During retrieval, the text description with the highest matching degree is obtained according to the image feature and output as the text matched by the remote sensing image.

[0022] In one embodiment, when only the description text is obtained during the remote sensing image-text retrieval task, the description text is processed through the remote sensing image text retrieval network to obtain the text features after the cross-modal fusion of the description text and the auxiliary image features. For example, the fused features are associated with visual patterns such as "regularly arranged factory buildings" and "highly reflective roofs", excluding the image of "scattered residential houses"; the semantics of "oasis" in the features are strongly associated with "small patches of vegetation" and "contrast colors of sand" in vision, which improves the fine-grained retrieval accuracy. During the retrieval, the image with the highest matching degree is obtained according to the text features and output as the image matching the description text.

[0023] In one embodiment, during the execution of the remote sensing image-text retrieval task, when the remote sensing image and the text describing the remote sensing image are obtained at the same time, the remote sensing image and the description text are cross-modal fused through the remote sensing image text retrieval network to obtain image features and text features corresponding to the remote sensing image and the description text, respectively. After performing a search based on the image features and text features, the text and image matching the remote sensing image and the description text are obtained and output.

[0024] refer to Figure 2 As shown, a schematic diagram of a process of processing a remote sensing image and / or a description text by a remote sensing image text retrieval network provided in an embodiment of the present application. The steps of processing a remote sensing image and / or a description text by a remote sensing image text retrieval network provided in an embodiment of the present application include: S11, extracting initial image features and / or initial text features from the remote sensing image and / or description text by an image text encoder; In an embodiment of the present application, after receiving the remote sensing image and / or the description text, the image text encoder extracts the initial image features from the received remote sensing image through the image encoder. , extracting initial text features from the received description text through a text encoder In one embodiment, a vision transformer (ViT) is selected as an image encoder to extract initial image features from the remote sensing image. , select Bidirectional Encoder Representations from Transformers (BERT) as the text encoder to extract the initial text features in the description text In this embodiment, both encoders perform feature extraction through a transformer block, which includes 12 transformer blocks in total.

[0025] S12, performing cross-modal fusion processing on the initial image features and / or initial text features through a cross-modal asymmetric adapter to obtain cross-modal fused image features and / or text features; In an embodiment of the present application, the cross-modal asymmetric adapter performs cross-modal fusion on the extracted image features and / or text features, as follows: Figure 3 The steps shown are processed: S121, performing dimensionality reduction processing on initial image features and initial text features; S122, using a nonlinear activation function GELU (Gaussian error linear unit) to process the initial image features and initial text features after the dimensionality reduction process; S123, performing feature expression enhancement processing on the initial image features and the initial text features processed by the nonlinear activation function respectively; S124, inputting the initial image features and initial text features processed based on feature expression enhancement into a shared layer for cross-modal fusion processing to obtain cross-modal fused image features and / or text features.

[0026] In an embodiment of the present application, the cross-modal asymmetric adapter first extracts the initial image features through the Transformer block. and initial text features Perform dimensionality reduction and project it into a low-dimensional representation; then use the nonlinear activation function GELU to optimize the initial image features and initial text features Then, the initial image features of the image modality are processed by the differential attention mechanism, and the initial text features of the text modality are processed by the hierarchical attention mechanism to further enhance the respective feature expressions. Then, these processed features are input into the shared layer for interaction between the image modality and the text modality, and finally restored to the original input size through the dimensionality increase layer. In this embodiment, the cross-modal asymmetric adapter includes a visual enhancement adapter and a text semantic adapter, wherein a typical adapter includes a series of linear transformations and nonlinear activation functions, and its calculation formula is as follows: .

[0027] Among them, Adapter represents the adapter, is the extracted features; and are the weights of the lower and upper projections respectively; is a nonlinear activation function, is a GELU activation function, and the superscript T represents transposition.

[0028] In an embodiment of the present application, the visual enhancement adapter of the cross-modal asymmetric adapter, when performing feature expression enhancement processing on the initial image features, introduces a differential attention mechanism to extract the initial image features of the image modality in a more fine-grained manner. The differential attention mechanism maps the query, key vector, and value vector to the output, and uses the query and key vectors to calculate the attention score, and then weighted sums the value vector. The core is to introduce a pair of softmax functions to eliminate noise in the attention score. The specific calculation formula is as follows:

[0029] in, Q 1 and Q 2 is the query vector, K 1 and K 2 is the key vector, V is the value vector; They are Parameters, where Q for Q 1 and Q 2 collectively, K for K 1 and K 2; DiffAttn represents the differential attention mechanism; d The dimension of the key vector is used to scale the dot product result to prevent the gradient from exploding or disappearing; is a learnable scalar. In order to coordinate the dynamic changes of the learning process, Reparameterize to:

[0030] in, is a learnable vector; Is used for initialization The constant.

[0031] In this embodiment, the features extracted by the differential attention mechanism are passed to the shared layer to share information with the text modality, thereby achieving the complementarity of the two modalities. For example, when some key information in the text is omitted or unclear, the visual information of the image can be used as a supplement to help the model better understand and reason. Finally, before the feature is upgraded to the input dimension, a gating mechanism is introduced to regulate the information interaction between different modalities to prevent a certain modality feature from being overly dominant or ignored, thereby achieving a more balanced cross-modal fusion.

[0032] In an embodiment of the present application, the text semantic adapter of the cross-modal asymmetric adapter, when performing feature expression enhancement processing on the initial text features, extracts key information words by introducing a hierarchical attention mechanism, and aggregates their representations to construct a sentence vector, and is executed according to the following steps, wherein the calculation formula for each step is as follows:

[0033]

[0034]

[0035] in, For word-level annotation, for The hidden representation of W w represents the learnable weight matrix, b w represents the learnable bias term, is the normalized importance weight, is the word-level context vector, is the sentence vector.

[0036] First, the word-level annotation Pass a single-layer multi-layer perceptron (MLP) to obtain its hidden representation .

[0037] Next, through With word-level context vector The importance of the word is measured by the similarity of the word, and the normalized importance weight is obtained through the softmax function. .

[0038] Finally, the word-level annotations are weighted and summed according to these weights to obtain the sentence vector Among them, the word-level context vector It can be seen as a high-level representation of a fixed query "Which word is the most informative word", similar to the query mechanism used in Memory Networks.

[0039] In this embodiment, during the training process, It is randomly initialized and jointly learned with other parameters of the model. After being processed by the hierarchical attention mechanism, the subsequent operations of the visual enhancement adapter and the text semantic adapter are consistent.

[0040] S13. Optimizing image features and / or text features through a dual-task consistency loss function.

[0041] In an embodiment of the present application, after the cross-modal asymmetric adapter obtains the fused image features and text features, the two final features are subjected to similarity calculation, cross entropy loss calculation and mean square error calculation. These three items are used as components of the loss function and are dynamically weighted and optimized.

[0042] Among them, the loss term corresponding to the mean square error calculation is the consistency constraint loss term. By introducing the indicator moving mean EMA mechanism, the text modality is used as the teacher model and the image modality is used as the student model. The classification accuracy of the image modality is improved by using the guidance of the text modality. The learning process of the image modality is optimized by measuring the gap between the predicted outputs of the image modality and the text modality. The calculation formula of the mean square error is as follows:

[0043] in, L consist is the mean square error, is the sample size; It is The true value of samples; It is The predicted value of a sample.

[0044] Among them, the loss term corresponding to the similarity calculation is the cross-modal loss term. The cross-modal alignment of remote sensing images and texts is achieved by measuring their global similarity. The formula for similarity calculation is as follows:

[0045] in, L cross For similarity, represents a matched pair of samples; For remote sensing images Unmatched text; For and text Mismatched remote sensing images; is the cosine similarity; is the boundary value of the cross-modal constraint; .

[0046] The calculation formula of cosine similarity is as follows:

[0047] in, I i For images I No. i Dimension values, T i For text T No. iThe loss term corresponding to the cross entropy loss calculation is the classification constraint loss term, which measures the classification performance of the model by calculating the difference between the probability distribution predicted by the model and the true category distribution. The formula for calculating the cross entropy loss is as follows:

[0048] in, L cls is the cross entropy loss, is the number of categories; is the one-hot encoding of the true label. If the sample belongs to the category ,but ,otherwise ; is the model for the category The predicted probability of .

[0049] In the embodiments of the present application, in view of the different effects of different loss terms on model training, in order to avoid the weight imbalance problem caused by directly accumulating each loss term in previous studies, an adaptive weight adjustment strategy is introduced to dynamically weighted optimize the above three loss terms. This strategy can dynamically adjust the weight ratio according to the contribution of each loss term during the training process, thereby more effectively guiding model learning and improving the effect of cross-modal feature alignment. The specific form of the dual-task consistency loss function is as follows:

[0050] in, L total is the dual-task consistency loss, It is The learnable parameters corresponding to the loss terms; They are , representing a specific loss item; For dynamic weighting; It is a regularization term used to limit the growth of weights and avoid excessive bias towards a certain loss term.

[0051] refer to Figure 4 The present application also provides a remote sensing cross-modal retrieval system based on large model fine-tuning, which includes: an image text input module, a network composition and processing module, and an image text output module.

[0052] In an embodiment of the present application, after acquiring a remote sensing image or a descriptive text, the remote sensing cross-modal retrieval system based on large model fine-tuning inputs the remote sensing image or the descriptive text into a network composition and processing module through an image-text input module for cross-modal fusion processing to obtain image features or text features after cross-modal fusion, and an image-text output module obtains text information or image information that matches the remote sensing image or the descriptive text based on the image features or text features.

[0053] In an embodiment of the present application, the network composition and processing module is composed of an image-text encoder, a cross-modal asymmetric adapter and a dual-task consistency loss function, wherein the image-text encoder adopts a vision transformer (ViT) large model and a bidirectional encoder representation method Bidirectional Encoder Representationsfrom Transformers (BERT) large model to extract initial image features or initial text features respectively; the cross-modal asymmetric adapter processes the image modality and the text modality respectively by adopting a differential attention mechanism and a hierarchical attention mechanism, and interacts between modalities; the dual-task consistency loss function dynamically adjusts the weight ratio according to the contribution of each loss item during the training process by using an adaptive weight adjustment strategy, so as to more effectively guide model learning.

[0054] In this embodiment, the cross-modal asymmetric adapter first extracts the initial image features through the Transformer block. and initial text features Perform dimensionality reduction and project it into a low-dimensional representation; then use the nonlinear activation function GELU to optimize the initial image features and initial text features Then, the initial image features of the image modality are processed by the differential attention mechanism, and the initial text features of the text modality are processed by the hierarchical attention mechanism to further enhance the respective feature expressions. Then, these processed features are input into the shared layer for interaction between the image modality and the text modality, and finally restored to the original input size through the dimensionality increase layer. In this embodiment, the cross-modal asymmetric adapter includes a visual enhancement adapter and a text semantic adapter, wherein a typical adapter includes a series of linear transformations and nonlinear activation functions, and its calculation formula is as follows:

[0055] in, is the extracted features; and are the weights of the lower and upper projections respectively; is a nonlinear activation function and is a GELU activation function.

[0056] In an embodiment of the present application, the visual enhancement adapter of the cross-modal asymmetric adapter, when performing feature expression enhancement processing on the initial image features, introduces a differential attention mechanism to extract the initial image features of the image modality in a more fine-grained manner. The differential attention mechanism maps the query, key vector, and value vector to the output, and uses the query and key vectors to calculate the attention score, and then weighted sums the value vector. The core is to introduce a pair of softmax functions to eliminate noise in the attention score. The specific calculation formula is as follows:

[0057] in, They are query, key vector and value vector respectively; They are Parameters; is a learnable scalar. In order to coordinate the dynamic changes of the learning process, Reparameterize to:

[0058] in, is a learnable vector; Is used for initialization The constant.

[0059] In this embodiment, the features extracted by the differential attention mechanism are passed to the shared layer to share information with the text modality, thereby achieving the complementarity of the two modalities. For example, when some key information in the text is omitted or unclear, the visual information of the image can be used as a supplement to help the model better understand and reason. Finally, before the feature is upgraded to the input dimension, a gating mechanism is introduced to regulate the information interaction between different modalities to prevent a certain modality feature from being overly dominant or ignored, thereby achieving a more balanced cross-modal fusion.

[0060] In an embodiment of the present application, the text semantic adapter of the cross-modal asymmetric adapter, when performing feature expression enhancement processing on the initial text features, extracts key information words by introducing a hierarchical attention mechanism, and aggregates their representations to construct a sentence vector, and is executed according to the following steps, wherein the calculation formula for each step is as follows:

[0061]

[0062]

[0063] First, the word-level annotation Pass a single-layer multi-layer perceptron (MLP) to obtain its hidden representation .

[0064] Next, through With word-level context vector The importance of the word is measured by the similarity of the word, and the normalized importance weight is obtained through the softmax function. .

[0065] Finally, the word-level annotations are weighted and summed according to these weights to obtain the sentence vector Among them, the word-level context vector It can be seen as a high-level representation of a fixed query "Which word is the most informative word", similar to the query mechanism used in Memory Networks.

[0066] In the embodiment of the present application, the dual-task consistency loss function avoids the weight imbalance problem caused by directly accumulating each loss term, and dynamically adjusts the weight ratio according to the contribution of each loss term during the training process, thereby more effectively guiding model learning and improving the effect of cross-modal feature alignment. Among them, each loss term includes: Corresponding to the consistency constraint loss term of the mean square error calculation, by introducing the indicator moving mean EMA mechanism, the text modality is used as the teacher model and the image modality is used as the student model, and the classification accuracy of the image modality is improved by using the guidance of the text modality. By measuring the gap between the predicted outputs of the image modality and the text modality, the learning process of the image modality is optimized. The calculation formula of the mean square error is as follows:

[0067] in, is the sample size; It is The true value of samples; It is The predicted value of a sample.

[0068] Corresponding to the cross-modal loss term of similarity calculation, the cross-modal alignment of remote sensing images and texts is achieved by measuring their global similarity. The formula for similarity calculation is as follows:

[0069] in, represents a matched pair of samples; For remote sensing images Unmatched text; For and text Mismatched remote sensing images; is the boundary value of the cross-modal constraint; ; The calculation formula of cosine similarity is as follows: .

[0070] The classification constraint loss term is calculated corresponding to the cross entropy loss. The classification performance of the model is measured by calculating the difference between the probability distribution predicted by the model and the true category distribution. The formula for calculating the cross entropy loss is as follows:

[0071] in, is the number of categories; is the one-hot encoding of the true label. If the sample belongs to the category ,but ,otherwise ; is the model for the category The predicted probability of .

[0072] In the embodiments of the present application, in view of the different effects of different loss terms on model training, in order to avoid the weight imbalance problem caused by directly accumulating each loss term in previous studies, an adaptive weight adjustment strategy is introduced to dynamically weighted optimize the above three loss terms. This strategy can dynamically adjust the weight ratio according to the contribution of each loss term during the training process, thereby more effectively guiding model learning and improving the effect of cross-modal feature alignment. The specific form of the dual-task consistency loss function is as follows:

[0073] in, It is The learnable parameters corresponding to the loss terms; They are , representing a specific loss item; For dynamic weighting; It is a regularization term used to limit the growth of weights and avoid excessive bias towards a certain loss term.

[0074] In an embodiment of the present application, the image-text output module obtains matching text information from a text library based on image features after cross-modal fusion, and obtains matching image information from an image library based on text features after cross-modal fusion.

[0075] refer to Figure 5 As shown, the present application also provides a training method for a remote sensing image text retrieval network, including: Construct training samples for remote sensing image text retrieval network, select RSICD and RSITMD datasets, and divide the datasets into training set, validation set, and test set, accounting for 80%, 10%, and 10% respectively. Among them, the training set is used for model training, and the parameters of the model are optimized through the data in the training set; the validation set is used for tuning during the model training process; and the test set is used to evaluate the final performance of the model. The remote sensing images in the training set, validation set, and test set are adjusted to a fixed size of 224×224; the final features are linearly projected to a 512-dimensional common space.

[0076] Construct a remote sensing cross-modal retrieval network based on large model fine-tuning, including an image-text encoder, a cross-modal asymmetric adapter, and a dual-task consistency loss function.

[0077] The vision transformer (ViT) was selected as the image encoder, and the bidirectional encoder representation method (BERT) was selected as the text encoder. At the beginning of training, the parameters of the ViT image encoder and the BERT text encoder themselves remained frozen and were not updated during the training process; all images were resized to a fixed size of 224×224 for training; the dropout value and the cross-modal loss function were set to 0.2; the initial learning rate of the network was set to 2e-4, and a weight decay of 0.7 was performed after every 20 training cycles; during the training process, the Adam optimizer was used to optimize the network parameters, the batchsize was set to 32, and the total training cycle was set to 30; to ensure the stability and reliability of the experimental results, the experiment used k-fold cross validation for evaluation, and finally took the average value of the 5-fold cross validation (k=5) for performance evaluation and presentation.

[0078] In the remote sensing image-text retrieval task, existing methods usually design adapters with the same architecture for image modality and text modality. However, due to the significant representation differences between image and text data, using the same adapter architecture to process both often fails to accurately extract the key information of each modality, thus affecting the performance of the model. In the embodiment of the present application, a cross-modal asymmetric adapter including a visual enhancement adapter and a text semantic adapter is designed for the characteristics of the two modalities, which is used to fine-tune the remote sensing image text retrieval network.

[0079] In remote sensing image-text retrieval tasks, bidirectional triplet loss has become the mainstream optimization strategy. However, due to the differences in image and text modal features, there are obvious differences in the classification performance between the two, and it is impossible to accurately align and match features. In addition, this task usually involves multiple loss terms. If the losses of all samples are directly accumulated, it may cause difficulties in the training process to converge, making it difficult for the model to effectively capture subtle feature differences. To this end, this application expands the single-task model to a multi-task model and proposes a dual-task consistency loss function with adaptive weight adjustment capabilities. This function can dynamically mine and optimize difficult samples, strengthen feature alignment and consistency between modalities, and thus significantly improve retrieval performance.

[0080] During training, the remote sensing image text retrieval network was run on an NVIDIA RTX 4070 12GB GPU. In terms of feature representation, the final features of the CLIP (Contrastive Language-Image Pre-training) or GeoRSCLIP (Geographic Remote Sensing Contrastive Language-Image Pre-training) model were linearly projected to a 512-dimensional common space. For remote sensing images, all images were resized to a fixed size of 224×224 for training. In terms of parameter setting, the dropout value and the cross-modal loss function were both set to 0.2. The initial learning rate of the network was set to 2e-4, and a weight decay of 0.7 was performed after every 20 training cycles. During training, the Adam optimizer was used to optimize the network parameters, the batchsize was set to 32, and the total training cycle was set to 30. To ensure the stability and reliability of the experimental results, the experiment was evaluated using k-fold cross-validation, and the average of the 5-fold cross-validation (k=5) was finally taken for performance evaluation and presentation.

[0081] In order to more intuitively illustrate the effectiveness of the remote sensing cross-modal retrieval method based on large model fine-tuning provided by this application, this application uses two indicators, R@K (K=1, 5 and 10) and mR (mean recall rate), to evaluate the model performance. Among them, R@K is used to measure the proportion of true matches in the query results that appear in the first K retrieval results, where K is 1, 5 and 10, to evaluate the accuracy of the model at different retrieval depths; mR represents the average value of each R@K indicator in text retrieval and image retrieval tasks, which is used to comprehensively evaluate the overall performance of the model in cross-modal retrieval tasks. The specific calculation formula is as follows:

[0082] As shown in Tables 1 and 2, the performance comparisons of the remote sensing cross-modal retrieval method based on large model fine-tuning in this application and other retrieval methods on the RSICD (Remote Sensing Image Captioning) and RSITMD (Remote Sensing Image Cross-Modal Retrieval Fine-Grained Multi-Scale) datasets are given. The bold black represents the method proposed in this application and the fully fine-tuned GeoRSCLIP, respectively.

[0083] Table 1 Performance comparison of the retrieval method of this application and other retrieval methods on the RSICD dataset

[0084] Table 2 Performance comparison of the retrieval method of this application and other retrieval methods on the RSITMD dataset

[0085] In Tables 1 and 2, AMFMN-soft is an asymmetric multimodal feature matching network-soft method, AMFMN-fusion is an asymmetric multimodal feature matching network-fusion method, AMFMN-sim is an asymmetric multimodal feature matching network-similarity method, GaLR w / o MR is a geographic attention guided cross-modal retrieval method without edge reconstruction, GaLR with MR is a geographic attention guided cross-modal retrieval method with edge reconstruction, PIR is a private information retrieval method, Full-FT CLIP is a full-scale fine-tuned contrast language-image pre-training model, Full-FT GeoRSCLIP is a full-scale fine-tuned geographic remote sensing contrast language-image pre-training model, and PE-RSITR is a remote sensing image target recognition technology based on piezoelectric effect.

[0086] As shown in Tables 1 and 2, the remote sensing cross-modal retrieval method based on large model fine-tuning in this application has improved performance by 13%-23% compared with traditional methods, and has surpassed advanced traditional retrieval methods in recent years. The CLIP-based method fully utilizes its powerful generalization ability and rich visual-language prior knowledge obtained through pre-training on large-scale natural scene datasets, thereby achieving significant performance improvement in remote sensing image text retrieval tasks. Specifically, the remote sensing cross-modal retrieval method based on large model fine-tuning provided in this application has improved MR (edge ​​reconstruction) on RSICD by about 16.44% compared with CLIP-Adapter and 6.97% compared with PE-RSITR; on RSITMD, the retrieval method provided in this application has improved by 13.98% compared with Cross-ModalAdapter and 7.57% compared with PE-RSITR. It is worth noting that with the help of the GeoRSCLIP pre-trained model, the retrieval method provided in this application outperforms all current parameter-efficient fine-tuning methods on the RSICD and RSITMD datasets, and even exceeds the retrieval performance of the fully fine-tuned GeoRSCLIP model, verifying the effectiveness of the retrieval method provided in this application.

[0087] In order to further verify the superiority of the remote sensing cross-modal retrieval method based on large model fine-tuning in this application, the following Figure 6 and Figure 7 The visual analysis shown. Figure 6 It can be seen from the image-text retrieval results that the remote sensing cross-modal retrieval method based on large model fine-tuning provided by the present application is superior to the fully fine-tuned GeoRSCLIP in retrieval accuracy. For example, in the second image, the erroneous text retrieved by the remote sensing cross-modal retrieval method based on large model fine-tuning provided by the present application can correctly identify the number of "baseballfield", while Text-1 and Text-3 retrieved by the fully fine-tuned GeoRSCLIP only focus on key targets and fail to capture the quantity information of the targets. The above phenomenon shows that when performing image feature extraction in complex remote sensing scenes, the remote sensing cross-modal retrieval method based on large model fine-tuning provided by the present application shows stronger capabilities in the fine-grained understanding and expression of image modal features. From Figure 7It can be seen from the text-image retrieval results that the remote sensing cross-modal retrieval method based on large model fine-tuning provided in this application is significantly better than the fully fine-tuned GeoRSCLIP in retrieval performance. For example, for the first text, the remote sensing cross-modal retrieval method based on large model fine-tuning provided in this application can accurately identify the image corresponding to the text in Top-1, while the fully fine-tuned GeoRSCLIP only retrieves the correct result in Top-3. These results fully demonstrate the advantages of the remote sensing cross-modal retrieval method based on large model fine-tuning provided in this application in fine-grained feature extraction and cross-modal information alignment, and effectively bridges the differences between image and text modalities, significantly improving the retrieval performance and generalization ability of the model.

[0088] Based on the above analysis results, the experimental results on multiple benchmark datasets show that the remote sensing cross-modal retrieval method based on large model fine-tuning in this application is superior to other state-of-the-art retrieval methods in multiple indicators, and surpasses the effect of full fine-tuning, verifying its effectiveness and reliability in remote sensing image text retrieval tasks.

[0089] The present application also provides a terminal device, including: a memory and a processor, the memory is used to store a remote sensing cross-modal retrieval program based on large model fine-tuning, and when the remote sensing cross-modal retrieval program based on large model fine-tuning is executed by the processor, the steps of the remote sensing cross-modal retrieval method based on large model fine-tuning as above are implemented.

[0090] The present application also provides a computer-readable storage medium, wherein the storage medium stores a computer program, and when the computer program is executed, it implements the above-mentioned remote sensing cross-modal retrieval method based on large model fine-tuning.

[0091] In summary, the solution provided in this application effectively addresses the differences in image and text modal features through a cross-modal asymmetric adapter, combined with the collaborative work of a visual enhancement adapter and a text semantic adapter, and achieves better alignment and optimization of image and text modal features during fine-tuning. At the same time, by expanding the single-task model to a multi-task model and introducing a dual-task consistency loss function, the consistency learning between modalities is strengthened, thereby improving the retrieval performance. This method has demonstrated excellent adaptability and significant performance improvement in remote sensing image-text retrieval tasks.

[0092] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0093] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A remote sensing cross-modal retrieval method based on large model fine-tuning, characterized in that: The method comprises: S1. Using a text describing a remote sensing image as a description text, inputting the remote sensing image and / or the description text into a trained remote sensing image text retrieval network for processing to obtain image features and / or text features after cross-modal fusion; S2. Based on the image features and / or the text features, obtain text and / or image matching the remote sensing image and / or the description text for output; Wherein, the step S1 comprises: S11, extracting initial image features and / or initial text features from the remote sensing image and / or the description text by using an image text encoder; S12, performing cross-modal fusion processing on the initial image features and / or the initial text features through a cross-modal asymmetric adapter to obtain the cross-modal fused image features and / or text features; S13, optimizing the image features and / or the text features through a dual-task consistency loss function.

2. The remote sensing cross-modal retrieval method based on large model fine-tuning according to claim 1 is characterized in that: The image text encoder includes an image encoder and a text encoder; the step S11 includes: extracting initial image features from the remote sensing image by means of the image encoder; Initial text features are extracted from the description text by the text encoder.

3. The remote sensing cross-modal retrieval method based on large model fine-tuning according to claim 1 is characterized in that: The step S12 comprises: S121, performing dimensionality reduction processing on the initial image features and the initial text features; S122, using a nonlinear activation function GELU to process the initial image features and the initial text features after the dimensionality reduction process; S123, performing feature expression enhancement processing on the initial image features and the initial text features processed by the nonlinear activation function respectively; S124, inputting the initial image features and the initial text features processed based on feature expression enhancement into a shared layer for cross-modal fusion processing to obtain image features and / or text features after cross-modal fusion.

4. The remote sensing cross-modal retrieval method based on large model fine-tuning according to claim 3 is characterized in that: The cross-modal asymmetric adapter includes: a visual enhancement adapter and a text semantic adapter; wherein the visual enhancement adapter and the text semantic adapter respectively contain a linear transformation and a nonlinear activation function.

5. The remote sensing cross-modal retrieval method based on large model fine-tuning according to claim 4 is characterized in that: In step S123, the visual enhancement adapter performs feature expression enhancement processing on the initial image features through a differential attention mechanism, including: Map the query, key vector, and value vector to output; Calculate an attention score based on the query and the key vector, and then perform a weighted summation with the value vector; Eliminate attention score noise based on the softmax function.

6. The remote sensing cross-modal retrieval method based on large model fine-tuning according to claim 4 is characterized in that: In step S123, the text semantic adapter performs feature expression enhancement processing on the initial text features through a hierarchical attention mechanism, including: Obtain word-level hidden representations from word-level annotations through a multi-layer perceptron; Calculating the importance weight of a word based on the word-level hidden representation and the word-level context vector; The sentence vector is obtained by weighted summing of the word-level annotations based on the importance weights.

7. The remote sensing cross-modal retrieval method based on large model fine-tuning according to claim 1 is characterized in that: The dual-task consistency loss function includes the following loss terms: Consistency constraints, based on the exponential moving average mechanism, use the text modality as a teacher model to guide the image modality learning; Cross-modal constraints, measuring the global alignment of images and texts through cosine similarity; Classification constraints, which calculate the difference between the predicted probability distribution and the true category distribution; The consistency constraint loss term, the cross-modal constraint loss term and the classification constraint loss term are dynamically weighted based on an adaptive weight adjustment strategy to optimize the image features and / or the text features.

8. A remote sensing cross-modal retrieval system based on large model fine-tuning, characterized in that: The system comprises: An image text input module, used to use the text describing the remote sensing image as the description text, and output the remote sensing image and / or the description text; A network composition and processing module, used for processing the remote sensing image and / or the description text input by the image text input module to obtain image features and / or text features after cross-modal fusion; An image text output module, which obtains text and / or image matching the remote sensing image and / or the description text based on the image features and / or text features after the cross-modal fusion for output; Wherein, the network composition and processing module includes: An image-text encoder that extracts initial image features and / or initial text features from the remote sensing image and / or the description text; Performing cross-modal fusion processing on the initial image features and / or the initial text features to obtain a cross-modal asymmetric adapter of the cross-modal fused image features and / or text features; A dual-task consistency loss function for optimizing the image features and / or the text features.

9. A terminal device, characterized in that: The terminal device includes: a memory and a processor, the memory is used to store a remote sensing cross-modal retrieval program based on large model fine-tuning, and when the remote sensing cross-modal retrieval program based on large model fine-tuning is executed by the processor, the steps of the remote sensing cross-modal retrieval method based on large model fine-tuning as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed, it implements the remote sensing cross-modal retrieval method based on large model fine-tuning as described in any one of claims 1 to 7.

Images