A remote sensing image-text retrieval method based on self-paced learning and noise robustness enhancement

Abstract

The application provides a remote sensing image-text retrieval method based on self-paced learning and noise robustness enhancement. By designing a self-paced learning strategy, the cognitive process of humans from easy to difficult is simulated, the training samples are dynamically divided into clean, fuzzy and noise samples, and a cross-modal self-paced weighting function is constructed to realize adaptive regulation of the learning order and weight of the samples. A robust triplet loss function is further proposed, an adaptive soft margin mechanism is introduced, and the tolerance of the model to noise is enhanced. At the same time, fine-grained cross-modal semantic alignment is realized through global and local contrast learning, and multi-modal feature fusion is realized through a multi-head attention mechanism. The application effectively alleviates the problem of retrieval performance degradation caused by noise correspondence, significantly improves the accuracy and robustness of cross-modal retrieval under various noise proportions, and provides high-quality and reliable cross-modal retrieval support for practical remote sensing fields such as smart cities, environmental monitoring, disaster assessment, agricultural investigation and urban planning.

Description

Technical Field

[0001] This invention relates to the fields of remote sensing image processing and multimodal information retrieval technology, and in particular to a robust remote sensing image-text retrieval method for noisy correspondences. This method effectively handles mismatch problems in training data through a self-stepping learning strategy and robust triplet loss, thereby improving retrieval accuracy and robustness. Background Technology

[0002] Remote sensing image-text retrieval aims to establish semantic connections between remote sensing images and natural language descriptions, enabling cross-modal bidirectional retrieval. It has significant application value in fields such as environmental monitoring, urban planning, and disaster assessment. In recent years, deep learning-based remote sensing image-text retrieval methods have mainly fallen into two categories: those based on deep convolutional neural networks and those based on Transformers. While these methods have made some progress, they all implicitly assume that the image-text pairs in the training data are perfectly matched. However, in practical applications, remote sensing images often have a top-down or vertical perspective, lacking human-centered top-down visual priors, leading to inherent ambiguity in the text descriptions. Large-scale construction of precisely aligned remote sensing image-text data is costly and impractical. Existing datasets (such as RSITMD) contain some inaccurate or incorrectly matched text descriptions, i.e., noisy correspondences. Such erroneous supervisory signals can mislead the model learning process and severely weaken retrieval performance.

[0003] In the prior art, Chinese patent application CN202411540620.4 discloses a multi-scale cross-modal satellite remote sensing image image-text retrieval method. This method uses a multi-source, multi-scale cross-modal remote sensing image semantic understanding model to process the input image retrieval information or text retrieval information, obtaining image retrieval feature vectors or text retrieval feature vectors. The similarity of these feature vectors is then compared with source feature vectors in a vector library to filter out target source satellite remote sensing images, thereby improving retrieval accuracy and efficiency. However, this existing method does not explicitly reveal and address the problem of noise-based correspondences based on mismatches, which still leads to performance degradation of the model on noisy real data. Therefore, it cannot significantly improve the accuracy and robustness of cross-modal retrieval under various noise levels.

[0004] Therefore, there is an urgent need to improve the design of a new method that can robustly handle noisy correspondences and improve cross-modal retrieval performance, thereby effectively alleviating the retrieval performance degradation problem caused by noisy correspondences. Summary of the Invention

[0005] (a) Technical problems to be solved

[0006] Based on the above, the technical problem to be solved by this invention is: to address the problem that existing remote sensing image-text retrieval methods are sensitive to noise correspondences in training data, leading to a decrease in retrieval performance, this invention proposes a remote sensing image-text retrieval method based on self-stepping learning and noise robustness enhancement. By simulating the human cognitive learning process from easy to difficult, this method adaptively distinguishes and processes clean, fuzzy, and noisy sample pairs, effectively suppressing noise interference and improving the robustness and retrieval accuracy of the model under noisy data.

[0007] (II) Technical Solution

[0008] This invention discloses a remote sensing image-text retrieval method based on auto-step learning and noise robustness enhancement, comprising the following steps:

[0009] Step 1: Process the input remote sensing image set and text set Feature extraction is performed using a pre-trained CLIP visual encoder. and text encoder Obtain global features of the image Local features of the image Global features of text and local text features Its expression is:

[0010]

[0011]

[0012] Step 2: Calculate the global and local similarities of the remote sensing image-text pairs using the fine-grained alignment module.

[0013] The global similarity is:

[0014]

[0015] The local similarity matrix is:

[0016]

[0017] Local similarity is:

[0018]

[0019] in, , ;

[0020] Global and local contrastive learning were performed using the InfoNCE loss function respectively:

[0021]

[0022]

[0023] In the above formula, and Let $\mathbf{j}$ represent the global similarity and local similarity of the $j$-th image-text pair, respectively. This represents the global similarity between the j-th image and the i-th text. This represents the global similarity between the i-th image and the j-th text. and These are the corresponding local similarities; Here, N represents the temperature hyperparameter, and N represents the batch size.

[0024] Step 3: Based on the contrastive loss value calculated in Step 2, dynamically divide the training sample pairs into three categories; It is the global contrastive loss for the i-th sample. This is the local contrast loss of the i-th sample; calculate the total contrast loss. And compare with a preset threshold: when When, it is a clean sample pair; when When, it is a fuzzy sample pair; when When, it is a noise sample pair; where, and This is a preset threshold parameter;

[0025] Step 4: Design a multimodal self-stepping learning function for clean sample pairs and fuzzy sample pairs; apply the self-stepping learning loss function to the clean sample pairs:

[0026]

[0027] Applying the self-stepping learning loss function to fuzzy sample pairs:

[0028]

[0029] In the above formula, Let be the weight of the i-th sample pair. The value is set to a preset threshold. or The self-stepping regularization term is:

[0030]

[0031] Sample weights The optimal solution is:

[0032] Step 5: Design a robust triplet loss function for all samples:

[0033]

[0034] In the above formula, , and The most similar negative sample within the batch is determined by the adaptive soft margin:

[0035]

[0036]

[0037] In the above formula, The basic interval hyperparameter;

[0038] Step 6: Construct the overall objective function:

[0039]

[0040] In the above formula, and As a balancing factor, the model is trained end-to-end using the backpropagation algorithm;

[0041] Step 7: In the inference phase, fuse global and local similarities to obtain fine-grained image-text similarity:

[0042]

[0043] In the above formula, To fuse hyperparameters, based on fine-grained similarity Sort the results and return the most relevant search results.

[0044] Furthermore, the feature extraction step uses a pre-trained CLIP model with the ViT-B-32 architecture as the encoder, which has powerful cross-modal representation capabilities.

[0045] Furthermore, in the sample classification step, the threshold... and The value is , It is used to distinguish sample pairs of different qualities.

[0046] Furthermore, in the self-stepping learning step, the self-stepping regularization term is designed with an inverse cosine function, which can smoothly adjust the sample weights, enabling the model to learn from easy to difficult.

[0047] Furthermore, in the robust triplet loss step, the basic interval hyperparameter... The value is 0.6, and the adaptive soft margin mechanism can dynamically adjust according to the similarity relationship between positive and negative samples.

[0048] Furthermore, in the overall objective function step, the balance factor takes the value of... , This is used to balance the contributions of each loss function.

[0049] Furthermore, in the inference step, hyperparameters are fused. This controls the relative contributions of global and local features to the final similarity.

[0050] Furthermore, the method also includes a performance evaluation step, using Rank@k and average rank mR as evaluation metrics, where k=1,5,10 in Rank@k.

[0051] In another aspect, the present invention discloses a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any of the preceding claims.

[0052] In another aspect, the present invention discloses a computer program product comprising a computer program that, when executed by a processor, implements the method described in any of the preceding claims.

[0053] (III) Beneficial Effects

[0054] Compared to existing technologies, this invention is the first to explicitly reveal and systematically address the problem of noisy correspondences in remote sensing image-text retrieval, and its technical problem discovery and handling possess certain unique characteristics. By designing a multimodal self-synchronous learning strategy, simulating the human cognitive process from easy to difficult, the reliability of samples is dynamically evaluated and the learning order is adjusted, enabling the model to robustly learn effective associations from noisy data. Simultaneously, the proposed robust triplet loss effectively mitigates the negative impact of noisy samples on model training through an adaptive soft-margin mechanism. Experimental data comparison shows that this invention exhibits superior retrieval performance and robustness under different noise ratios, significantly surpassing existing advanced methods, and providing a reliable solution for remote sensing cross-modal retrieval tasks with data quality issues in practical applications. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in this invention or the prior art, the accompanying drawings used in the embodiments will be briefly described below:

[0056] Figure 1 is a schematic diagram of the basic process of the method according to an embodiment of the present invention.

[0057] Figure 2 is a schematic diagram of a clean sample pair in an embodiment of the present invention.

[0058] Figure 3 is a schematic diagram of a noisy mismatched sample pair in an embodiment of the present invention.

[0059] Figure 4 is a schematic diagram of the network architecture of the method in an embodiment of the present invention. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of this invention. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0061] To address the problem of noise-induced correspondences in real-world remote sensing data severely weakening retrieval performance, and to improve retrieval performance and robustness under this specific noise condition, this invention provides a remote sensing image-text retrieval method based on auto-step learning and noise robustness enhancement. This method achieves robust cross-modal retrieval under noisy data through the organic combination of three core innovative modules: (1) a multimodal auto-step learning strategy that simulates the human cognitive process from easy to difficult, dynamically adjusting the learning order and weights of samples; (2) a robust triplet loss that enhances the model's tolerance to noise through an adaptive soft margin mechanism; and (3) a fine-grained alignment module that simultaneously integrates global and local information to achieve high-precision cross-modal semantic alignment.

[0062] like Figure 1-4 As shown, this invention discloses a remote sensing image-text retrieval method based on auto-step learning and noise robustness enhancement, comprising the following steps:

[0063] Step 1: Feature Extraction from Remote Sensing Images and Text

[0064] For the input remote sensing image set and text set Feature extraction is performed using a pre-trained CLIP visual encoder. and text encoder Obtain global features of the image Local features of the image Global features of text and local text features Its expression is:

[0065]

[0066]

[0067] In the above formula, and Let represent the global features of the image and text, respectively, with dimension d; and They represent the corresponding local feature sets, where and These represent the number of local features in the image and text, respectively. These features fully capture the spatial details of the remote sensing image and the semantic connotations of the text description, laying the foundation for subsequent alignment and matching.

[0068] In particular, the use of the pre-trained CLIP (Contrastive Language-Image Pre-training) model enables this method to leverage large-scale vision-language pre-training knowledge, significantly improving the understanding of remote sensing data in this specialized domain. Global features capture the overall semantic representation, while local features preserve fine-grained visual and textual details; the combination of the two makes the feature representation more comprehensive and accurate.

[0069] Step 2: Fine-grained alignment

[0070] The fine-grained alignment module aims to fully explore the semantic relationships between images and text at a detailed level, achieving more accurate cross-modal alignment by simultaneously calculating global and local similarity.

[0071] Global similarity calculation: Cosine similarity is used to measure the degree of correlation between global features.

[0072]

[0073] Global similarity reflects the degree of matching between image-text pairs at the overall semantic level, with a value range of [-1, 1]. Higher similarity indicates better semantic alignment.

[0074] Local similarity calculation: Calculate the similarity matrix for each pair of local features:

[0075]

[0076] In the above formula, , , obtained For one The similarity matrix.

[0077] To obtain the overall local similarity, L2 norm aggregation is used:

[0078]

[0079] This aggregation method can preserve the multidimensional relationship information between local features, avoiding the information loss caused by simple summation.

[0080] Contrastive learning: The InfoNCE loss function is used to perform comparative learning on global and local features, which promotes the increase of similarity between positive sample pairs and the decrease of similarity between negative sample pairs.

[0081] The global contrast loss is defined as:

[0082]

[0083] Local contrast loss is defined as:

[0084]

[0085] In the above formula, and Let represent the global and local positive sample similarity of the j-th image-text pair, respectively. The temperature parameter is used to adjust the sharpness of the similarity distribution, N is the batch size of the sample pairs, and the two summation terms correspond to the symmetry comparison between image to text and text to image, respectively.

[0086] The joint contrast loss is the sum of the two:

[0087]

[0088] This fine-grained alignment strategy ensures that the model can fully utilize multi-scale information in images and text, improving discriminability in the feature space.

[0089] Step 3: Sample Classification

[0090] Based on the contrast loss value calculated in step 2, this invention introduces a dynamic sample classification mechanism to adaptively divide training samples into three quality levels, thereby achieving differentiated processing of data of different quality.

[0091] For the i-th sample pair, calculate its total contrast loss:

[0092]

[0093] It is the global contrastive loss of the i-th sample ( (This refers to the overall comparison loss across the entire batch). It is the local contrast loss of the i-th sample ( This is the local contrastive loss for the entire batch, and the loss value reflects the learning difficulty of that sample pair under the current model. It is used in conjunction with two preset thresholds. and Comparison ( ), and divide into three categories:

[0094] Clean sample pairs ( This type of sample pair has a smaller loss value, indicating that the current model's understanding of semantic alignment is relatively accurate. The semantics of the image and text in the sample pair are highly consistent, with almost no mismatches or ambiguities, and should be fully utilized in the early stages of training. In the experiments... Set to 5. For details, please refer to [link / reference]. Figure 2As shown.

[0095] Fuzzy sample pairs ( This type of sample has a moderate loss value, indicating that there is some semantic relationship between the image and the text, but it is not clear enough. There may be slight descriptive ambiguity or the image may contain multiple land cover categories, leading to incomplete text matching. Although these samples contain noise, they still contain useful information and should be gradually introduced during training. In the experiments... Set it to 18.

[0096] Noise sample pairs ( This type of sample has a large loss value, indicating a significant semantic inconsistency between the image and the text. This could be due to labeling errors, completely unrelated text, or differences in image capture angle. These samples carry misleading information and should be removed or downweighted during training. For details on this type of sample, please refer to [link / reference needed]. Figure 3 As shown.

[0097] The advantage of this classification mechanism lies in its dynamism: as training progresses, the model's performance continuously improves, and the loss value of the samples changes. Some samples may move from the noisy class to the fuzzy class, or from the fuzzy class to the clean class, thus achieving a dynamic re-evaluation of the samples. This adaptive classification aligns with the process of humans gradually deepening their understanding.

[0098] Step 4: Multimodal self-walking learning

[0099] Multimodal self-walking learning is one of the core innovations of this invention. Differentiated learning strategies are designed for samples of different qualities, and progressive learning from easy to difficult is achieved through dynamic weight allocation.

[0100] Self-learning loss for clean sample pairs:

[0101]

[0102] Self-learning loss for fuzzy sample pairs:

[0103]

[0104] In the above formula, The weight is the learning weight for the i-th sample pair, quantitatively reflecting its contribution to the model's learning. A higher weight value indicates a higher reliability of the sample pair, and it should receive more attention during training.

[0105] Self-stepping regularization term By employing an inverse cosine function design, the sample weights can be smoothly constrained, guiding the nonlinear relationship between the weight values ​​and the sample loss values.

[0106]

[0107] The characteristic of this regular expression is: The value is set to a preset threshold. or ,when Relative to threshold When the value is small, the model is encouraged to give the sample a higher weight; when... Approaching or exceeding When the value of the regularization term is zero, the weight is set to zero.

[0108] By taking the derivative and setting it to zero, the optimal weight expression can be obtained:

[0109]

[0110] This weighting function has the following properties: 1) Monotonically decreasing: when From 0 to hour, 1) Smoothly decreases from 1 to 0; 2) Smoothness: The cosine function is used to ensure that the weight changes smoothly and avoids steep jumps; 3) Physical interpretability: The mapping between weight and loss value intuitively reflects the logic that "the smaller the loss, the higher the weight".

[0111] During training, the model prioritizes learning clean sample pairs (lower loss results in higher weights), and then gradually introduces fuzzy sample pairs. This design fully utilizes the information contained in samples of different quality, avoiding being misled by noisy samples while fully exploring the learning value of useful samples.

[0112] Step 5: Robust Triple Loss

[0113] To further enhance the model's tolerance to noise, this invention designs a robust triplet loss calculation method and dynamically adjusts the learning difficulty through an adaptive soft margin mechanism.

[0114] The robust triplet loss function is defined as:

[0115]

[0116] In the above formula, This is the ReLU function, which has a gradient only when the value inside the parentheses is positive.

[0117] Negative sample selection:

[0118] Is it related to the current image? Other texts with the highest similarity (the most difficult negative text sample)

[0119] Is with the current text Other images with the highest similarity (the most difficult negative image sample)

[0120] also, This is the global similarity calculation function. Not for the current text Other texts, Not the current image Other images, and It is a soft margin parameter. This hard negative sample mining strategy ensures that the model focuses on easily confused sample pairs, thereby improving the discrimination ability.

[0121] Adaptive soft interval: Unlike the traditional fixed-interval triplet loss, this invention dynamically calculates the soft interval. and :

[0122]

[0123]

[0124] In the above formula, The baseline interval hyperparameter is preferably set to 0.6 in the experiment.

[0125] The dynamism of soft margin is manifested in: 1) Adaptability: The margin size depends on the similarity difference between the current negative sample and the positive sample; 2) Noise tolerance: When the similarity of positive samples is low (which may be noise), the margin automatically increases, relaxing the learning requirements; 3) Hard sample mining: When the similarity of positive samples is close to that of negative samples, the margin also increases, strengthening the learning of that sample pair; 4) Balance: Through bidirectional symmetrical processing (image-text and text-image), the balance of learning is ensured.

[0126] Compared to the fixed-interval method, the adaptive soft-interval mechanism of this invention is more flexible in handling noise. It will not over-penalize noisy positive sample pairs, and can also effectively constrain the similarity of negative samples, thereby accurately calculating the robust triplet loss under noise.

[0127] Step 6: Overall Objective Function and Model Training

[0128] To fully leverage the advantages of each loss function, this invention employs a weighted summation method to construct the overall objective function:

[0129]

[0130] In the above formula, Learning from clean samples is the fundamental guarantee of model performance; Responsible for progressive learning of fuzzy samples, gradually expanding the learning range of the model; Responsible for robust triple constraints on all samples, reinforcing discriminative learning; balancing factor and The contribution of different loss terms was adjusted and set to 0.8 and 0.9 respectively in the experiments. These parameter settings were based on extensive experimental verification, ensuring the optimal balance among the various loss functions.

[0131] Training process: The backpropagation algorithm is used to optimize the model parameters end-to-end. Specifically, in each training iteration, given a batch of image-text pairs, features are first calculated in step 1, then similarity and initial loss are calculated in step 2, and finally sample classification and evaluation are performed according to step 3, calculating the following respectively. , and Then, the weighted total loss is calculated, gradient backpropagation is performed, and finally the parameters of the encoder and feature space are updated.

[0132] This end-to-end training approach enables the various modules to collaborate with each other. Sample classification, weight calculation, and loss calculation form a unified optimization framework, which jointly drives the model to learn robust cross-modal representations.

[0133] Step 7: Reasoning and Retrieval

[0134] In the reasoning phase, to obtain a more accurate similarity estimate, this invention integrates global and local similarity:

[0135]

[0136] In the above formula, To integrate hyperparameters, a value of 0.9 was preferred in the experiment, indicating that global similarity dominates the final judgment, while local similarity provides detailed clues as supplementary information.

[0137] This method of merging similarity has a significant advantage: global constraints. This ensures overall semantic matching, while also addressing local details. Fine-grained feature correspondences were added, and then through The parameters can be adjusted to balance the weights of global and local information according to the specific application scenario, ensuring a flexible trade-off.

[0138] Retrieval Process: For the query image, extract its global and local features. First, calculate the global and local similarity with all candidate texts. Then, calculate the fused similarity using a formula. Then, based on fusion similarity Candidate texts are sorted from highest to lowest, and the top k texts are returned as the search results.

[0139] Similarly, it can perform text queries and image retrieval tasks. This retrieval method based on fusion similarity makes full use of multi-level feature information, and is more accurate and robust than methods that only use global or local similarity.

[0140] To illustrate the optimal weight analytical expression in the above main steps The design of implicit objective functions is not a conventional technique in this field. It can achieve enhanced robustness by significantly reducing sensitivity to large loss values, thereby solving the problem that noise correspondences commonly found in real remote sensing data cause serious degradation of retrieval performance. In this invention, the derivation process of the formula and its superiority are further demonstrated as follows.

[0141] 1. Theoretical Analysis

[0142] For clarity, the following symbols are used throughout this invention. This represents network parameters. This represents the sum of the global and local contrast loss for the i-th sample pair, which is the total contrast loss mentioned earlier. and It is a threshold. Refers to robust triplet loss. and It is a balancing factor.

[0143] In the main text, the optimal weight solution is derived as follows:

[0144]

[0145] This invention proposes some important properties of the objective function, which may help to elucidate the basic principles of remote sensing image-text retrieval methods based on autostep learning and noise robustness enhancement.

[0146] Before analyzing remote sensing image-text retrieval methods based on autosynchronous learning and noise robustness enhancement, for the sake of simplicity, Recorded as This invention introduces two key variables: and . Specifically, Defined as right The integral is given as:

[0147]

[0148]

[0149] Then the present invention is defined for exist First-order expansion at the point:

[0150]

[0151]

[0152] like As shown, this invention observes that the proposed objective function is analytically challenging because it requires simultaneously solving for the self-stepping weights. and network parameters In the following proposition, this invention demonstrates that a remote sensing image-text retrieval method based on autostep learning and noise robustness enhancement actually optimizes a simplified implicit objective function, where the autostep weights... Completely eliminated.

[0153] Proposition 1 (Implicit Objective Function)

[0154] For fixed and , used to minimize The Alternating Optimization Strategy (AOS) is equivalent to applying the master optimization-minimization algorithm to the function.

[0155] prove:

[0156] The following is true:

[0157]

[0158]

[0159] Furthermore, the following inequalities hold:

[0160]

[0161] Therefore, the master optimization-minimization algorithm can be used to minimize the objective function, with the key step being minimizing the surrogate function. The steps for the k-th iteration are as follows:

[0162] Main optimization step: In this step, Fixed as To obtain the surrogate function, it is necessary to compute by solving the following problem. and :

[0163]

[0164]

[0165] This calculation follows the AOS standard. Updated program.

[0166] Minimize step: In this step, fix for Update by minimizing the proxy function :

[0167]

[0168] This process is related to AOS. The update steps are the same.

[0169] It can be easily deduced that the AOS strategy of this invention is equivalent to the aforementioned master optimization-minimization algorithm. Furthermore, This can be considered as the basic objective function for the actual optimization of this invention, and and Let represent the basic loss function for the i-th sample.

[0170] A remote sensing image-text retrieval method based on auto-step learning and noise robustness enhancement minimizes an implicit loss function, thereby eliminating... Based on this derivation, this invention now proposes a follow-up proposition to demonstrate the robustness of the remote sensing image-text retrieval method based on auto-step learning and noise robustness enhancement to difficult samples.

[0171] Proposition 2 (Robustness)

[0172] To simplify the symbols, Record Assuming and For constants and Then, for the training dataset Any two different instances The following holds true:

[0173] For functions The present invention includes:

[0174]

[0175] for ,when and At that time, the present invention has:

[0176]

[0177] Where the constant and Defined as:

[0178]

[0179]

[0180] prove

[0181] set up and .because and They are continuously differentiable on their respective domains, and according to the mean value theorem, there exists... Make:

[0182]

[0183]

[0184] based on and Definition.

[0185] for The present invention includes:

[0186]

[0187] for The present invention includes:

[0188]

[0189] because and The present invention has and ,in and Use respectively and definition.

[0190] Therefore, for any two distinct instances that satisfy their respective conditions The inequality holds.

[0191] According to Proposition 2, the transformed loss function and Showing a difference compared to the original loss function Stronger robustness to difficult samples with large loss values. Specifically, when considering difficult samples... and simple samples At the same time, the measured loss difference in remote sensing image-text retrieval methods based on auto-step learning and noise robustness enhancement and The difference in loss becomes significantly smaller than the original loss. ,because and This result confirms and All of these methods achieve enhanced robustness by significantly reducing sensitivity to large loss values.

[0192] To achieve optimal remote sensing image-text retrieval performance, this embodiment trains and tests the model on the large-scale remote sensing image-text benchmark dataset RSITMD (see the paper Yuan Z, Zhang W, Fu K, et al. Exploring a fine-grained multiscale method for cross-modal remote sensing image retrieval[J]. arXivpreprint arXiv:2204.09868, 2022.) to obtain the best image-text retrieval model. The RSITMD dataset contains 4,743 remote sensing images across 32 land cover categories, with each image measuring 256×256 pixels; it consists of 3,861 training images, 430 validation images, and 452 test images. To simulate the noise correspondence problem in reality, synthetic noise is introduced into the training set by randomly shuffling text descriptions to generate training data with controlled noise. Four noise levels (20%, 40%, 60%, and 80%) are set to evaluate the robustness of the model under different noise intensities. As a comparison with the method in this embodiment, the CUP method is used (see the paper Wang Y, Tang X, Ma J, et al. Cross-modal remote sensing image-text retrieval via context and uncertainty-aware prompt[J]. IEEE transactions on neural networks and learning systems, 2025.). Furthermore, Rank@k (R@k, k=1,5,10) and average rank (mR) are used to measure the accuracy of the retrieval results; higher values ​​indicate better retrieval performance. Rank@k measures the percentage of correct matches appearing in the top k retrieval results; mR is the average of all Rank@k metrics. Table 1 shows the retrieval results of the method in this embodiment and the CUP method on the RSITMD dataset.

[0193] Table 1. Comparison of retrieval results of the method in this embodiment and the CUP method on the RSITMD dataset.

[0194]

[0195] As shown in Table 1, the remote sensing image-text retrieval method based on self-stepping learning and noise robustness enhancement in this embodiment outperforms the CUP method in all indicators. Under noise-free conditions, the mR of this method reaches 44.16%, an improvement of 5.81 percentage points compared to CUP (38.35%). The robustness advantage of this method gradually becomes apparent with increasing noise levels: the mR is 42.49% under 20% noise, 41.37% under 40% noise, 39.28% under 60% noise, and 32.64% under 80% noise, representing improvements of 4.93, 5.9, 6.61, and 6.42 percentage points respectively compared to CUP. In particular, the R@1 of this method reaches 16.68% under extreme noise conditions of 80%, far exceeding CUP's 10.71%, fully demonstrating its superior performance in high-noise environments.

[0196] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A remote sensing image-text retrieval method based on auto-step learning and noise robustness enhancement, characterized in that, Includes the following steps: Step 1: Process the input remote sensing image set and text set Feature extraction is performed using a pre-trained CLIP visual encoder. and text encoder Obtain global features of the image Local features of the image Global features of text and local text features Its expression is: Step 2: Calculate the global and local similarities of the remote sensing image-text pairs using the fine-grained alignment module. The global similarity is: The local similarity matrix is: Local similarity is: in, , ; Global and local contrastive learning were performed using the InfoNCE loss function respectively: In the above formula, and Let $\mathbf{j}$ represent the global similarity and local similarity of the $j$-th image-text pair, respectively. This represents the global similarity between the j-th image and the i-th text. This represents the global similarity between the i-th image and the j-th text. and These are the corresponding local similarities; Here, N represents the temperature hyperparameter, and N represents the batch size. Step 3: Based on the contrastive loss value calculated in Step 2, dynamically divide the training sample pairs into three categories; It is the global contrastive loss for the i-th sample. This is the local contrast loss of the i-th sample; calculate the total contrast loss. And compare with a preset threshold: when When, it is a clean sample pair; when When, it is a fuzzy sample pair; when When, it is a noise sample pair; where, and This is a preset threshold parameter; Step 4: Design a multimodal self-stepping learning function for clean sample pairs and fuzzy sample pairs; apply the self-stepping learning loss function to the clean sample pairs: Applying the self-stepping learning loss function to fuzzy sample pairs: In the above formula, Let be the weight of the i-th sample pair. The value is set to a preset threshold. or The self-stepping regularization term is: Sample weights The optimal solution is: Step 5: Design a robust triplet loss function for all samples: In the above formula, , and The most similar negative sample within the batch is determined by the adaptive soft margin: In the above formula, The basic interval hyperparameter; Step 6: Construct the overall objective function: In the above formula, and As a balancing factor, the model is trained end-to-end using the backpropagation algorithm; Step 7: In the inference phase, fuse global and local similarities to obtain fine-grained image-text similarity: In the above formula, To fuse hyperparameters, based on fine-grained similarity Sort the results and return the most relevant search results.

2. The method according to claim 1, characterized in that, The feature extraction step uses a pre-trained CLIP model with the ViT-B-32 architecture as the encoder, which has powerful cross-modal representation capabilities.

3. The method according to claim 1, characterized in that, In the sample classification step, the threshold and The value is , It is used to distinguish sample pairs of different qualities.

4. The method according to claim 1, characterized in that, In the self-stepping learning step, the self-stepping regularization term is designed with an inverse cosine function, which can smoothly adjust the sample weights, enabling the model to learn from easy to difficult.

5. The method according to claim 1, characterized in that, In the robust triplet loss step, the basic interval hyperparameter The value is 0.6, and the adaptive soft margin mechanism can dynamically adjust according to the similarity relationship between positive and negative samples.

6. The method according to claim 1, characterized in that, In the overall objective function step, the balance factor takes the value of... , This is used to balance the contributions of each loss function.

7. The method according to claim 1, characterized in that, In the inference step, hyperparameters are fused. This controls the relative contributions of global and local features to the final similarity.

8. The method according to claim 1, characterized in that, The method also includes a performance evaluation step, using Rank@k and average rank mR as evaluation metrics, where k=1,5,10 in Rank@k.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method of any one of claims 1 to 8.

10. A computer program product, comprising a computer program, characterized in that, When the program is executed by the processor, it implements the method of any one of claims 1 to 8.

Images