Remote sensing image-text retrieval method based on correlation division

Abstract

The invention provides a remote sensing image-text retrieval method based on correlation division. The remote sensing image-text retrieval method comprises the following steps: cross-modal feature extraction; based on the last layer of attention weight of the Transform, selecting image block features and word features of a preset proportion before the attention weight ranking; respectively calculating the global feature similarity of the image and the text and the local feature similarity optimized through nonlinear transformation so as to complete cross-modal feature similarity calculation; based on the global and local feature similarities, a probability distribution model is adopted to divide the image-text pairs, positive sample pairs and negative sample pairs are obtained according to posterior probability division, the posterior probability of the positive sample pairs is larger than a preset threshold value, and the posterior probability of the negative sample pairs is smaller than or equal to the preset threshold value; training the model according to the correlation differentiation of the sample pair by using a loss function of dynamically adjusting the weight; and fusing global and local features to calculate image-text similarity, and returning a matching result.

Description

Technical Field

[0001] The present invention belongs to the technical field of information processing, and particularly relates to a remote sensing image-text retrieval method based on relevance division. It aims to solve the semantic alignment problem between remote sensing images and text descriptions, and improve the retrieval accuracy and model training effect in the remote sensing image-text retrieval task. Background Art

[0002] Remote sensing image-text retrieval aims to return remote sensing images with the same semantics according to a given query text. This technology is of great significance in many practical applications, especially in the fields of remote sensing data analysis, environmental monitoring, disaster emergency response, urban planning, etc. Through effective image-text retrieval, the required image or text information can be quickly obtained from a large amount of remote sensing data, improving the efficiency of data processing and analysis.

[0003] The semantic alignment between remote sensing images and text is a core challenge in remote sensing image-text retrieval. Existing cross-modal retrieval methods mostly rely on attention mechanisms or simple linear transformations to achieve feature alignment, and have the following limitations: (1) lack of in-depth optimization of the fine-grained relationship between image patches and text word vectors; (2) most methods rely on global similarity, and the cross-modal nature of remote sensing image-text features is insufficiently measured; (3) the influence of the correlation degree of sample pairs on the model training process is not considered, and the effectiveness of model training needs to be improved. In fact, there are often large differences between the detailed information in remote sensing images and the high-level abstract descriptions in text, which leads to incomplete semantic alignment of image-text pairs, thus affecting the model training effect and retrieval accuracy. Summary of the Invention

[0004] Aiming at the defects and deficiencies of the existing technology, the present invention provides a remote sensing image-text retrieval method based on relevance division, which solves the problem of limited retrieval accuracy caused by rough sample division, insufficient local feature representation, and low training data quality in traditional methods through a multi-level cross-modal alignment and dynamic optimization mechanism. The innovative designs of the present invention include:

[0005] Global-local multi-level feature alignment: Combining the global features of the image with the local features optimized by residual blocks (MLP + FC + MaxPool), and combining the global and fine-grained word vector features of the text to achieve cross-modal fine-grained matching from the overall semantics to the detailed differences;

[0006] Probabilistic sample division mechanism: Modeling the image-text similarity based on the Gaussian mixture model (GMM), and dynamically dividing high / low correlation sample pairs through posterior probability, breaking through the limitations of artificial thresholds or random sampling, and significantly improving the quality of training data;

[0007] Dynamic weighted triple loss function: Introduce a temperature parameter to adjust the weight distribution of positive and negative samples, strengthen the model's learning ability for difficult samples (highly similar negative samples, lowly similar positive samples), and enhance the robustness in complex scenarios;

[0008] Attention-driven local feature screening: Extract key image patches and text word vectors from the attention weights of the last layer of Transformer, and optimize the fine-grained representation through residual blocks to solve the problems of noise interference and insufficient local alignment.

[0009] Through the collaboration of the above technologies, in the RSICD dataset, the mR index is improved by 9.75%, and the text retrieval image (R@1) is improved by 61.3%; in the RSITMD dataset, the mR index is improved by 7.85%, and the comprehensive retrieval accuracy is significantly better than the existing technologies, providing an efficient and robust cross-modal alignment framework for remote sensing image-text retrieval.

[0010] The technical solution specifically adopted by the present invention to solve its technical problems is as follows:

[0011] A remote sensing image-text retrieval method based on correlation division, comprising the following steps:

[0012] Segment the input remote sensing image into multiple image patches and extract global features and local features, and tokenize the input text and extract global features and local features to achieve cross-modal feature extraction;

[0013] Based on the attention weights of the last layer of Transformer, select the image patch features and word features with the top preset proportion of attention weights; calculate the global feature similarity between the image and the text, and the local feature similarity optimized by non-linear transformation respectively to complete the cross-modal feature similarity calculation;

[0014] Based on the global and local feature similarities, use a probability distribution model to divide the image-text pairs, and obtain positive sample pairs and negative sample pairs according to the posterior probability division, where the posterior probability of the positive sample pairs is greater than a preset threshold, and the posterior probability of the negative sample pairs is less than or equal to the preset threshold;

[0015] Use a loss function with dynamically adjusted weights to differentially train the model according to the correlation of the sample pairs;

[0016] Fuse the global and local features to calculate the image-text similarity and return the matching result.

[0017] Further, the probability distribution model is a Gaussian mixture model GMM, which specifically includes:

[0018] Perform two-component Gaussian distribution modeling on the global and local feature similarities;

[0019] Estimate the mean, variance, and weight parameters by the Expectation-Maximization algorithm;

[0020] Dynamically partition the positive and negative sample pairs according to the posterior probability.

[0021] Furthermore, the loss function for dynamically adjusting the weights is the weighted triplet loss function:

[0022]

[0023] where α i is the distance weight of each positive sample, β j is the distance weight of each negative sample, exp(x)=e x , ∈ is the temperature hyperparameter that adjusts the sharpness of the probability distribution; [·] + =max(·,0), d(·,·) represents the cosine distance between the inputs, a + (i) represents the i-th positive sample related to the anchor a, a - (i) represents the i-th negative sample, and the index i traverses all positive and negative samples within a training batch; m is a margin constant used to ensure that the distance of the negative sample is greater than that of the positive sample.

[0024] Furthermore, during the calculation of the cross-modal feature similarity, the optimization of the local features includes:

[0025] Perform non-linear transformations on the selected local image patch features and text word vector features through a multi-layer perceptron and a fully connected layer respectively;

[0026] Aggregate the optimized local features through max pooling.

[0027] Furthermore, the global and local feature similarity calculation uses the cosine similarity metric.

[0028] Furthermore, the calculation of the image-text similarity is specifically:

[0029]

[0030] where v i is the sum of the global and local image features, t j is the sum of the global and local text features, · represents the dot product operation, and the calculation result is the cosine similarity between the image and text features.

[0031] Furthermore, the non-linear transformation is implemented through a residual block, specifically including the joint operation of a multi-layer perceptron and a fully connected layer, and performing a residual connection with the input features.

[0032] And, a remote sensing image-text retrieval system based on correlation division, including:

[0033] Feature extraction module: used to segment the input remote sensing image into multiple image patches and extract global features and local features, as well as tokenize the input text and extract global features and local features;

[0034] Similarity calculation module: used to calculate the global feature similarity between images and texts, as well as the local feature similarity optimized by non-linear transformation;

[0035] Correlation division module: used to divide positive and negative sample pairs based on global and local feature similarities using the Gaussian mixture model GMM;

[0036] Training module: used to differentially train the model through a weighted triplet loss function;

[0037] Retrieval module: used to calculate the image-text similarity by fusing global and local features and return the matching results.

[0038] And, an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, and when the processor executes the program, the steps of the above method are implemented.

[0039] A non-transitory computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0040] Compared with the prior art, the present invention and its preferred solutions at least include the following beneficial effects:

[0041] Significantly improve the cross-modal alignment accuracy: through the joint calculation of global and local multi-level features, taking into account the overall semantic consistency and the ability to capture fine-grained differences, overcoming the matching deviation problem caused by the traditional method relying on a single feature dimension;

[0042] Optimize the quality of training data: based on the probabilistic sample division mechanism of the Gaussian mixture model, it can more accurately distinguish high / low correlation image-text pairs, reduce the interference of noise samples to model training, and improve the representativeness and effectiveness of training data;

[0043] Enhance the robustness and generalization ability of the model: through the differential processing of samples with different degrees of correlation by the dynamically weighted triplet loss function, strengthen the model's ability to distinguish complex samples (such as similar negative samples, fuzzy positive samples), and reduce the risk of overfitting;

[0044] Improve the local feature representation ability: based on the local feature screening based on the attention mechanism and the optimized design of residual blocks, effectively improve the fine-grained expression of image patches and text word vectors, and solve the problem of loss or weakening of local semantic information in traditional methods;

[0045] Improving Retrieval Efficiency and Practicality: By means of a feature fusion strategy, the similarity calculation process is simplified, reducing the computational complexity while ensuring accuracy, which is applicable to the real-time retrieval scenario of large-scale remote sensing image-text data.

[0046] Through the above-mentioned collaborative optimization, this solution provides a systematic solution for remote sensing image-text retrieval tasks that balances accuracy, robustness, and practicality. Brief Description of the Drawings

[0047] The present invention will be further described in detail below in conjunction with the drawings and specific embodiments:

[0048] Figure 1 This is the flowchart of model training for an embodiment of the present invention.

[0049] Figure 2 This is the flowchart of model testing for an embodiment of the present invention. Specific Embodiments

[0050] To make the features and advantages of the present invention more obvious and understandable, specific embodiments are hereinafter given and described in detail as follows:

[0051] It should be noted that the following detailed descriptions are all illustrative and are intended to provide further explanations for the present application. Unless otherwise specified, all technical and scientific terms used in this specification have the same meaning as commonly understood by those of ordinary skill in the technical field to which the present application belongs.

[0052] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular forms are also intended to include the plural forms. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or their combinations.

[0053] An embodiment of the present invention proposes a remote sensing image-text retrieval method based on relevance partitioning, aiming to construct a remote sensing image-text relevance evaluation mechanism, re-partition the training data to make the training process more refined, not only extract information from significantly relevant image-text pairs, but also reduce the negative impact caused by the inconsistency of image-text pairs, thereby improving the image-text alignment effect and thus enhancing the performance of remote sensing image-text retrieval.

[0054] In the embodiments of the present invention, by introducing the distance calculation of global and local features, the cross-modal correlation division based on the Gaussian mixture model, and the weighted triplet loss training mechanism, the present invention can more finely evaluate the relevance between remote sensing images and texts, optimize the training data according to the actual matching degree of images and texts, and thus improve the accuracy and stability of retrieval. The present invention proposes the calculation of the global and local dual feature similarities between images and texts, realizes a comprehensive evaluation of the matching degree between images and texts from the fine-grained to the overall level, helps the model to more accurately capture the subtle differences and overall relevance between images and texts; proposes to use the Gaussian mixture model to divide the relevance of remote sensing image-text pairs, which can finely distinguish positive and negative sample pairs, thereby improving the quality of training data and ensuring that the model can accurately identify highly relevant samples during the learning process; designs a weighted triplet loss function, which can improve the stability of training and reduce the interference of weakly relevant samples on model training by differentiating image-text pairs with different relevance degrees. The present invention extracts the global and local features of images and texts and adds them to perform similarity ranking, and returns the image that best matches the query text, improving the accuracy of remote sensing image-text retrieval. The technical solutions for achieving the objectives of the present invention include the following steps:

[0055] (1) Cross-modal feature extraction: To effectively represent the features of images and texts and ensure that as rich feature information as possible can be extracted from the two modalities. The input remote sensing image is segmented into multiple image patches of the same size and non-overlapping, and each image patch is flattened into a one-dimensional vector. For the input text, preprocessing such as word segmentation is performed, and the text is converted into a fixed-length vector using a unified vocabulary mapping table. The image and text vectors are then fed into an encoder to extract global features and local features of each image patch and word vector.

[0056] (2) Cross-modal feature distance calculation: To establish a fine-grained alignment relationship between images and texts, based on the attention mechanism, select the local partial features that are more important for the image features, further optimize the representation of these local features by introducing residual blocks, and then calculate the cosine similarity of these local features to evaluate the matching degree between images and texts at the fine-grained level. Global features usually contain the core information of images and texts. Evaluating the similarity of global features can provide a rough matching basis for retrieval and help the model identify the most relevant image-text pairs. Calculate the cosine similarity of the global features of the remote sensing image and text to evaluate the matching degree between the image and text as a whole.

[0057] (3) Image-text relevance division: Combine the global and local feature distances of images and texts, and use the Gaussian mixture model to divide the relevance of image-text pairs. Specifically, according to the calculated similarity, perform a two-component Gaussian mixture model modeling, and divide the image-text pairs into positive sample pairs with higher relevance and negative sample pairs with lower relevance according to the posterior probability, providing high-quality labeled samples for subsequent training.

[0058] (4) Weighted triplet loss training: Design a weighted triplet loss function to train the re-partitioned image-text pairs. Traditional triplet loss training methods often ignore the correlation differences between different sample pairs, which may lead to poor performance of the model when dealing with low-correlation samples. Design a weighted triplet loss function that considers all positive and negative sample pairs within a training batch. By weighting the distances of samples with different correlations, the image-text pair samples are differentially processed, which can better improve the robustness and accuracy of the model in complex scenarios.

[0059] (5) Model test and verification: Use the trained model to extract the global and local features of remote sensing images and texts. Add the features of both as the final features to calculate the similarity between the image and the text, and sort according to the similarity to return the image that best matches the query text.

[0060] The core advantage of the embodiments of the present invention is that by introducing multi-level alignment of global and local features, correlation partitioning based on Gaussian mixture models, and weighted triplet loss, the present invention can effectively address the problem of image-text correlation measurement in remote sensing image-text retrieval, optimize the quality of training data and the alignment effect of the model, show stronger robustness in complex scenarios, and have broad application potential. Compared with existing methods, the embodiments of the present invention design a correlation partitioning method based on global and local features, which can more finely partition the relevance between remote sensing images and texts, improve the quality of training data, and enhance model training. The specific innovations are as follows: introducing a multi-level alignment method of global and local features, using Gaussian mixture models to partition the correlation degree of cross-modal sample features, breaking through the limitations of single feature dimension measurement; selecting local features based on the attention mechanism, and then optimizing the cross-modal fine-grained feature representation through residual blocks, solving the problem of insufficient local feature representation, and jointly calculating the cross-modal feature distance with global and local features; designing a dynamically weighted triplet loss function to differentially process samples with different correlations of remote sensing image-text pairs, and optimizing the model's learning ability for complex samples.

[0061] As Figure 1 and Figure 2 shown, the specific implementation steps of the embodiments of the present invention are as follows:

[0062] Step 1, Cross-modal feature extraction

[0063] Feature extraction is the key step to achieve cross-modal alignment. The present invention uses the image and text encoders of the vision-language pre-training model to extract global and local features from images and texts.

[0064] 1.1 Image feature extraction

[0065] Input image I iDivided into N * fixed - size, non - overlapping image patches. Each patch is regarded as a local region of the image. To effectively extract global features, the [CLS] token is introduced as the global feature identifier of the image. Both the image patches and the [CLS] token are input into the image encoder to extract image features:

[0066]

[0067] where d is the dimension of the latent feature space, represents the global feature of the image, and each reflects the features of each local region.

[0068] 1.2 Text Feature Extraction

[0069] For the input text T i , first, the text is converted into a sequence of symbols. The small - case byte - pair encoding method with a vocabulary of 49,152 is used to map each word in the text to an embedding vector. Before inputting the sequence of embedding vectors into the text encoder, the start - of - sequence symbol [SOS] and end - of - sequence symbol [EOS] are added to both ends of the sequence, and then this text sequence is input into the text encoder to extract text features:

[0070]

[0071] where N o represents the length of the input text, represents the feature of each word, and represents the global feature that represents the overall meaning of the text.

[0072] Step 2, Cross - modal Feature Distance Calculation

[0073] For the image - text pair (I i , T j ), the global feature distance is defined as the cosine similarity between the global features of the image and the text:

[0074]

[0075] However, relying solely on global similarity may overlook the subtle but important differences between images and texts. To address this limitation, the present invention introduces residual blocks to calculate fine-grained feature connections. To effectively distinguish global features and local features, the top R proportion of image patch features and word features with high global weight relationships are selected from the last attention layer of the Transformer block in the feature extractor. These selected local features are passed to the residual blocks for further fine-grained feature extraction. Specifically, the relationships between the selected local features are extracted by combining a multi-layer perceptron and a fully connected layer. Then, a max pooling operation is used to aggregate these local features. The process of obtaining the local feature representation of the input image can be expressed by the following mathematical formula:

[0076]

[0077] where, represents the matrix composed of the selected local image patch features. MLP(·) and FC(·) represent the multi-layer perceptron and fully connected layer operations respectively. Similarly, the process of obtaining the fine-grained text representation can be expressed as:

[0078]

[0079] where, represents the matrix composed of the selected word features. Then, the local feature distance of the image-text pair (I i , T j ) is defined as follows:

[0080]

[0081] Step 3, Image-Text Relevance Division

[0082] To address the problem of relevance measurement in remote sensing image-text retrieval, the present invention proposes an image-text relevance division method, which uses a Gaussian Mixture Model (GMM) to divide the image-text pairs based on global and local feature distances into relevant categories. Specifically, it is assumed that the image-text pairs follow a mixture Gaussian distribution. The GMM learns two Gaussian distributions by fitting the feature distance vectors in the training set, and divides the image-text pairs into two categories: high-relevance positive samples and low-relevance negative samples. The GMM is trained using the Expectation-Maximization algorithm to obtain the optimal model parameters:

[0083]

[0084] where, π1 and π2 are the weights of the two Gaussian components, satisfying π1 + π2 = 1, μ1 and μ2 are the means of the two components, and are the variances of the two components.

[0085] After obtaining the Gaussian mixture model, the feature distance vector of each image-text pair is assigned to one of the two categories. For example, considering the global feature distance, the posterior probability of each image-text pair belonging to each category is calculated as follows:

[0086]

[0087] Among them, P(C k ) is Category C k The prior probability, C k ∈{0,1} indicates that the image-text pair belongs to the positive sample set (C k =1) or negative sample set (C k =0), yes Belongs to category C k Based on the maximum posterior probability, the image text pair (I i ,T j ) into the most likely category.

[0088] According to the global feature distance model, the image I in the training batch B i Positive sample set and negative sample sets The definition is as follows:

[0089]

[0090] Among them, τ is the threshold division. Positive sample set Contains image I i Texts with higher semantic matching degree, on the contrary, negative sample sets Contains text with a low semantic match.

[0091] Similarly, we can get the positive sample set based on the local feature distance and negative sample sets based on local feature distance The final set of positive and negative sample pairs is determined by the intersection of the global and local partitioning results:

[0092]

[0093] Similarly, we can also get the text T i The cross-modal positive sample set T i + and negative sample set T i - This fine-grained division allows the model to obtain higher-quality training data, thereby improving the performance of image-text retrieval tasks.

[0094] Step 4: Weighted triplet loss training

[0095] Traditional triple loss functions mainly focus on the distance between an anchor and a single positive or negative sample. However, they ignore the relative positional relationships among all samples. This limitation makes the model vulnerable to noise during training, causing the model to fall into local optimal solutions and reducing its generalization ability. To address this problem, the present invention proposes weighted triple loss, which controls the influence of each sample on model training by introducing a sample distance weighting coefficient. For example, when the distance between the anchor and the negative sample is large, the weight of the negative sample is reduced to minimize its interference in the training process. Conversely, when the distance between the anchor and the negative sample is small, it indicates that the negative sample is similar to the positive sample. In this case, the model pays more attention to these difficult samples to better adjust the decision boundary and prevent overfitting on easily distinguishable samples.

[0096] The formula for the traditional triplet loss function is:

[0097]

[0098] where, [·] + =max(·,0), d(·,·) represents the cosine distance between the inputs, a + (i) is a random positive sample related to a, a - (i) is a random negative sample. m is a margin constant that ensures the distance of the negative sample is greater than that of the positive sample.

[0099] In the weighted triplet loss proposed in the embodiments of the present invention, the distances between all positive and negative sample pairs in a training batch are considered. Each sample pair uses a different weight coefficient to more accurately control the influence on the training process. The expression of the weighted triplet loss function is as follows:

[0100]

[0101] where, α i is the distance weight of each positive sample, β j is the distance weight of each negative sample, exp(x) = e x , ∈ is a temperature hyperparameter that adjusts the sharpness of the probability distribution.

[0102] Step 5, Model verification and retrieval

[0103] After completing the model training, image and text verification and retrieval are performed. Input the test image and text, calculate their feature representations through the trained model, and sort them according to the similarity between the image and the text, and return the text that best matches the query image. The similarity calculation between image I i and text T j is as follows:

[0104]

[0105] Among them, v i is the sum of the global and local features of the image, and t j is the sum of the global and local features of the text. · represents the dot product operation, and the calculation result is the cosine similarity between the image and text features. Based on this similarity, the test images and texts are ranked, and the most matching text is selected for return.

[0106] The effects of the embodiments of the present invention can be further illustrated by the following experiments.

[0107] 1. Simulation conditions

[0108] The present invention conducts simulations on a central processing unit of Intel(R) Xeon(R) Gold 5220R CPU @ 2.20 GHz, with 224G of memory and an Ubuntu 20.04 operating system, using Python and other related toolkits.

[0109] The training and test datasets used in the experiments are RSICD and RSITMD.

[0110] 2. Simulation content

[0111] The experiment is conducted using the method of the present invention according to the following steps:

[0112] First, the network is trained with the training data and tested on the test set. When an input query image is provided, the model returns the text description that best matches the image. When an input query text is provided, the model returns the image that best matches the text.

[0113] Index calculation method: The performance of the retrieval algorithm is evaluated using the R@k and mR metrics. R@k calculates the ratio of the number of correct targets found in the top k retrieval results to the total number of test samples. The present invention reports R@1, R@5, and R@10 in both the cases of text-to-image retrieval and image-to-text retrieval. mR calculates the average of the above six R@k metrics.

[0114] The method of the present invention is compared with other methods. The comparison results on the RSICD dataset are shown in Table 1, and the comparison results on the RSITMD dataset are shown in Table 2.

[0115] Table 1 Comparison of retrieval accuracies of different methods on the RSICD dataset

[0116]

[0117] Table 2 Comparison of retrieval accuracies of different methods on the RSITMD dataset

[0118]

[0119] In Tables 1 and 2, three currently mainstream methods, SMLGN, MSA-CLIP, and PERSVL, are selected for comparison. Among them, SMLGN was proposed in the literature "Y. Chen, J. Huang, S. Xiong, and X. Lu: Integrating multisubspace joint learning with multilevel guidance for cross-modal retrieval of remote sensing images, IEEE Transactions on Geoscience and Remote Sensing, vol. 62, pp. 1–17, 2024". MSA-CLIP was proposed in the literature "R. Yang, S. Wang, Y. Han, Y. Li, D. Zhao, D. Quan, Y. Guo, L. Jiao, and Z. Yang: Transcending fusion: A multi-scale alignment method for remote sensing image-text retrieval, IEEE Transactions on Geoscience and Remote Sensing, 2024". PERSVL was proposed in the literature "X. Tang, D. Huang, J. Ma, X. Zhang, F. Liu, and L. Jiao: Prior-experience-based vision-language model for remote sensing image-text retrieval, IEEE Transactions on Geoscience and Remote Sensing, 2024". It can be found from the results in the table that the present invention can well complete the remote sensing image-text retrieval task.

[0120] Based on the same inventive concept, the present invention further provides a computer device, which includes: one or more processors, and a memory for storing one or more computer programs; the program includes program instructions, and the processor is configured to execute the program instructions stored in the memory. The processor may be a Central Processing Unit (CPU), or may also be other general-purpose processors, Digital Signal Processors (DSPs), Application-Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, and is used to implement one or more instructions, specifically for loading and executing one or more instructions in the computer storage medium to implement the above method.

[0121] It should be further noted that, based on the same inventive concept, the present invention further provides a computer storage medium, on which a computer program is stored, and the computer program, when run by a processor, executes the above method. The storage medium may adopt any combination of one or more computer-readable media. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may, for example, but not be limited to, an electrical, magnetic, optical, electrical, magnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples (non-exhaustive list) of the computer-readable storage medium include: an electrical connection having one or more wires, a portable computer disk, a hard disk, a Random Access Memory (RAM), a Read-Only Memory (ROM), an Erasable Programmable Read-Only Memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, the computer-readable storage medium may be any tangible medium that contains or stores a program, and the program may be used by or combined with an instruction execution system, apparatus, or device.

[0122] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the present invention should have the ordinary meanings understood by those with ordinary skills in the field to which the present invention pertains. The "first", "second" and similar terms used in the present invention do not denote any order, quantity or importance, but are only used to distinguish different components. The terms such as "comprising" or "including" mean that the elements or objects appearing before this word cover the elements or objects listed after this word and their equivalents, without excluding other elements or objects. The terms such as "connected" or "coupled" are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The terms such as "upper", "lower", "left", "right" are only used to represent relative positional relationships, and when the absolute position of the object being described changes, the relative positional relationship may also change accordingly.

[0123] As described above, these are only the preferred embodiments of the present invention, and the present invention is not limited to other forms. Any person skilled in the art may use the technical content disclosed above to make changes or modifications into equivalent embodiments with equivalent changes. However, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the technical solution content of the present invention still fall within the protection scope of the technical solution of the present invention.

[0124] The present invention is not limited to the above best mode. Anyone inspired by the present invention can obtain other various forms of a remote sensing image and text retrieval method based on relevance division. All equal changes and modifications made according to the scope of the patent application of the present invention shall fall within the coverage scope of the present invention.

Claims

1. A remote sensing image and text retrieval method based on relevance division, characterized in that It includes the following steps: Segment the input remote sensing image into multiple image patches and extract global features and local features, tokenize the input text and extract global features and local features to achieve cross-modal feature extraction; Based on the attention weights of the last layer of the Transformer, select the image patch features and word features with the top preset proportion of attention weights; Calculate the global feature similarity between the image and the text respectively, and the local feature similarity optimized by non-linear transformation to complete the cross-modal feature similarity calculation; Based on the global and local feature similarities, use a probability distribution model to divide the image-text pairs, and obtain positive sample pairs and negative sample pairs according to the posterior probability division, where the posterior probability of the positive sample pair is greater than the preset threshold, and the posterior probability of the negative sample pair is less than or equal to the preset threshold; Use a loss function with dynamically adjusted weights to differentially train the model according to the relevance of the sample pairs; Fuse the global and local features to calculate the image-text similarity and return the matching result.

2. The remote sensing image-text retrieval method based on relevance division according to claim 1, wherein: The probability distribution model is a Gaussian mixture model GMM, which specifically includes: Perform two-component Gaussian distribution modeling on the global and local feature similarities; Estimate the mean, variance and weight parameters through the expectation-maximization algorithm; Dynamically divide the positive and negative sample pairs according to the posterior probability.

3. The remote sensing image-text retrieval method based on relevance division according to claim 1, wherein: The loss function with dynamically adjusted weights is a weighted triplet loss function: where α i is the distance weight of each positive sample, β j is the distance weight of each negative sample, exp(x) = e x , ∈ is the temperature hyperparameter that adjusts the sharpness of the probability distribution; [·] + = max(·, 0), d(·, ·) represents the cosine distance between inputs, a + (i) represents the i-th positive sample related to the anchor a, a - (i) represents the i-th negative sample, and the index i traverses all positive and negative samples within a training batch; m is a margin constant used to ensure that the distance of negative samples is greater than that of positive samples.

4. The remote sensing image-text retrieval method based on relevance division according to claim 1, wherein: During the cross-modal feature similarity calculation, the optimization of the local features includes: Perform non-linear transformations on the selected local image patch features and text word vector features through multi-layer perceptrons and fully connected layers respectively; Aggregate the optimized local features through max pooling.

5. A remote sensing image and text retrieval method based on correlation division according to claim 1, characterized in that: The global and local feature similarities are calculated using cosine similarity metrics.

6. The remote sensing image-text retrieval method based on relevance division according to claim 1, wherein: The calculation of the image-text similarity is specifically: where, v i is the sum of the global and local features of the image, t j is the sum of the global and local features of the text, · represents the dot product operation, and the calculation result is the cosine similarity between the image and text features.

7. A remote sensing image and text retrieval method based on correlation division according to claim 4, characterized in that: The non-linear transformation is implemented through a residual block, which specifically includes the joint operation of a multi-layer perceptron and a fully connected layer, and a residual connection with the input features.

8. A remote sensing image-text retrieval system based on relevance division, characterized in that It includes: Feature extraction module: used to segment the input remote sensing image into multiple image patches and extract global features and local features, and tokenize the input text and extract global features and local features; Similarity calculation module: used to calculate the global feature similarity between the image and the text, and the local feature similarity optimized by non-linear transformation; Relevance division module: used to divide the positive and negative sample pairs based on the global and local feature similarities using the Gaussian mixture model GMM; Training module: used to differentially train the model through the weighted triplet loss function; Retrieval module: used to fuse the global and local features to calculate the image-text similarity and return the matching result.

9. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein when the processor executes the program, the steps of the method according to any one of claims 1-7 are implemented.

10. A non-transitory computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1-7 are implemented.

Images