Cross-modal remote sensing image-text matching network based on collaborative learning and matching method thereof

Abstract

The application discloses a cross-modal remote sensing image-text matching network based on cooperative learning and a matching method thereof, and mainly solves the problem of insufficient remote sensing image-text matching capability. The image-text matching network is composed of a feature embedding module, an information interaction enhancement module and a retrieval module. The feature embedding module extracts the final embedding features of the text and the remote sensing image; the information interaction enhancement module performs cooperative learning on the embedding features output by the feature embedding module in terms of the image-text semantic relationship, and enhances the learning of the channel and local information of the remote sensing image, and outputs preliminary image-text interaction features; the retrieval module aligns the preliminary image-text interaction features output by the information interaction enhancement module in terms of the image-text semantic relationship, and outputs the final image-text matching score. The application learns the image-text semantic relationship through the information interaction enhancement module, improves the network performance, further excavates the channel and local information of the remote sensing image, and enhances the generalization performance of the network, and can be used for geographical position search, natural disaster analysis and early warning.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of computer vision and natural language processing, and specifically relates to a cross-modal matching network for remote sensing images and text, which can be used for geographic location search, natural disaster analysis and early warning. Background Technology

[0002] Cross-modal matching of remote sensing images and related text data aims to process information from remote sensing images and related text data, and further realize cross-modal content matching search. That is, given a set of remote sensing images and text, it is necessary to first process the information between the images and text, and then match related text based on the images or match corresponding images based on the text. Its implementation requires the cross-modal matching model to have the ability to understand the semantic content of remote sensing images and text, and to perform image-text matching and correspondence in the cross-modal semantic space.

[0003] With the help of deep learning technology, existing technologies have proposed a variety of cross-modal remote sensing image-text matching methods.

[0004] Yuan Zhiqiang et al. developed an asymmetric cross-modal feature matching network (AMFMN) for cross-modal remote sensing image-text matching tasks at TGRS1–19, 2022. AAMFMN fully considers the multi-scale features of remote sensing images to deeply interpret their diverse information: First, it finds salient regions in remote sensing images by introducing a self-attention method based on multi-scale vision, while simultaneously using remote sensing image information to guide text representation learning; then, it connects the relationships between data from different modalities through a developed triplet loss. The AAMFMN network exhibits satisfactory results on their constructed dataset, suggesting its effectiveness.

[0005] To enhance the feasibility of matching methods, Yuan Zhiqiang et al. proposed a lightweight multi-scale cross-modal matching network, LW-MCR, in TGRS 1–19, 2021. LW-MCR combines a multi-scale feature learning scheme, a rich feature filtering mechanism, a lightweight text representation extraction strategy, and a knowledge extraction strategy to perform cross-modal remote sensing image-text matching.

[0006] Meanwhile, another model from the same group, GaLR (Global and Local Attention Network), was published in TGRS 1–16, 2022. In GaLR, the authors emphasized the importance of visual feature learning in cross-modal remote sensing image-text retrieval. By considering both global and local information when converting remote sensing images into visual features, the complex content hidden within the remote sensing images can be understood and thus used for cross-modal image-text matching.

[0007] All three network models described above are dual-branch networks. One branch learns visual features from remote sensing images, while the other focuses on extracting textual representations from the text. After obtaining cross-modal features, they are mapped to a unified semantic space to directly measure the similarity between remote sensing images and text. The commonly used tool for learning visual features is Deep Convolutional Neural Networks (DCNNs); the commonly used technique for learning textual representations is Recurrent Neural Networks (RNNs), with triplet loss and its variants being common loss functions. Although these dual-branch methods are feasible, their performance still falls short of expectations. There are two main reasons for this: First, since these models typically learn features from remote sensing images and text independently, the semantic gaps between different feature representations are different, which reduces the reliability of similarity measurements. Second, the aforementioned cross-modal remote sensing image-text matching methods primarily emphasize feature learning while neglecting appropriate feature interaction between remote sensing images and text, negatively impacting the discovery of semantic correspondences in cross-modal scenarios.

[0008] The Transformer has recently attracted scholarly attention due to its advantages in uncovering relationships hidden within input data. Currently, Transformer-based models are popular in various computer vision algorithms. On one hand, due to its unique structure, images and text can be input into a unified Transformer network, allowing corresponding visual and textual features to be learned simultaneously. This not only ensures semantic consistency between different features but also, thanks to the Self-Attention (SAM) mechanism, enables in-depth modeling of the relationship between images and text within the Transformer-based model. This facilitates the establishment of semantic correspondences between the two modalities.

[0009] While the aforementioned transformer-based methods have demonstrated their advantages in natural image processing, several challenges remain when applying them to remote sensing images. First, transformer-based methods focus on establishing cross-modal relationships while neglecting image feature learning. This limitation is not significant for natural images but is disastrous for remote sensing images, which contain a wealth of information of different types and scales. Second, when applying self-attention mechanisms to model cross-modal relationships, only the spatial information of the image is considered, while the channel knowledge hidden in the corresponding feature maps is not given due attention. This limits the ability of transformer-based networks to understand remote sensing images, negatively impacting the performance of image-text matching and preventing the achievement of satisfactory accuracy. Summary of the Invention

[0010] The purpose of this invention is to address the shortcomings of the prior art by proposing a cross-modal remote sensing image-text matching network and its matching method based on collaborative learning, so as to improve the accuracy of image-text matching.

[0011] The technical solution of this invention is as follows: It utilizes a transformer to perform feature interaction between images and text, thereby learning the semantic relationship between them. Furthermore, considering the complex content contained in remote sensing images, it enhances the interpretation of remote sensing image information. The implementation is as follows:

[0012] 1. A cross-modal remote sensing image-text matching network based on collaborative learning, characterized in that it includes: a feature embedding module, an information interaction enhancement module, and a retrieval module;

[0013] The feature embedding module includes a text embedding submodule and an image embedding submodule, which are used to obtain the final embedding features of the text and image and output them to the information interaction enhancement module.

[0014] The information interaction enhancement module consists of three serially connected feature interaction enhancement sub-modules. Each feature interaction enhancement sub-module includes an inter-modal relationship interaction unit, a channel angle feature enhancement unit, and a local angle feature enhancement unit. It is used to collaboratively input the embedded features of text and images into the transformer to align the semantic relationship between the text and images. At the same time, it further interprets the information contained in the features of the remote sensing image from both channel and local perspectives and outputs it to the retrieval module.

[0015] The retrieval module includes a multi-modal inter-relation interaction submodule, an image-text matcher submodule, and a cross-entropy loss function. It is composed of features that are output by the information interaction enhancement module and then cross-modal feature interaction is performed to output the final matching score.

[0016] Furthermore, the text embedding submodule consists of Bert's tokenizer and embedder, and text classification tags. Text position embedding matrix and text category discrimination matrix The system comprises components used to map the entire text into its final embedded features, providing a relatively sufficient feature representation for subsequent text understanding and analysis; the image embedding submodule consists of simple linear mapping and image classification labeling. Image position embedding matrix and image category discrimination matrix The composition is used to map an image into a final embedded representation of the image. This embedded feature retains both the visual semantic information of the image and contains the spatial information of the image, providing a more comprehensive and structured feature expression for subsequent understanding and analysis of image information.

[0017] Furthermore, the intermodal relationship interaction unit, used for collaborative learning of cross-modal information based on input image and text features, consists of a transformer containing two normalization layers, one multi-head attention layer, and one multi-layer perceptron layer. Its structure is: first normalization layer → multi-head attention layer → second normalization layer → multi-layer perceptron layer. The input of the first normalization layer is residually connected to the output of the multi-head attention layer, and the input of the second normalization layer is residually connected to the output of the multi-layer perceptron layer. The channel angle feature enhancement unit, used for interpreting and learning the channel information of the remote sensing image, has the same structure as the intermodal relationship interaction unit. The local angle feature enhancement unit, used for learning local information of the remote sensing image using deep convolution to enhance the network's ability to distinguish between background and important regions, includes one normalization layer, three 1×1 convolutional kernels, two 3×3 deep convolutions, and one Gaussian error linear unit.

[0018] Furthermore, the multi-modal inter-relationship interaction submodule is used for deep mining of semantic relationships between images and text. It consists of three transformer layers, with the specific structure as follows: first transformer layer → second transformer layer → third transformer layer. Each transformer contains two normalization layers, one multi-head attention layer, and one multi-layer perceptron layer, with the specific structure as follows: first normalization layer → multi-head attention layer → second normalization layer → multi-layer perceptron layer. The input of the first normalization layer is residually connected to the output of the multi-head attention layer, and the input of the second normalization layer is residually connected to the output of the multi-layer perceptron layer.

[0019] The image-text matcher submodule, used to finally calculate the matching score between the image and text, consists of a multilayer perceptron and an activation function tanh(·). Its specific structure is as follows: First perceptron layer E itm (·) → Activation function tanh(·) → Second perceptron layer E match (·); the cross-entropy loss function The formula is:

[0020]

[0021] Where S i Let m be the matching score of the i-th image-text pair, and m be its image-text relationship label. m = 1 when it is a matching image-text pair, and m = 0 otherwise.

[0022] 2. A method for collaborative learning cross-modal remote sensing image-text matching using the above-mentioned network, characterized by comprising the following steps:

[0023] 1) Obtain the training and test sample sets from the remote sensing image-text matching database, each containing the corresponding image, text data, and image-text relationship labels;

[0024] 2) Obtain the embedding features of text and images through the feature embedding module:

[0025] 2a) Input the text into the text embedding submodule to obtain the final text embedding features;

[0026] 2b) Input the image into the image embedding submodule to obtain the final image embedding features;

[0027] 3) Utilize the information interaction enhancement module to learn multimodal features, align semantic relationships between images and text, and interpret information contained in remote sensing image features:

[0028] 3a) Concatenate text embedding features and image embedding features to generate image-text interaction features, and input these features into the intermodal relationship interaction sub-block to bridge the semantic gap between the two modalities through collaborative learning of interaction features;

[0029] 3b) Separate remote sensing image modal features from the features output by the intermodal relationship interaction sub-blocks;

[0030] 3c) Convert the modal features of the remote sensing image into channel features of the remote sensing image, and input them into the channel angle feature enhancement unit to obtain the global features of the remote sensing image channels, so as to mine the long-distance dependencies in the image channels and further learn the channel information of the remote sensing image;

[0031] 3d) Convert the global features of the remote sensing image channel into local features of the remote sensing image, and input them into the local angle feature enhancement unit to obtain local angle enhancement features of the remote sensing image. Use its depth convolution to capture local information and obtain the ability to distinguish the background or salient regions.

[0032] 3e) The local angle enhancement features of the remote sensing image are first converted into one-dimensional features, and then concatenated with the features separated in 2b) to generate image-text interaction features, which serve as the input of the next round of feature interaction enhancement blocks or the initial image-text interaction features of the overall output of the information interaction enhancement module;

[0033] 4) Input the preliminary image-text interaction features into the multi-modal relationship interaction sub-module to further align the semantic relationships between image-text modalities to obtain the final image-text interaction features, and then pass the final image-text interaction features through the image-text matcher sub-module to obtain the image-text matching score;

[0034] 5) Input the training sample set into the network for forward propagation, obtain its image-text matching score, and input the image-text matching score and the corresponding image-text relationship label into the cross-entropy loss function. In the next step, the backpropagation is used to iteratively update the image-text matching network. The update stops when the maximum number of iterations is reached, so as to obtain an optimized and updated image-text matching network.

[0035] 6) Input the image and text data of the test sample set into the optimized image-text matching network, calculate the matching score between the image and the text, and sort these scores in descending order to obtain the matching results of the image and the text.

[0036] Compared with the prior art, the present invention has the following advantages:

[0037] The image-text matching network model constructed in this invention, by setting up three parts in its information interaction enhancement module—intermodal relationship interaction unit, channel angle feature enhancement unit, and local angle feature enhancement unit—can avoid the shortcomings of existing technologies in neglecting the interaction of image and text information and focusing only on the global information of remote sensing images while ignoring the image channels and local information. This ensures a more comprehensive semantic relationship modeling of image and text semantic relationships and a more complete interpretation of remote sensing image information.

[0038] Meanwhile, because the retrieval module includes a multi-modal inter-relationship interaction submodule and an image-text matcher submodule, the training process avoids the shortcomings of existing technologies that only use distance metrics to obtain image-text matching scores while ignoring the huge semantic gap between images and text, thus effectively improving the accuracy of remote sensing image-text matching.

[0039] Furthermore, since the image embedding submodule in the embedding module only uses lightweight linear mapping to obtain image embedding features, it can avoid the shortcomings of existing technologies that use convolutional neural networks with more parameters to extract image embedding features during training, which leads to increased network redundancy parameters and effectively reduces the number of parameters in the network model. Attached Figure Description

[0040] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings used in the description of the specific embodiments or the prior art will be briefly introduced below.

[0041] Figure 1 This is a diagram of the cross-modal remote sensing image-text matching network structure based on collaborative learning in this invention;

[0042] Figure 2 yes Figure 1 Structure diagram of the feature embedding module;

[0043] Figure 3 yes Figure 1 Structure diagram of the intermodal relationship interaction sub-blocks;

[0044] Figure 4 yes Figure 1Structure diagram of the mid-channel angular feature enhancement unit;

[0045] Figure 5 yes Figure 1 Structure diagram of the local angular feature enhancement unit;

[0046] Figure 6 This invention utilizes Figure 1 A flowchart illustrating the implementation of cross-modal remote sensing image-text matching through collaborative learning on a network. Detailed Implementation

[0047] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0048] Reference Figure 1 This example is based on a cross-modal remote sensing image-text matching network using collaborative learning, including a feature embedding module 1, an information interaction enhancement module 2, and a retrieval module 3. The feature embedding module 1 is used to extract the final embedding representation G of the text. i The final embedding representation F of the image i The two are then concatenated to generate the graphic-text interaction feature X. i =[G i ,F i The information interaction enhancement module 2 is input; the information interaction enhancement module 2 aligns the semantic relationships between the images and text, and at the same time, considering the complex information contained in the remote sensing image features, it enhances the remote sensing image features from the perspectives of channels and localities to achieve a more comprehensive interpretation of the information in the remote sensing images; the features output by the information interaction enhancement module 2 are transmitted to the retrieval module 3 to further align the semantic relationships between the image and text modalities, and obtain the image-text matching score through multilayer perceptron layer mapping;

[0049] Reference Figure 2 The feature embedding module 1 includes a text embedding submodule 11 and an image embedding submodule 12:

[0050] This text embedding submodule 11 consists of Bert's tokenizer and embedder, and text classification tags. Text position embedding matrix and text category discrimination matrix Composition, used to combine text t i The final embedding features mapped to the text as a whole.

[0051] In this example, the text For l = 39 words, the word segmenter will segment t i Convert to an index vector in It is the index vector of the j-th word, with dimension d. o =30522;

[0052] The sequence vector O i via embedder Obtaining word embedding features In this example, d = 768, and the resulting word embedding feature formula is as follows:

[0053] in Let j be the word embedding features of words;

[0054] The obtained word embedding features With a fixed text classification tag Concatenate the data and then add a text position embedding matrix. To obtain the initial embedded representation of the text G' i :

[0055]

[0056] Then, the initial embedding representation of the text G' i Add a text category discrimination matrix Obtain the final embedding features of the text.

[0057] The image embedding submodule 12 consists of a simple linear mapping E image Image classification labeling Image position embedding matrix and image category discrimination matrix Composition, used to transform remote sensing images v i The final embedding representation mapped to the image:

[0058] In this example, the remote sensing image v i The height, width, and channel are respectively set to, but are not limited to, H=384, W=384, and C=3;

[0059] remote sensing image v i Perform the image block division operation and flatten it into n = 144 image blocks, represented as follows:

[0060]

[0061] in For the j-th image patch, in this example, the height and width of the image patch are set to, but not limited to, h = w = 32;

[0062] Image block P i Through learnable linear mappings Mapped to d=768-dimensional visual features The formula is shown below:

[0063] in Visual features of the j-th image patch

[0064] The obtained visual features Image classification tags The images are stitched together, and an image position embedding matrix is ​​added to obtain the initial embedding features F' of the image. i :

[0065]

[0066] Then embed the initial feature F' i and image category discrimination matrix The final embedding features of the image are obtained by adding them together.

[0067] Reference Figure 3 The intermodal relationship interaction unit 21, consisting of a transformer, is used to convert text and image interaction features. Obtain global interactive features of text and images through collaborative learning Its transformer contains two normalization layers, one multi-head attention layer, and one multi-layer perceptron layer. The specific structure is as follows: the first normalization layer → multi-head attention layer → the second normalization layer → multi-layer perceptron layer. The input of the first normalization layer is residually connected to the output of the multi-head attention layer, and the input of the second normalization layer is residually connected to the output of the multi-layer perceptron layer.

[0068] Reference Figure 4 The channel angle feature enhancement unit 22 consists of a transformer, which is used to enhance the interpretation of remote sensing image information from a global channel angle. Its transformer contains two normalization layers, one multi-head attention layer and one multi-layer perceptron layer. The specific structure is as follows: first normalization layer → multi-head attention layer → second normalization layer → multi-layer perceptron layer. The input of the first normalization layer is residually connected to the output of the multi-head attention layer, and the input of the second normalization layer is residually connected to the output of the multi-layer perceptron layer.

[0069] Reference Figure 5 The local angular feature enhancement unit 23 consists of a normalization layer, three 1×1 convolutional layers, two 3×3 depthwise convolutional layers, and a Gaussian error linear activation function unit. It is used to learn local information from remote sensing images and enhance the network's ability to distinguish between the background and important regions of the remote sensing image.

[0070] The local features of the image are normalized through a layer. Standardize and reshape into local two-dimensional features of the image.

[0071] The local two-dimensional features of the image are processed through the first 1×1 convolutional layer. Mapped to first local high-dimensional features The local two-dimensional features of the image are then processed through a second 1×1 convolutional layer. Mapped to second local high-dimensional features

[0072] Local high-dimensional features of the first image Local background features of the image are obtained through a first 3×3 depth convolutional layer and a Gaussian error linear unit. In this example, d u =2042, to suppress non-salient image background information;

[0073] Local high-dimensional features of the second image The input is fed into a second 3×3 depth convolutional layer to obtain local foreground features of the image. To highlight information in salient regions of the image;

[0074] Local background features and local foreground features Perform dot product operations to obtain local image fusion features. This feature C i Contains rich local information;

[0075] Local fusion feature C i The number of channels is reduced by a third 1×1 convolutional layer, and then combined with local two-dimensional features. Addition to obtain local angle enhancement features of the image To enable further learning of local information in an image.

[0076] Reference Figure 6 The method for cross-modal image-text matching using the network in this example is implemented as follows:

[0077] Step 1: Obtain the divided training sample set and test sample set from the remote sensing cross-modal image and text database.

[0078] This example uses four different remote sensing image-text matching datasets: RSICD, RSITMD, UCM-Captions, and Sydney-Captions. The training and testing sets comprise 90% and 10% of their respective datasets. The RSICD dataset contains 9828 training samples and 1093 testing samples; the RSITMD dataset contains 4270 training samples and 483 testing samples; the UCM-Captions dataset contains 2000 training samples and 100 testing samples; and the Sydney-Captions dataset contains 555 training samples and 58 testing samples. Each sample contains an image, five corresponding texts, and an image-text relationship label m. i .

[0079] Step 2: Obtain the final embedding representation G of the text through feature embedding module 1. i The final embedding representation F of the image i .

[0080] 2.1) Transfer the text t i The text embedding submodule 11 is used to map the final embedding feature G of the text as a whole. i ;

[0081] 2.2) Transfer the remote sensing image v i The final embedded representation F of the image is obtained through the image embedding submodule 12. i .

[0082] Step 3: The information interaction enhancement module 2 performs image-text information interaction learning and remote sensing image feature enhancement learning on the output of the feature embedding module 1 to obtain preliminary image-text interaction features.

[0083] 3.1) The final embedded representation G of the text i The final embedding representation F of the image i Splicing to generate interactive features of images and text

[0084] 3.2) Integrate the graphic-text interaction feature X i The first feature interaction enhancement submodule is input to align the semantic relationships between the image and the text. Simultaneously, considering the complex information contained in remote sensing image features, enhancement learning is performed on the remote sensing image features from both channel and local perspectives to achieve a more comprehensive interpretation of the information in the remote sensing image. The process is as follows:

[0085] X i The intermodal relationship interaction unit 21 in the first feature interaction enhancement submodule outputs the global text-image interaction feature. By using a collaborative learning approach based on interactive features, the semantic gap between the two modalities can be bridged, and semantic relationship alignment between images and text can be achieved.

[0086] Global text-image interaction features output from inter-modal relationship interaction sub-block 21 Other features and remote sensing image modal features were separated from the data.

[0087] Convert remote sensing image modal features into remote sensing image channel features. It is then input into the channel angle feature enhancement unit 22 to output the global features of the image channels. By mining long-range dependencies in image channels, further reinforcement learning is performed on the image channels to obtain a better interpretation of image channel information;

[0088] Global features of image channels Convert to local features of remote sensing images The input is then fed into the local angle feature enhancement unit 23, which outputs the local angle enhancement features of the image. It can utilize its depthwise convolution to capture local information, thereby enabling enhanced learning of local image information to better interpret the local information of the image;

[0089] Enhance local angle features of the image The feature is converted into a one-dimensional feature and concatenated with other features in 3.2) to generate the first image-text interaction feature, which serves as the input for the next feature interaction enhancement block.

[0090] 3.3) Input the first graphic-text interaction feature output by the first feature interaction enhancement submodule into the second feature interaction enhancement submodule to obtain the second graphic-text interaction feature. The process and function are the same as in step 3.2).

[0091] 3.4) Input the second graphic-text interaction feature output by the second feature interaction enhancement submodule into the third feature interaction enhancement submodule, and use the preliminary graphic-text interaction feature output by it as the input of the retrieval module 3. The process and function are the same as in step 3.2).

[0092] Step 4: The retrieval module 3 processes the semantic information of the preliminary text-image interaction features output by the information interaction enhancement module 2 to obtain the text-image matching score.

[0093] 4.1) Input the preliminary graphic-text interaction features output by the information interaction enhancement module 2 into the multi-modal inter-modal relationship interaction sub-module 31, align their graphic-text semantic relationships, establish connections between graphic-text semantic relationships, and obtain the final graphic-text interaction features.

[0094] 4.2) Incorporate graphic and text interaction features The input is fed into the image-text matcher submodule 32, and the final image-text matching score S is obtained through a multilayer perceptron mapping. i .

[0095] Step 5: Train the image-text matching network.

[0096] 5.1) Set the initial iteration count to t = 1, and the maximum iteration count to T:

[0097] 5.2) Randomly select N image-text training samples from the training sample set and input them into the image-text matching network to obtain the image-text matching scores S = [S1, ..., S2] of the N image-text training samples. i ,…S N ], where S i The image-text matching score for the i-th image-text sample;

[0098] 5.3) Set the matching score for each image and text to S = [S1, ..., S2]. i ,…S N ] and its corresponding true label vector M = [m1, ..., m i ,…,m N [Through cross-entropy loss function] Calculate the loss value, where m i The image-text relationship label for the i-th sample is used, and this loss value is backpropagated to iteratively update the image-text matching network.

[0099] 5.4) Determine whether t≥T holds true:

[0100] If so, an optimized and updated image-text matching network will be obtained;

[0101] Otherwise, let t = t + 1 and return to 5.2).

[0102] Step 6: Test the image-text matching network, which includes two tasks: matching text and matching images.

[0103] 6.1) Using the I images and T texts contained in the test sample set, form H = I × T image-text sample pairs for the test sample set. Input these H image-text sample pairs into the optimized image-text matching network and calculate the matching score of all image-text pairs in the test sample set. Where S (i-1)×T+t This represents the image-text matching score of the image-text sample composed of the i-th image and the j-th text in the test sample;

[0104] 6.2) Matching score S for the i-th image and all text i· =[S (i-1)×T+1 ,S (i-1)×T+2 ,…,S i×TSort the text in descending order, set the predicted labels of the image-text pairs consisting of the i-th image and the top 10 texts to match, and set the predicted labels of the image-text pairs consisting of the i-th image and the remaining texts to not match, thus completing the text matching.

[0105] 6.2) Matching score S between the j-th text and all images ·j =[S j ,S T+j ,…,S (I-1)×T+j Sort the images in descending order, and set the predicted labels of the image-text pairs consisting of the j-th text and the top 10 images as matching, while setting the predicted labels of the image-text pairs consisting of the j-th text and the remaining images as not matching, thus completing the image matching.

[0106] The technical effects of this invention can be further illustrated by the following simulation experiments.

[0107] I. Simulation Conditions

[0108] All experiments were conducted on two NVIDIA RTX A6000 GPUs with 48GB of memory each.

[0109] The four datasets used in the simulation are the RSICD dataset, the RSITMD dataset, the UCM-Captions dataset, and the Sydney-Captions dataset.

[0110] The RSICD dataset contains 10,921 remote sensing images, each with five matching texts;

[0111] The RSITMD dataset contains 4743 remote sensing images with a resolution of 256×256, and each remote sensing image has five matching texts;

[0112] The UCM-Captions dataset contains 2100 remote sensing images, each with a pixel resolution of 1 foot and a size of 256×256, and each image has five matching texts.

[0113] The Sydney-Captions dataset contains 613 remote sensing images, each 500×500 pixels in size and with a spatial resolution of approximately 0.5 meters. Each image contains five matching texts.

[0114] II. Simulation Content:

[0115] Under the above simulation conditions, remote sensing image-text matching simulations were conducted on four datasets using the present invention and eight existing methods: CLIP, MSIT, AMFMN-soft, AMFMN-fusion, AMFMN-sim, LW-WCR-b, LW-WCR-d, and GaLR. Four simulation experiments were included. To quantitatively illustrate the performance of the proposed network, seven commonly used numerical performance metrics for remote sensing image-text matching tasks were selected to measure the differences between the existing methods and the present invention.

[0116] Simulation 1: Using the present invention and eight existing methods, remote sensing image-text matching was performed on the RSICD dataset. The matching results were evaluated using seven metrics: the first-order accuracy (TR@1), top-5 accuracy (TR@5), and top-10 accuracy (TR@10) for matched text; and the first-order accuracy (IR@1), top-5 accuracy (IR@5), and top-10 accuracy (IR@10) for matched images, as well as the average matching accuracy (mR). The numerical results are shown in Table 1.

[0117] Table 1. Numerical results of the present invention and prior art on the RSICD dataset.

[0118]

[0119] As can be clearly seen from Table 1, compared with eight existing methods, the present invention has higher accuracy and stronger generalization ability in the seven commonly used evaluation metrics on the RSICD dataset, further illustrating the superiority of the network proposed in this invention.

[0120] Simulation 2: Remote sensing image-text matching was performed on the RSITMD dataset using the present invention and eight existing methods. The matching results were evaluated using seven metrics: the first-order accuracy (TR@1), top-5 accuracy (TR@5), and top-10 accuracy (TR@10) for matched text; and the first-order accuracy (IR@1), top-5 accuracy (IR@5), and top-10 accuracy (IR@10) for matched images; and the average matching accuracy (mR). The numerical results are shown in Table 2.

[0121] Table 2. Numerical results of the present invention and prior art on the RSITMD dataset.

[0122]

[0123] As can be clearly seen from Table 2, compared with eight existing methods, the present invention has higher accuracy in seven evaluation metrics on the RSITMD dataset and has a stronger ability to explore text-image matching relationships, further illustrating the superiority of the network proposed in this invention.

[0124] Simulation 3: Using the present invention and eight existing methods, remote sensing image-text matching was performed on the UCM-Captions dataset. The matching results were evaluated using seven metrics: the first-order accuracy (TR@1), top-5 accuracy (TR@5), and top-10 accuracy (TR@10) for the matched text; and the first-order accuracy (IR@1), top-5 accuracy (IR@5), and top-10 accuracy (IR@10) for the matched images; and the average matching accuracy (mR). The numerical results are shown in Table 3.

[0125] Table 3. Numerical results of the present invention and prior art on the UCM-Captions dataset.

[0126]

[0127]

[0128] As can be clearly seen from Table 3, the present invention achieves higher accuracy across seven evaluation metrics compared to eight existing methods on the UCM-Captions dataset, demonstrating a better ability to align semantic relationships between text and images, further illustrating the superiority of the network proposed in this invention.

[0129] Simulation 4: Using the present invention and eight existing methods, remote sensing image-text matching was performed on the Sydney-Captions dataset. The image-text matching results were evaluated for seven evaluation metrics: the first accuracy (TR@1), the top 5 accuracy (TR@5), the top 10 accuracy (TR@10) of the matched text, and the first accuracy (IR@1), the top 5 accuracy (IR@5), the top 10 accuracy (IR@10), and the average matching accuracy (mR) of the matched images. The numerical results are shown in Table 4.

[0130] Table 4. Numerical results of the present invention and prior art on the Sydney-Captions dataset.

[0131]

[0132]

[0133] As can be clearly seen from Table 4, the average matching accuracy mR of the present invention is consistently better than that of the eight existing methods on the Sydney-Captions dataset, demonstrating the better generalization performance of the present invention and indicating the effectiveness and superiority of the proposed remote sensing image-text matching network.

[0134] The sources of the eight prior art techniques mentioned above:

[0135] CLIP is a network for image-text matching published by Radford et al. at PMLR, namely: A. Radford, JW Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al., “Learning transferable visual models from natural language supervision,” in International Conference on Machine Learning. PMLR, 2021, pp. 8748–8763;

[0136] MSIT is a network for remote sensing image-text matching published by Wang Yijing et al. in IEEE IGARSS, namely: Y.Wang, J.Ma, M.Li, X.Tang, X.Han, and L.Jiao, “Multi-scale interactive transformer for remote sensing cross-modal image-text retrieval,” in IGARSS 2022-2022 IEEE International Geoscience and Remote Sensing Symposium. IEEE, 2022, pp. 839–842;

[0137] AMFMN-soft, AMFMN-fusion, and AMFMN-sim are networks for remote sensing image matching published by Zhiqiang Yuan et al. in IEEE TGRS, namely: Z. Yuan, W. Zhang, K. Fu, X. Li, C. Deng, H. Wang, and X. Sun, “Exploring a fine-grained multiscale method for cross-modal remote sensing image retrieval,” IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1–19, 2022;

[0138] LW-WCR-b and LW-WCR-d are networks for remote sensing image-image matching published by Zhiqiang Yuan et al. in IEEE TGRS, namely: Z.Yuan, W.Zhang, X.Rong, X.Li, J.Chen, H.Wang, K.Fu, and X.Sun, “A lightweight multi-scale crossmodal text-image retrieval method in remote sensing,” IEEE Transactions on Geoscience and Remote Sensing, vol.60, pp.1–19, 2021;

[0139] GaLR is a network for remote sensing text-image matching published by Zhiqiang Yuan et al. in IEEE TGRS, namely: Z.Yuan,W.Zhang,C.Tian,X.Rong,Z.Zhang,H.Wang,K.Fu,andX.Sun,“Remote sensing cross-modal text-image retrieval based on global and local information,”IEEE Transactions on Geoscience and Remote Sensing,vol.60,pp.1–16,2022;

[0140] The first-order accuracy (TR@1) of the above-mentioned text matching is as follows: After completing the text matching, for each image, 10 image-text pairs related to the image and predicted as matching are obtained, and the image-text relationship label of the image-text pair with the highest matching score is further obtained; for all images, the number of images with the highest matching score image-text pair and the image-text relationship label are matched is counted, and this number is divided by the total number of images. This value is the first-order accuracy (TR@1) of the text matching, which is between 0 and 100%. The larger this value is, the better the matching effect.

[0141] The top 5 accuracy (TR@5) for matching text is calculated as follows: After matching the text, for each image, 10 image-text pairs related to the image and predicted as matching are obtained. The image-text relationship labels of the top 5 matching image-text pairs are then obtained. For all images, the number of images with matching images in the image-text relationship labels of the top 5 image-text pairs is counted and divided by the total number of images. This value is the top 5 accuracy (TR@5) for matching text, which ranges from 0 to 100%. The larger the value, the better the matching effect.

[0142] The top 10 accuracy (TR@10) for matching text is calculated as follows: After matching the text, for each image, obtain 10 image-text pairs that are related to the image and whose predicted labels are matched, and obtain the image-text relationship labels for the 10 image-text pairs; for all images, count the number of images that match in the top 10 image-text relationship labels, and divide it by the total number of images. This value is the top 10 accuracy (TR@10) for matching text, which is between 0 and 100%. The larger the value, the better the matching effect.

[0143] The first-order accuracy (IR@1) of the above-mentioned image matching is as follows: After completing the image matching, for each text, 10 image-text pairs are obtained, consisting of images related to the text and whose predicted labels are matched. The image-text relationship label of the image-text pair with the highest matching score is obtained. For all texts, the number of texts with the highest matching score image-text relationship label is counted and divided by the total number of texts. This value is the first-order accuracy (IR@1) of the image matching, which is between 0 and 100%. The larger this value is, the better the matching effect.

[0144] The top 5 accuracy (IR@5) for the above-mentioned matching images is as follows: After completing the matching of images, for each text, 10 image-text pairs are obtained, consisting of images related to the text and whose predicted labels are matched. The image-text relationship labels of the top 5 sample pairs with the highest matching scores are obtained. For all texts, the number of texts that match in the top 5 image-text relationship labels is counted and divided by the total number of texts. This value is the top 5 accuracy (IR@5) for the matching images. Its value is between 0 and 100%. The larger this value is, the better the matching effect.

[0145] The top 10 accuracy (IR@10) for the above-mentioned matching images is as follows: After completing the matching of images, for each text, 10 image-text pairs are obtained, consisting of images related to the text and whose predicted labels are matched. The image-text relationship labels of the 10 image-text pairs are also obtained. For all texts, the number of texts that match in the 10 obtained image-text relationship labels is counted and divided by the total number of texts. This value is the top 10 accuracy (IR@10) for the matching images. Its value is between 0 and 100%. The larger this value is, the better the matching effect.

[0146] The average matching accuracy mR is calculated by taking the average of the following six evaluation metrics: TR@1 (first-place accuracy for matching text), TR@5 (top 5 accuracy for matching text), TR@10 (top 10 accuracy for matching text), IR@1 (first-place accuracy for matching text), IR@5 (top 5 accuracy for matching text), and IR@10 (top 10 accuracy for matching images). The value of mR is between 0 and 100%, and a higher value indicates a better matching effect.

Claims

1. A cross-modal remote sensing image-text matching network model based on collaborative learning, characterized in that, Comprise: Feature embedding module (1), information interaction enhancement module (2) and retrieval module (3); The feature embedding module (1) comprises a text embedding submodule (11) and an image embedding submodule (12) for obtaining the final embedding features of text and image, and outputting to the information interaction enhancement module (2); The information interaction enhancement module (2) comprises three feature interaction enhancement submodules connected in series, each of which comprises an inter-modal relationship interaction unit (21), a channel angle feature enhancement unit (22) and a local angle feature enhancement unit (23) for inputting the embedding features of text and image into the transformer for aligning the semantic relationship between image and text, and further interpreting the information contained in the features of remote sensing image from the channel and local aspects, and outputting to the retrieval module (3); The inter-modal relationship interaction unit (21) is used for learning cross-modal information by inputting image and text features cooperatively, which comprises a transformer containing two layers of normalization layers, one multi-head attention layer and one multi-layer perception layer, and the structure is: first layer normalization layer→multi-head attention layer→second layer normalization layer→multi-layer perception layer, and the input of the first normalization layer and the output of the multi-head attention layer are connected in residual error, and the input of the second normalization layer and the output of the multi-layer perception layer are connected in residual error, The channel angle feature enhancement unit (22) is used for interpreting and learning the channel information of remote sensing image, and the structure is the same as that of the inter-modal relationship interaction unit (21); The local angle feature enhancement unit (23) is configured to enhance the ability of the network to distinguish between background and important regions by using deep convolution learning to remotely sense the local information of an image, and includes one layer normalization layer, three convolution kernels, two depth convolutions, and one Gaussian error linear unit. The retrieval module (3) comprises a multi-layer inter-modal relationship interaction submodule (31), an image-text matcher submodule (32) and a cross-entropy loss function Composition, for cross-modal feature interaction of the features output by the information interaction enhancement module, and output of the final matching score; The multi-layer inter-modal relationship interaction submodule (31) is used for deeply mining the semantic relationship between image and text, and narrowing the semantic gap of cross-modal image and text; Each transformer comprises two layers of normalization layers, one multi-head attention layer and one multi-layer perception layer, and the structure is: first layer normalization layer→multi-head attention layer→second layer normalization layer→multi-layer perception layer, and the input of the first normalization layer and the output of the multi-head attention layer are connected in residual error, and the input of the second normalization layer and the output of the multi-layer perception layer are connected in residual error; The image-text matcher sub-module (32) is composed of multiple perceptron layers and activation functions The first perceptron layer is composed of The activation function is composed of The second perceptron layer is composed of The final output image and text matching score is used as the final retrieval result ranking score. The cross-entropy loss function , is given by: ； wherein is the first the matching score of the image-text pair is the image-text relationship label of the image-text pair, when it is a matching image-text pair otherwise .

2. The network model of claim 1, comprising the following steps: The text embedding submodule (11) is composed of a tokenizer and an embedder of Bert, text classification labels , a text position embedding matrix , and a text category discrimination matrix , for mapping the text as a whole into the final embedding features of the text, providing relatively sufficient feature expression for subsequent text understanding and analysis. The image embedding sub-module (12) is composed of a simple linear mapping, an image classification label , an image position embedding matrix , and a discriminant matrix of image categories , for mapping the image into a final embedding representation of the image. The embedding features not only retain the visual semantic information of the image, but also contain the spatial information of the image, providing a more comprehensive and structured feature expression for subsequent understanding and analysis of image information.

3. A method for cross-modal remote sensing image-text matching using collaborative learning based on the network model of claim 1, characterized in that, 1) obtaining training sample set and test sample set from remote sensing image-text matching database, each containing corresponding image, text data and image-text relationship label; 2) obtaining embedding features of text and image by feature embedding module (1): 2a) inputting text into text embedding submodule (11) to obtain text final embedding features; 2b) inputting image into image embedding submodule (12) to obtain image final embedding features; 3) learning multi-modal features by information interaction enhancement module (2) to align the semantic relationship between image and text, and interpret the information contained in the features of remote sensing image: ​ 3a) concatenate the text embedding features and the image embedding features to generate the image-text interaction features, and input the features into the inter-modal relationship interaction sub-block (21) to bridge the semantic gap between the two modalities through collaborative learning of the interaction features; 3b) separate the remote sensing image modal features from the features output by the inter-modal relationship interaction sub-block (21); 3c) convert the remote sensing image modal features into remote sensing image channel features, and input the features into the channel-angle feature enhancement unit (22) to obtain remote sensing image channel global features, so as to mine long-distance dependencies in the image channel and further learn the channel information of the remote sensing image; 3d) convert the remote sensing image channel global features into remote sensing image local features, and input the features into the local-angle feature enhancement unit (23) to obtain remote sensing image local-angle enhanced features, which utilize deep convolution to capture local information and obtain the ability to distinguish background or salient regions; 3e) convert the remote sensing image local-angle enhanced features into one-dimensional features, and then concatenate the features with the features separated in 2b) to generate image-text interaction features, which are used as the input of the next round of feature interaction enhancement block or the preliminary image-text interaction features output by the information interaction enhancement module (2) as a whole; 4) input the preliminary image-text interaction features into the multi-layer inter-modal relationship interaction sub-module (31) to further align the semantic relationship between the image and text modalities to obtain the final image-text interaction features, and then input the final image-text interaction features into the image-text matcher sub-module (32) to obtain the image-text matching score; 5) input the training sample set into the network to obtain the image-text matching score thereof, and input the image-text matching score and the corresponding image-text relationship label into a cross-entropy loss function Further, the image-text matching network is updated iteratively in combination with back propagation, and the updating is stopped when the maximum number of iterations is reached, so as to obtain the optimized and updated image-text matching network. 6) input the image and text data of the test sample set into the optimized image-text matching network, calculate the matching score between the image and the text, and sort the scores in descending order to obtain the predicted matching label of the image and the text.

4. The method of claim 3, wherein, Step 3a) input the image-text interaction features into the inter-modal relationship interaction sub-block (21) for collaborative learning, which is implemented as follows: 3a1 ) obtaining the text-image interaction features obtaining standardized text-image interaction features by a first layer of normalization layers ; 3a2) Collaborative processing of standardized text-image interaction features by multi-head attention layers Modeling text-image semantic relationships to obtain text-image information interaction features ; 3a3) the sum of and is input into the second layer normalization layer and the multi-layer perception layer to obtain the image-text global interaction feature , further learning the global relationship between the image-text features, and taking as the output of the inter-modal relationship interaction sub-block (21).

5. The method of claim 3, wherein, In step 3c), the remote sensing image channel features are further learned by the channel-angle feature enhancement unit (22) to learn the channel information of the remote sensing image, which is implemented as follows: 3c1) image channel features obtaining standardized image channel features by a first layer normalization layer ; 3c3) output image channel augmented features through one layer of multi-head attention layers to model relationships between image channels; 3c4) the sum of and is input into the second layer normalization layer and the multi-layer perception layer in turn, further enhancing the interpretation of remote sensing image information from the channel global perspective, obtaining image channel global features and as the output of the channel angle feature enhancement unit (22).

6. The method of claim 3, wherein, In step 3d), the local-angle feature enhancement unit (23) utilizes deep convolution to capture local information, which is implemented as follows: 3d1) image local features After layer normalization, reshaped to image local two-dimensional features ; 3d2) using two convolutions will map as two image local high dimensional features and ; 3d3) to input to one deep convolution and a gaussian error linear unit to obtain local background features of the image to achieve suppression of non-significant image background information; 3d4) to input to another deep convolution obtains local foreground features of the image to highlight the salient region information of the image; 3d5) to and perform dot product operation to obtain image local fusion features , the features contain rich local information; 3d6) to by one convolution to reduce the number of channels, and addition to obtain the image local angle enhancement feature as the output of the local angle feature enhancement unit (23).

7. The method of claim 3, wherein, The final image-text interaction feature in step 4) obtains the image-text matching score through the image-text matcher submodule (32), which is to input the final image-text interaction feature to the first layer of the perception machine layer of the module and the activation function to obtain the image-text matching feature ; Matching the image and text features again a second layer of perceptual machines input to the module to obtain a final image-text matching score .

8. The method of claim 3, wherein, In step 5) a cross-entropy loss function is utilized In combination with the back propagation iterative updating of the image-text matching network, the following is realized 5a) initialize the iteration number to , the maximum iteration number to , and let : 5b) forward-propagating a randomly selected text-image training sample from the training sample set as input to the network model to obtain a set of predicted label vectors wherein represents the predicted label vector corresponding to the th training sample. 5c) using a cross-entropy loss function computing the relationship between each predicted label vector and its corresponding real label vector wherein represents the image-text relationship label corresponding to the th training sample, and using gradient descent method, the weight parameters of the image-text matching network are updated through the loss function value . 5d) judging whether the following holds: If yes, the optimized and updated image-text matching network is obtained, Else if , return 5b).

Images