Cross-modal remote sensing image retrieval method and device, electronic equipment and storage medium

Abstract

The application discloses a cross-modal remote sensing image retrieval method and device, electronic equipment and storage medium, and belongs to the technical field of remote sensing image processing. The method comprises the following steps: acquiring remote sensing image data sets; constructing a retrieval model; inputting optical image data sets into the retrieval model to obtain optical target domain fusion features; inputting radar image data sets into the retrieval model to obtain radar target domain fusion features; performing semantic alignment on the optical target domain fusion features and the radar target domain fusion features according to an alignment module to obtain aligned radar target domain fusion features; calculating a loss value between optical hash codes and radar hash codes according to a loss function module, and determining a retrieval model with the minimum loss value as a target retrieval model; and performing retrieval based on the target retrieval model. Through the cross-modal image generation module, the image is subjected to cross-modal processing, and the features of two modes are fused to realize retrieval based on the fused features, so that cross-modal remote sensing image retrieval is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing image processing, and in particular to a cross-modal remote sensing image retrieval method, device, electronic device and storage medium. Background Art

[0002] The rapid development of sensing technology has provided a vast amount of optical, hyperspectral, and synthetic aperture radar (SAR) imagery. Optical and SAR imagery demonstrate significant advantages in cross-modal retrieval. However, due to the different imaging mechanisms of optical and SAR images, target representation differs significantly, increasing the difficulty of retrieval. Specifically, optical images are electromagnetic reflection images, allowing direct observation of the brightness, texture, and structure of targets and scenes. However, this imaging method is susceptible to weather conditions. When clouds, fog, or water vapor accumulate on the signal propagation path, the image may be inaccurate or even obscured. SAR imagery, on the other hand, is microwave-based and utilizes SAR satellites capable of overcoming the limitations of cloud, fog, rain, snow, and darkness, enabling all-day, all-weather, ground-to-ground observations. However, due to the complex signal processing involved in the SAR imaging process, SAR images can suffer from speckle noise, which reduces image interpretation capabilities.

[0003] Optical images and synthetic aperture radar images are two different modal data in different image spaces. During the image retrieval process, existing technologies target images in the same image space and the same modality. The retrieval results are relatively simple and cannot meet the requirements of cross-modal remote sensing image retrieval.

[0004] Therefore, in the process of cross-modal remote sensing image retrieval, the existing technology has the problem of difficulty in realizing cross-modal remote sensing image retrieval. Summary of the Invention

[0005] In view of this, it is necessary to provide a cross-modal remote sensing image retrieval method, device, electronic device and storage medium to solve the problem of difficulty in realizing cross-modal remote sensing image retrieval in the process of cross-modal remote sensing image retrieval in the existing technology.

[0006] In order to solve the above problems, the present invention provides a cross-modal remote sensing image retrieval method, comprising:

[0007] Acquire a remote sensing image dataset, which includes an optical image dataset and a radar image dataset;

[0008] Build a retrieval model, including a cross-modal image generation module, a feature fusion module, an alignment module, a hash layer, and a loss function module;

[0009] The optical image dataset is input into the retrieval model, the radar target domain is generated based on the cross-modal image generation module, and the radar target domain and the optical image dataset are fused based on the feature fusion module to obtain the optical target domain fusion feature;

[0010] The radar image dataset is input into the retrieval model, and the optical target domain is generated based on the cross-modal image generation module. The optical target domain and the radar image dataset are fused based on the feature fusion module to obtain the radar target domain fusion features.

[0011] According to the alignment module, the optical target domain fusion features and the radar target domain fusion features are semantically aligned to obtain the aligned radar target domain fusion features;

[0012] The hash layer calculates the optical hash code and radar hash code based on the optical target domain fusion features and the aligned radar target domain fusion features, respectively. The loss value between the optical hash code and the radar hash code is calculated according to the loss function module. After iterative training, the retrieval model with the minimum loss value is determined as the target retrieval model.

[0013] The optical image / radar image to be retrieved is retrieved based on the target retrieval model to obtain the corresponding radar image data / optical image data.

[0014] In one possible implementation, the cross-modal image generation module includes a denoising network submodule and an autoencoder submodule. Generating a radar target domain based on the cross-modal image generation module includes:

[0015] Denoising the optical image dataset based on the denoising network submodule to obtain a denoised optical image;

[0016] In the latent space, the denoised optical image is encoded and decoded based on the autoencoder sub-module to reconstruct the radar target domain of the optical image dataset.

[0017] In one possible implementation, the feature fusion module includes a deep learning module and a fusion submodule. Based on the feature fusion module, the radar target domain and the optical image dataset are subjected to feature fusion to obtain optical target domain fusion features, including:

[0018] Based on the deep learning module, key features of the radar target domain and optical image dataset are extracted to obtain the key features of the radar target domain and the key features of the optical image dataset respectively;

[0019] Based on the fusion submodule, the key features of the radar target domain and the key features of the optical image dataset are fused to obtain the optical target domain image fusion features.

[0020] In one possible implementation, the deep learning module includes a multi-scale residual module and a contextual attention module. Key features of the radar target domain and the optical image dataset are extracted based on the deep learning module to obtain key features of the radar target domain and key features of the optical image dataset, respectively, including:

[0021] Adaptively detect radar target domain and optical image datasets based on the multi-scale residual module to obtain multi-scale residual radar features and multi-scale residual optical features respectively;

[0022] According to the contextual attention module, the multi-scale residual radar features and multi-scale residual optical features are targeted and the key features of the radar target domain and the key features of the optical image dataset are obtained respectively.

[0023] In one possible implementation, semantic alignment is performed on the optical target domain fusion features and the radar target domain fusion features according to the alignment module to obtain aligned radar target domain fusion features, including:

[0024] The similarity between the fusion feature vector of the optical target domain and the fusion feature vector of the radar target domain is calculated based on the similarity calculation formula;

[0025] Adjust the radar feature weights of radar target domain fusion features according to similarity;

[0026] The radar target domain fusion features are aligned according to the radar feature weights to obtain the aligned radar target domain fusion features.

[0027] In one possible implementation, the loss function module includes a semantic alignment loss function, an orthogonality constraint loss function, and a hash quantization loss function; and calculating the loss value between the optical hash code and the radar hash code according to the loss function module includes:

[0028] Calculate the semantic alignment loss between the optical hash code and the radar hash code according to the semantic alignment loss function;

[0029] The orthogonal constraint loss between the optical target domain fusion features and the radar target domain fusion features is calculated according to the orthogonal constraint loss function;

[0030] Calculating the hash quantization loss between the optical hash code and the radar hash code according to the hash quantization loss function;

[0031] The loss value is determined based on semantic alignment loss, orthogonality constraint loss, and hash quantization loss.

[0032] In one possible implementation, calculating the hash quantization loss between the optical hash code and the radar hash code according to the hash quantization loss function includes:

[0033] The initial hash quantization loss function is updated by a smoothing function to obtain a hash quantization loss function;

[0034] Among them, the calculation formula of the hash quantization loss function is:

[0035]

[0036] Quantify the loss for hashing, l is the dimension of the feature vector, i For the i samples, K is the number of hash code bits, t For the t hash code, cosh x express x The hyperbolic cosine function, ∑ represents the sum, Indicates the i samples and t The feature vector corresponding to the hash code.

[0037] In order to solve the above problems, the present invention further provides a cross-modal remote sensing image retrieval device, comprising:

[0038] A data set acquisition module is used to acquire a remote sensing image data set, which includes an optical image data set and a radar image data set;

[0039] Retrieval model construction module, used to build the retrieval model, including cross-modal image generation module, feature fusion module, alignment module, hash layer and loss function module;

[0040] The optical target domain fusion feature module is used to input the optical image dataset into the retrieval model, generate the radar target domain based on the cross-modal image generation module, and fuse the radar target domain and the optical image dataset based on the feature fusion module to obtain the optical target domain fusion feature;

[0041] The radar target domain fusion feature module is used to input the radar image dataset into the retrieval model, generate the optical target domain based on the cross-modal image generation module, and fuse the features of the optical target domain and the radar image dataset based on the feature fusion module to obtain the radar target domain fusion feature;

[0042] A semantic alignment module is used to semantically align the optical target domain fusion features and the radar target domain fusion features according to the alignment module to obtain aligned radar target domain fusion features;

[0043] In the target retrieval model determination module, the hash layer calculates the optical hash code and the radar hash code based on the optical target domain fusion features and the aligned radar target domain fusion features, respectively. The loss function module calculates the loss value between the optical hash code and the radar hash code. After iterative training, the retrieval model with the minimum loss value is determined as the target retrieval model.

[0044] The retrieval module is used to retrieve the optical image to be retrieved / the radar image to be retrieved based on the target retrieval model to obtain the corresponding radar image data / optical image data.

[0045] In order to solve the above problem, the present invention further provides an electronic device, comprising a memory and a processor, wherein:

[0046] Memory, used to store programs;

[0047] The processor is coupled to the memory and is configured to execute the program stored in the memory to implement the steps in the cross-modal remote sensing image retrieval method as described above.

[0048] In order to solve the above problems, the present invention also provides a computer-readable storage medium for storing computer-readable programs or instructions. When the program or instructions are executed by a processor, the steps in the cross-modal remote sensing image retrieval method described above can be implemented.

[0049] The beneficial effect of adopting the above embodiment is as follows: the present invention provides a cross-modal remote sensing image retrieval method, first, a radar target domain / optical target domain is generated by a cross-modal image generation module, thereby realizing cross-modal processing of input data; then, the radar target domain / optical target domain and the optical image dataset / radar image dataset are subjected to feature fusion by a feature fusion module, thereby obtaining an optical target domain fusion feature / radar target domain fusion feature that includes both optical and radar features; next, the optical target domain fusion feature and the radar target domain fusion feature are semantically aligned, thereby fully capturing the complex semantics of the cross-modal remote sensing image and ensuring that the features of images of different modalities are more consistent at the feature level; finally, by using the loss value as a training quantitative indicator of the retrieval model, it is ensured that the model can accurately represent the cross-modal image similarity and ensure the reliability of cross-modal remote sensing image retrieval. Since the retrieval model can process image data of two modalities and output the required remote sensing image retrieval results accordingly, cross-modal remote sensing image retrieval is realized. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 A schematic diagram of a flow chart of an embodiment of a cross-modal remote sensing image retrieval method provided by the present invention;

[0051] Figure 2 A schematic diagram of the structure of an embodiment of a retrieval model provided by the present invention;

[0052] Figure 3 Structure diagram of an embodiment of the alignment module structure provided by the present application;

[0053] Figure 4 Result diagram of an embodiment of the intra-class similarity classification result provided by the present application;

[0054] Figure 5 Structure block diagram of an embodiment of the cross-modal remote sensing image retrieval device provided by the present application;

[0055] Figure 6 Structure block diagram of an embodiment of the electronic device provided by the present application. DETAILED DESCRIPTION

[0056] The preferred embodiments of the present application will be described in detail below with reference to the accompanying drawings, which form a part of this application. The accompanying drawings and the embodiments together illustrate the principles of the present application, but are not intended to limit the scope of the present application.

[0057] Remote sensing images are images of the earth's surface obtained by remote sensing platforms such as airplanes and satellites using different types of sensors such as optical cameras, multispectral cameras, infrared cameras, and radars under remote conditions.

[0058] Target domain (TD) images refer to test sample images to be classified or recognized in a small amount or even no labeled sample dataset. These images have different data distributions from source domain images, but usually belong to the same task or problem domain. Target domain images play an important role in the fields of machine learning, deep learning, and computer vision. By reasonably utilizing the knowledge and information in source domain images, the challenges of data scarcity, domain difference, and labeling difficulty can be effectively addressed, further improving the accuracy and efficiency of target domain image classification or recognition.

[0059] Source domain images are an important concept in the fields of machine learning such as transfer learning and domain adaptation, used to refer to labeled or unlabeled image datasets used in the training phase, which have different data distributions from target domain images but belong to the same task or problem domain.

[0060] Domain alignment refers to connecting the source domain and the target domain through certain methods, so that the model can achieve good performance on both the source domain and the target domain. In machine learning, the source domain and the target domain usually refer to different datasets, which may have different data distributions or feature representations. Due to the difference in data distribution, directly extending the performance of the model on the source domain to the target domain may not be ideal. Therefore, domain alignment is needed to reduce the difference between the source domain and the target domain.

[0061] The loss function, also known as the cost function, is a non-negative, real-valued function whose inputs are the model's predicted values ​​and the actual observed values. Its output is a scalar representing the difference between the predicted and actual values. The smaller the difference, the more accurate the model's predictions. During model training, by minimizing the loss function, we can continuously optimize the model's parameters, thereby improving its predictive performance.

[0062] Backpropagation is one of the most core and commonly used optimization algorithms in deep learning, widely used in the training of neural networks. Based on the gradient descent method, it is a learning algorithm suitable for multi-layer neural networks. The basic principle of the backpropagation algorithm is to calculate the gradient of the loss function with respect to the network parameters in order to update the parameters to minimize the loss function.

[0063] A hash code is an algorithm that ensures that objects of the same class have unique hash codes based on their characteristics. However, different objects may have the same hash code, depending on the specific implementation of the hash code algorithm. In programming languages ​​like Java, hash codes are commonly used for fast lookups and duplicate removal, for example in hash tables and collection classes like HashMap and HashSet.

[0064] Remote sensing image retrieval technology leverages visual, object and spatial relationships, and semantic hierarchies to achieve fast and accurate image retrieval for tasks such as environmental and disaster monitoring. The scope of the retrieval directly determines the breadth of information obtained. However, existing image retrieval technologies typically target images in the same image space and modality, resulting in relatively limited retrieval results and failing to meet the requirements of cross-modal remote sensing image retrieval.

[0065] Therefore, in the process of cross-modal remote sensing image retrieval, the existing technology has the problem of difficulty in realizing cross-modal remote sensing image retrieval.

[0066] In order to solve the above problems, the present invention provides a cross-modal remote sensing image retrieval method, device, electronic device and storage medium, which are described in detail below.

[0067] Figure 1 This is a flow chart of an embodiment of the cross-modal remote sensing image retrieval method provided by the present invention, as shown in FIG. Figure 1 As shown in Figure 2, the cross-modal remote sensing image retrieval method includes:

[0068] S101: Acquire a remote sensing image dataset, where the remote sensing image dataset includes an optical image dataset and a radar image dataset;

[0069] S102: Construct a retrieval model, including a cross-modal image generation module, a feature fusion module, an alignment module, a hash layer, and a loss function module;

[0070] S103: Input the optical image dataset into the retrieval model, generate a radar target domain based on the cross-modal image generation module, and fuse features of the radar target domain and the optical image dataset based on the feature fusion module to obtain optical target domain fusion features;

[0071] S104: Inputting the radar image dataset into the retrieval model, generating an optical target domain based on the cross-modal image generation module, and performing feature fusion on the optical target domain and the radar image dataset based on the feature fusion module to obtain radar target domain fusion features;

[0072] S105: semantically aligning the optical target domain fusion features and the radar target domain fusion features according to the alignment module to obtain aligned radar target domain fusion features;

[0073] S106: The hash layer calculates the optical hash code and the radar hash code based on the optical target domain fusion features and the aligned radar target domain fusion features, respectively. The loss value between the optical hash code and the radar hash code is calculated using the loss function module. The training is iteratively performed to determine the retrieval model with the minimum loss value as the target retrieval model.

[0074] S107: Retrieve the optical image / radar image to be retrieved based on the target retrieval model to obtain corresponding radar image data / optical image data.

[0075] In this embodiment, first, a radar target domain / optical target domain is generated by a cross-modal image generation module, thereby realizing cross-modal processing of the input data; then, the radar target domain / optical target domain and the optical image dataset / radar image dataset are subjected to feature fusion by a feature fusion module to obtain optical target domain fusion features / radar target domain fusion features that include both optical and radar features; next, semantic alignment is performed on the optical target domain fusion features and the radar target domain fusion features to fully capture the complex semantics of the cross-modal remote sensing image and ensure that the features of images of different modalities are more consistent at the feature level; finally, the loss value is used as a training quantitative indicator for the retrieval model to ensure that the model can accurately represent the cross-modal image similarity and ensure the reliability of cross-modal remote sensing image retrieval. Since the retrieval model can process image data of two modalities and output the required remote sensing image retrieval results accordingly, cross-modal remote sensing image retrieval is realized.

[0076] It should be noted that the radar image in this application refers to a synthetic aperture radar image (SAR image).

[0077] As a preferred embodiment, in S101, the remote sensing image dataset comes from a public remote sensing image dataset, and the collected remote sensing image data can also be manually screened and grouped according to actual needs to meet the personalized needs of subsequent model training.

[0078] As a preferred embodiment, in S102, in order to clearly express the relationship between the cross-modal image generation module, the feature fusion module, the alignment module, the hash layer and the loss function module, as shown in FIG. Figure 2 As shown, Figure 2 This is a structural diagram of an embodiment of the retrieval model provided by the present invention.

[0079] As a preferred embodiment, in S103, the cross-modal image generation module includes a denoising network submodule and an autoencoder submodule; in order to generate a radar target domain based on the cross-modal image generation module, first, the optical image dataset is denoised based on the denoising network submodule to obtain a denoised optical image; then, in the latent space, the denoised optical image is encoded and decoded based on the autoencoder submodule to reconstruct the radar target domain of the optical image dataset.

[0080] Similarly, in order to generate the optical target domain based on the cross-modal image generation module, the radar image dataset is first denoised based on the denoising network sub-module to obtain the denoised radar image; then, in the latent space, the denoised radar image is encoded and decoded based on the autoencoder sub-module to reconstruct the optical target domain of the radar image dataset.

[0081] It should be noted that the structure of the cross-modal image generation module is based on the diffusion model. The cross-modal image generation module can generate images through the optical images in the dataset. Generate synthetic aperture radar images , and also through synthetic aperture radar images Generating optical images .

[0082] The core of the diffusion model is to use the Markov chain principle. The model is divided into forward diffusion process and backward diffusion process. Forward diffusion is the process of gradually adding noise to a real image until it becomes a pure noise image. In the forward diffusion process, Gaussian noise is T step gradually added to the input image In the example above, we generate noisy image samples. ，…， ,when T When it is big enough, Approximately Gaussian distribution. The overall noise diffusion process is as follows:

[0083] (1)

[0084] where the diffusion transfer variance of each step is as follows:

[0085] N( ) (2)

[0086] where, is the prior distribution, N is the Gaussian distribution, is the time t corresponding to the noisy data, I is an all-one matrix, ∈(0，1) is satisfied < < is the variance hyperparameter of the Gaussian distribution. Let =1- ， then the data distribution at any time can be derived from formula (2):

[0087] (3)

[0088] where is the standard Gaussian noise. As t increases, the noise ratio rises and the original data ratio falls. As the noise ratio rises, the distribution of N(0, I) will gradually tend to the standard Gaussian distribution When the original image is converted into a completely noisy image, the diffusion process of the model has been completed.

[0089] The reverse diffusion process of the diffusion model is to start from a completely noisy image and use the noise estimation network to infer the true distribution until a real image is generated. This process is shown in formula (4):

[0090] ( ) (4)

[0091] where, is the model parameter the joint probability distribution from time step T to time step 0. represents the original real image, represents the pure noise image after T-step diffusion process. ( ) is the marginal probability distribution at time step T, that is, the probability distribution of pure noise images generated by the model after T steps. Is a continuous multiplication term, which represents the product of the conditional probability distribution from time step T to 1. Each step t Given the current noise image Next Image Conditional probability.

[0092] (5)

[0093] in, Given the current noise image Under the condition of The conditional probability of this probability distribution is a Gaussian distribution, whose parameters are determined by the model parameters Decide. yes The representation of Gaussian distribution, where represents a Gaussian distribution, is a Gaussian distributed random variable, is the mean of the Gaussian distribution, is the covariance matrix of the Gaussian distribution.

[0094] By Decompose into and noise, the approximate value of the mean can be obtained as:

[0095] (6)

[0096] in, is the model at time step t Given the current noise image When , the predicted image of the previous time step The mean of , which is the expected value of the Gaussian distribution, is used to guide the process of restoring the real image from the noisy image. Indicates that at time step t When , the ratio of the real image signal to the current noise image is a value between 0 and 1, which is usually determined by a predefined variance scheduling strategy. is the total of all previous time steps The cumulative result of the value indicates the proportion of the real image signal accumulated in the noise image up to the current time step. is the noise term in the model prediction.

[0097] The variance Set to Related constants , then the trainable parameters only exist in the mean, and the generation process can be expressed as:

[0098] (7)

[0099] in, represents a neural network with the same input and output, where The noise predicted at each step is used in the back-diffusion process. Indicates the current time step t Noisy image. Indicates that at time step t-1 image. express The complement of , indicating that at time step t The variance of the real signal.

[0100] It should be noted that the cross-modal image generation module diffuses in the latent space rather than in the pixel space, thus saving computing resources.

[0101] Image reconstruction is performed through the autoencoder model. The encoder E maps the input image code to the latent space z, and the decoder D reconstructs the image from z. Therefore, the entire network consists of a denoising network and an autoencoder. The total loss function can be expressed as follows:

[0102] (8)

[0103] in, t is the diffusion step, It is a denoising Unet network. is real noise and network prediction noise The mean square error between is used to measure the performance of the denoising network. is added to the original image x The noise in is assumed to be standard normal distribution.

[0104] The target domain image generated by the cross-modal image generation module has an optical image that is consistent with the original image in content and very close in color, and a synthetic aperture radar image that is similar in texture to the original image. This demonstrates the effectiveness of the cross-modal image generation module in addressing the visual gap between optical and synthetic aperture radar images, effectively narrowing the difference between the optical and synthetic aperture radar image spaces.

[0105] Furthermore, remote sensing images contain some useless background information that is not helpful for image analysis and processing. Therefore, to extract the required key features, optical images also require feature fusion of the radar target domain and optical image datasets using a feature fusion module to obtain optical target domain fused features. The feature fusion module includes a deep learning module and a fusion submodule.

[0106] Specifically, based on the deep learning module, key features of the radar target domain and optical image dataset are extracted to obtain the key features of the radar target domain and the key features of the optical image dataset respectively; based on the fusion submodule, feature fusion is performed on the key features of the radar target domain and the key features of the optical image dataset to obtain the optical target domain image fusion features.

[0107] Correspondingly, for radar images, in order to fuse the features of the optical target domain and the radar image dataset based on the feature fusion module to obtain the radar target domain fusion features, first, the key features of the optical target domain and the radar image dataset are extracted based on the deep learning module to obtain the key features of the optical target domain and the key features of the radar image dataset respectively; based on the fusion sub-module, the key features of the optical target domain and the key features of the radar image dataset are fused to obtain the radar target domain image fusion features.

[0108] It should be noted that the deep learning module includes a multi-scale residual module and a contextual attention module. In order to extract key features of the radar target domain and optical image datasets based on the deep learning module, the key features of the radar target domain and the key features of the optical image dataset are obtained respectively, including:

[0109] Adaptively detect radar target domain and optical image datasets based on the multi-scale residual module to obtain multi-scale residual radar features and multi-scale residual optical features respectively;

[0110] According to the contextual attention module, the multi-scale residual radar features and multi-scale residual optical features are targeted and the key features of the radar target domain and the key features of the optical image dataset are obtained respectively.

[0111] Correspondingly, for radar images, in order to extract key features of the optical target domain and radar image dataset based on the deep learning module, the key features of the optical target domain and the key features of the radar image dataset are obtained respectively, including:

[0112] Adaptively detect the optical target domain and radar image datasets based on the multi-scale residual module to obtain multi-scale residual optical features and multi-scale residual radar features respectively;

[0113] The multi-scale residual optical features and multi-scale residual radar features are targeted based on the contextual attention module to obtain the key features of the optical target domain and the key features of the radar image dataset, respectively.

[0114] In this embodiment, a contextual attention module is embedded into the network to focus on important target information and suppress useless information expression.

[0115] Image features are extracted through a multi-scale residual module, and the feature representation of the optical image is: , the optical target domain feature generated by the cross-modal image generation module is represented as , the synthetic aperture radar image features are expressed as , the generated synthetic aperture radar target domain features can be expressed as , where C represents the number of channels, H is the height, and W is the width.

[0116] In a specific embodiment, in order to fuse the two relevant features extracted from the optical image (synthetic aperture radar image) and the generated synthetic aperture radar target domain (optical target domain) into a single feature vector, so that it has more discriminative information. and ( and ) into a single vector in four ways:

[0117] 1. Combine the two features;

[0118] 2. Superimpose the two features on pixels to increase the feature information of the image;

[0119] 3. Multiply the two features point by point to enhance the common features and weaken the unimportant features;

[0120] 4. Differentiate and fuse the feature values ​​to obtain the difference information of the two features, capture the relationship between the features and the different response levels to the data.

[0121] The fusion function is defined as follows:

[0122]

[0123] in, The feature vector representing the optical image, is the processed optical image feature, represents the original synthetic aperture radar image features, is the processed synthetic aperture radar image feature, is the fusion function, is the splicing function.

[0124] Two eigenvectors are obtained by fusion function u and v , which captures information from both optical remote sensing images (synthetic aperture radar images) and generated synthetic aperture radar images (optical images). This module further fuses the feature vectors of the two modalities to enable cross-modal information interaction, generating feature vectors with richer semantics and stronger expressiveness, thereby improving model performance.

[0125] In this embodiment, feature fusion helps reduce the differences between modalities, allowing the model to better understand and match the image content of different modalities. The model can learn a unified feature space, which helps improve the accuracy and robustness of cross-modal retrieval.

[0126] It should be noted that although remote sensing images from different sources have been mapped into a latent space through the cross-domain image generation module and the feature fusion module, and the global correspondence between optical images and synthetic aperture radar images has been mined, such global information ignores the semantic structure.

[0127] As a preferred embodiment, in S105, in order to better mine and establish semantic consistency correspondences between different data domains, the optical target domain fusion features and the radar target domain fusion features are semantically aligned according to the alignment module to obtain aligned radar target domain fusion features, specifically including:

[0128] The similarity between the fusion feature vector of the optical target domain and the fusion feature vector of the radar target domain is calculated based on the similarity calculation formula;

[0129] Adjust the radar feature weights of radar target domain fusion features according to similarity;

[0130] The radar target domain fusion features are aligned according to the radar feature weights to obtain the aligned radar target domain fusion features.

[0131] The alignment module mines the correspondence between optical image segments and synthetic aperture radar image segments, not only performing cross-domain alignment of image domains, but also aligning semantic structures in the cross-domain shared space.

[0132] In one specific embodiment, optical images and synthetic aperture radar (SAR) images from an industrial context, while depicting the same scene in different ways, exhibit significant visual differences. To better mine and establish semantically consistent correspondence between optical and SAR images, an alignment module is employed to align object representations at the semantic level, strengthening the semantic correspondence between optical and SAR image segments and narrowing the discrepancies between subsequently calculated hash codes.

[0133] Specifically, an alignment module is used to further constrain the relationship between samples, that is, industrial optical images and synthetic aperture radar images belonging to the same category are brought closer together, while synthetic aperture radar images of different categories are pushed further apart. The optical image and synthetic aperture radar image are not only aligned in the image space, but also further refined in the feature space.

[0134] like Figure 3 As shown, Figure 3 This is a schematic diagram of an embodiment of the alignment module structure provided by the present invention, which guides the radar target domain fusion feature through the optical target domain fusion feature. 𝑢 , 𝑢 Fusion features with radar target domain 𝑣 The similarity between the two can be calculated using the following formula:

[0135] (11)

[0136] in, express u and v The similarity, represents the dot product operation, ‖ ‖ represents the vector modulus operation. Then according to u , v The similarity between them is calculated by using the sigmoid function to construct different weights. Specifically, the cosine similarity is mapped to the range of (0, 1) by the sigmoid function, and the weight in the alignment process is dynamically adjusted according to the degree of similarity. The process can be formulated as follows:

[0137] (S(u , v)) (12)

[0138] in, represents the weight parameter, ( ) represents the sigmoid function, The higher the similarity between the optical target domain fusion feature and the radar target domain fusion feature, the greater the corresponding weight, and vice versa. Alignment of radar target domain fusion feature The radar target domain fusion feature V can be obtained by The formula is as follows:

[0139] V=v× (13)

[0140] Therefore, the alignment module can enhance the semantic consistency relationship between optical images and SAR images. u , V is input into the hashing layer, which consists of tanh Activation function implementation to obtain the hash code of the optical image and hash codes of synthetic aperture radar images , where K represents the number of hash code bits.

[0141] The alignment module fully aligns the optical image features and the synthetic aperture radar image features, thereby optimizing the data distribution of the two and making the feature distribution of the source domain and the target domain as close as possible. Therefore, the alignment module can ensure that the features of images of different modalities are consistent in the feature space. In cross-modal image retrieval, the alignment module optimizes the distribution of the feature space through semantic alignment and hash quantization loss, making the image features of the same category closer in the feature space and the image features of different categories remain separate, which helps the model more accurately identify and match cross-modal image content in retrieval tasks.

[0142] As a preferred embodiment, in S106, the total model loss function value L includes: semantic alignment loss function and As well as the orthogonal constraint loss function and hash quantization loss function of the category.

[0143] and We learn the semantic structure between optical and synthetic aperture radar images, which can be compressed into hash codes with shared semantics in Hamming space.

[0144] The cross-modal hash loss function needs to effectively handle the visual semantic differences when mapping optical images and synthetic aperture radar images into hash codes in a common Hamming space, that is, to maintain intra-class similarity. Due to the different imaging technologies of optical images and synthetic aperture radar images, the visual semantic differences are large, and intra-class similarity is not easy to learn. Therefore, this application directly uses class labels as supervisory signals, and directly uses class labels to guide hash learning through the semantic consistency loss function, thereby fully capturing the complex semantics of cross-modal remote sensing images. Specifically, if Figure 4 As shown, Figure 4 This is a schematic diagram of an embodiment of the intra-class similarity classification result provided by the present invention. In the Hamming space, the optical image and synthetic aperture radar images belong In order to maintain the intra-class similarity between optical images and synthetic aperture radar images, the semantic consistency loss function constrains and Around the class vector Distribution, you can and Mapped into similar hash codes, effectively eliminating the differences between modalities, where is a binary vector of categories. In addition, the optical image and synthetic aperture radar images Do not belong to the same category, the semantic consistency loss function increases and The Hamming distance between them is used to maintain the distance between classes, where * { ，y} .

[0145] The semantic information provided by the category-level labels in the dataset helps the network correctly classify samples. It can learn hash codes with shared semantics in the Hamming space, placing images of the same category closer together and images of different categories farther apart. This maintains both intra-class similarity and inter-class differences between optical and synthetic aperture radar images, and semantic similarity between different modalities. The hash network is optimized using class labels and the softmax function. The output of the softmax function is a probability distribution over multiple class labels, defined as follows:

[0146] (14)

[0147] in, Indicates the The normalized probability of samples, Represents the category score vector The j elements, of which j∈{1，2，…，l} , Represents the class embedding vector and eigenvectors h Optimizing the loss is equivalent to optimizing In the retrieval task, It can be expressed as:

[0148] (15)

[0149] In the formula in j represents a class, and express or Based on formula (15), if is approximately a constant, then and There is a positive correlation, and the formula can be obtained as follows:

[0150] (16)

[0151] In order to make Close to a constant, let and Orthogonal, we can get:

[0152] 0(17)

[0153] According to formulas (16) and (17), we can get the maximum The optimal solution for classification can be approximated, namely and Parallel and in the same direction, the formula can be obtained as follows:

[0154] ⊥ ，k {1, 2,…, l}∧k≠j (18)

[0155] in, j Represents a certain category. The cross-modal retrieval task of optical images and synthetic aperture radar images requires optimizing all category labels, which means that the columns of 𝐸 are orthogonal to each other. In addition, the Hamming distance Ham ( , ) is linearly related to the inner product as follows:

[0156] (19)

[0157] In the formula is the length of the hash code. Based on the above formula, the softmax loss can be expressed as follows:

[0158] (20)

[0159] From formula (20), we can know that maximizing log Classification and hash learning can be optimized simultaneously, that is, the hash codes of optical images and synthetic aperture radar images of the same category are close to the common class vector, thereby aligning the semantic information.

[0160] Therefore, the total loss function is obtained according to formulas (14), (16), (18), and (19). Specifically, it includes the semantic alignment loss and Class orthogonality constraint loss and hash quantization loss . and The semantic structure between optical images and synthetic aperture radar images is learned, and the two can be compressed into hash codes with shared semantics in the Hamming space. And these two loss functions can maintain the intra-class similarity and inter-class distinction between optical images and synthetic aperture radar images. Semantic alignment loss function and The formula is as follows:

[0161] (twenty one)

[0162] (twenty two)

[0163] yes The orthogonality constraint loss constrains two vectors to be orthogonal by making their inner product close to zero. The definition is as follows:

[0164] (twenty three)

[0165] in represents the identity matrix, and is the vector of the jth category and the kth category in the class embedding vector E, yes The Euclidean norm between and the identity matrix I is used to measure the difference between the two matrices.

[0166] Hash quantization loss The definition is as follows:

[0167] (twenty four)

[0168] in Calculate the absolute value of an element, represents L1 normalization. Reduces the quantization error between continuous features and binary features. However, for generating and of sign The (∙) function is not differentiable at zero, which affects the optimization of the backpropagation algorithm and the update of the hash network. To solve this problem, a smooth function is used instead: , formula (24) can be rewritten as follows:

[0169] (25)

[0170] Quantify the loss for hashing, l is the dimension of the feature vector, i For the i samples, Kwherein n is the number of bits of the hash code, t wherein n is the number of bits of the hash code, t wherein n is the number of bits of the hash code, x wherein n is the number of bits of the hash code, x wherein n is the number of bits of the hash code, wherein n is the number of bits of the hash code, i wherein n is the number of bits of the hash code, t wherein n is the number of bits of the hash code.

[0171] Accordingly, the total loss function is:

[0172] (26)

[0173] The present application learns the cross-modal feature representation between optical images and synthetic aperture radar images through a deep learning model, thereby improving the accuracy of cross-modal image retrieval. This cross-modal feature learning is particularly important for remote sensing image analysis, as remote sensing image data often contains multiple modalities that need to be effectively fused and utilized.

[0174] In the above manner, first, the radar target domain / optical target domain is generated through the cross-modal image generation module, realizing cross-modal processing of the input data; then, the radar target domain / optical target domain and the optical image dataset / radar image dataset are fused through the feature fusion module, obtaining optical target domain fusion features and radar target domain fusion features that simultaneously include optical and radar features; next, the optical target domain fusion features and the radar target domain fusion features are semantically aligned, realizing sufficient capture of the complex semantics of cross-modal remote sensing images and ensuring that the features of different modal images are more consistent at the feature level; by taking the loss value as a training quantitative indicator of the retrieval model, the model can accurately represent the cross-modal image similarity, thereby ensuring the reliability of cross-modal remote sensing image retrieval.

[0175] To solve the above problems, the present application also provides a cross-modal remote sensing image retrieval device, as shown in Figure 5 FIG. 1 is a structural block diagram of an embodiment of the cross-modal remote sensing image retrieval device provided by the present application, and the cross-modal remote sensing image retrieval device 500 includes: Figure 5 The dataset acquisition module 501 is configured to acquire a remote sensing image dataset, and the remote sensing image dataset includes an optical image dataset and a radar image dataset.

[0176] The retrieval model construction module 502 is configured to construct a retrieval model, including a cross-modal image generation module, a feature fusion module, an alignment module, a hash layer, and a loss function module.

[0177]

[0178] ​The optical target domain fusion feature module 503 is used to input the optical image dataset into the retrieval model, generate the radar target domain based on the cross-modal image generation module, and fuse the radar target domain and the optical image dataset based on the feature fusion module to obtain the optical target domain fusion feature;

[0179] The radar target domain fusion feature module 504 is used to input the radar image dataset into the retrieval model, generate the optical target domain based on the cross-modal image generation module, and fuse the optical target domain and the radar image dataset based on the feature fusion module to obtain the radar target domain fusion feature;

[0180] A semantic alignment module 505 is configured to semantically align the optical target domain fusion features and the radar target domain fusion features according to the alignment module to obtain aligned radar target domain fusion features;

[0181] The target retrieval model determination module 506 has a hash layer that calculates an optical hash code and a radar hash code based on the optical target domain fusion features and the aligned radar target domain fusion features, respectively. The loss function module calculates the loss between the optical hash code and the radar hash code, iterates the training, and determines the retrieval model with the minimum loss as the target retrieval model.

[0182] The retrieval module 507 is configured to retrieve the optical image / radar image to be retrieved based on the target retrieval model to obtain corresponding radar image data / optical image data.

[0183] like Figure 6 As shown, the present invention also provides an electronic device 600. The electronic device 600 includes a processor 601, a memory 602 and a display 603. Figure 6 Only some of the components of the electronic device 600 are shown, but it should be understood that implementation of all of the shown components is not required, and more or fewer components may be implemented instead.

[0184] In some embodiments, the processor 601 may be a central processing unit (CPU), a microprocessor, or other data processing chip, configured to execute program codes or process data stored in the memory 602 , such as the cross-modal remote sensing image retrieval method of the present invention.

[0185] In some embodiments, processor 601 may be a single server or a server group. The server group may be centralized or distributed. In some embodiments, processor 601 may be local or remote. In some embodiments, processor 601 may be implemented on a cloud platform. In one embodiment, the cloud platform may include a private cloud, a public cloud, a hybrid cloud, a community cloud, a distributed cloud, an internal cloud, multiple clouds, or any combination thereof.

[0186] In some embodiments, the memory 602 may be an internal storage unit of the electronic device 600, such as a hard disk or memory of the electronic device 600. In other embodiments, the memory 602 may also be an external storage device of the electronic device 600, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 600.

[0187] Furthermore, the memory 602 may include both an internal storage unit of the electronic device 600 and an external storage device. The memory 602 is used to store application software installed in the electronic device 600 and various data.

[0188] In some embodiments, the display 603 can be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display 603 is used to display information about the electronic device 600 and to display a visual user interface. Components 601-603 of the electronic device 600 communicate with each other via a system bus.

[0189] In one embodiment, when the processor 601 executes the cross-modal remote sensing image retrieval program in the memory 602, the following steps may be implemented:

[0190] Acquire a remote sensing image dataset, which includes an optical image dataset and a radar image dataset;

[0191] Build a retrieval model, including a cross-modal image generation module, a feature fusion module, an alignment module, a hash layer, and a loss function module;

[0192] The optical image dataset is input into the retrieval model, the radar target domain is generated based on the cross-modal image generation module, and the radar target domain and the optical image dataset are fused based on the feature fusion module to obtain the optical target domain fusion feature;

[0193] The radar image dataset is input into the retrieval model, and the optical target domain is generated based on the cross-modal image generation module. The optical target domain and the radar image dataset are fused based on the feature fusion module to obtain the radar target domain fusion features.

[0194] According to the alignment module, the optical target domain fusion features and the radar target domain fusion features are semantically aligned to obtain the aligned radar target domain fusion features;

[0195] The hash layer calculates the optical hash code and radar hash code based on the optical target domain fusion features and the aligned radar target domain fusion features, respectively. The loss value between the optical hash code and the radar hash code is calculated according to the loss function module. After iterative training, the retrieval model with the minimum loss value is determined as the target retrieval model.

[0196] The optical image / radar image to be retrieved is retrieved based on the target retrieval model to obtain the corresponding radar image data / optical image data.

[0197] It should be understood that, when the processor 601 executes the cross-modal remote sensing image retrieval program in the memory 602 , in addition to the above functions, it can also implement other functions. For details, please refer to the description of the corresponding method embodiment above.

[0198] Furthermore, the embodiments of the present invention do not specifically limit the type of electronic device 600 mentioned. The electronic device 600 may be a portable electronic device such as a mobile phone, tablet computer, personal digital assistant (PDA), wearable device, or laptop computer. Exemplary embodiments of portable electronic devices include, but are not limited to, portable electronic devices running iOS, Android, Microsoft, or other operating systems. The portable electronic devices mentioned above may also be other portable electronic devices, such as a laptop computer with a touch-sensitive surface (e.g., a touch panel). It should also be understood that in other embodiments of the present invention, the electronic device 600 may not be a portable electronic device, but rather a desktop computer with a touch-sensitive surface (e.g., a touch panel).

[0199] Accordingly, an embodiment of the present invention also provides a computer-readable storage medium, which is used to store computer-readable programs or instructions. When the program or instructions are executed by a processor, it can implement the steps or functions in the cross-modal remote sensing image retrieval method provided by the above-mentioned method embodiments.

[0200] Those skilled in the art will appreciate that all or part of the process flow of the above-described method embodiment can be implemented by instructing related hardware (such as a processor, controller, etc.) through a computer program, and the computer program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a magnetic disk, an optical disk, a read-only memory, or a random access memory.

[0201] The above is a detailed introduction to the cross-modal remote sensing image retrieval method, device, electronic device and storage medium provided by the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core ideas. At the same time, for those skilled in the art, according to the ideas of the present invention, there will be changes in the specific implementation methods and application scopes. In summary, the contents of this specification should not be understood as limiting the present invention.

Claims

1. A cross-modal remote sensing image retrieval method, characterized in that: include: Acquire a remote sensing image dataset, where the remote sensing image dataset includes an optical image dataset and a radar image dataset; Build a retrieval model, including a cross-modal image generation module, a feature fusion module, an alignment module, a hash layer, and a loss function module; Inputting the optical image dataset into the retrieval model, generating a radar target domain based on the cross-modal image generation module, and performing feature fusion on the radar target domain and the optical image dataset based on the feature fusion module to obtain an optical target domain fusion feature; Inputting the radar image dataset into the retrieval model, generating an optical target domain based on the cross-modal image generation module, and performing feature fusion on the optical target domain and the radar image dataset based on the feature fusion module to obtain radar target domain fusion features; Semantically aligning the optical target domain fusion feature and the radar target domain fusion feature according to the alignment module to obtain an aligned radar target domain fusion feature, including: calculating the similarity between the optical target domain fusion feature vector and the radar target domain fusion feature vector based on a similarity calculation formula; adjusting the radar feature weight of the radar target domain fusion feature according to the similarity; aligning the radar target domain fusion features according to the radar feature weight to obtain the aligned radar target domain fusion feature; The hash layer calculates an optical hash code and a radar hash code according to the optical target domain fusion feature and the aligned radar target domain fusion feature, respectively, calculates a loss value between the optical hash code and the radar hash code according to the loss function module, iteratively trains, and determines the retrieval model with the minimum loss value as the target retrieval model; The optical image to be retrieved / the radar image to be retrieved is retrieved based on the target retrieval model to obtain corresponding radar image data / optical image data.

2. The cross-modal remote sensing image retrieval method according to claim 1, characterized in that: The cross-modal image generation module includes a denoising network submodule and an autoencoder submodule; Generating a radar target domain based on the cross-modal image generation module includes: Performing denoising processing on the optical image dataset based on the denoising network submodule to obtain a denoised optical image; In the latent space, encoding and decoding processing are performed on the denoised optical image based on the autoencoder submodule to reconstruct the radar target domain of the optical image dataset.

3. The cross-modal remote sensing image retrieval method according to claim 1, characterized in that: The feature fusion module includes a deep learning module and a fusion submodule; The step of fusing features of the radar target domain and the optical image dataset based on the feature fusion module to obtain an optical target domain fusion feature includes: Extract key features of the radar target domain and the optical image dataset based on the deep learning module to obtain key features of the radar target domain and key features of the optical image dataset respectively; Based on the fusion submodule, feature fusion is performed on the radar target domain key features and the optical image data set key features to obtain optical target domain image fusion features.

4. The cross-modal remote sensing image retrieval method according to claim 3, characterized in that: The deep learning module includes a multi-scale residual module and a contextual attention module; the key feature extraction of the radar target domain and the optical image dataset based on the deep learning module to obtain the radar target domain key features and the optical image dataset key features respectively includes: Adaptively detecting the radar target domain and the optical image dataset according to the multi-scale residual module to obtain multi-scale residual radar features and multi-scale residual optical features, respectively; The multi-scale residual radar features and the multi-scale residual optical features are targeted and focused according to the contextual attention module to obtain radar target domain key features and optical image dataset key features, respectively.

5. The cross-modal remote sensing image retrieval method according to claim 1, characterized in that: The loss function module includes a semantic alignment loss function, an orthogonal constraint loss function, and a hash quantization loss function; and calculating the loss value between the optical hash code and the radar hash code according to the loss function module includes: Calculating a semantic alignment loss between the optical hash code and the radar hash code according to the semantic alignment loss function; Calculating the orthogonal constraint loss between the optical target domain fusion feature and the radar target domain fusion feature according to the orthogonal constraint loss function; Calculating a hash quantization loss between the optical hash code and the radar hash code according to the hash quantization loss function; The loss value is determined according to the semantic alignment loss, the orthogonality constraint loss, and the hash quantization loss.

6. The cross-modal remote sensing image retrieval method according to claim 5, characterized in that: The calculating the hash quantization loss between the optical hash code and the radar hash code according to the hash quantization loss function includes: The initial hash quantization loss function is updated by a smoothing function to obtain the hash quantization loss function; The calculation formula of the hash quantization loss function is: Quantify the loss for the hash, l is the dimension of the feature vector, i For the i samples, K is the number of hash code bits, t For the t hash codes, cosh ( x )express x The hyperbolic cosine function, ∑ represents the sum, Indicates the i samples and t The feature vector corresponding to the hash code.

7. A cross-modal remote sensing image retrieval device, characterized in that: include: A data set acquisition module is used to acquire a remote sensing image data set, wherein the remote sensing image data set includes an optical image data set and a radar image data set; Retrieval model construction module, used to build the retrieval model, including cross-modal image generation module, feature fusion module, alignment module, hash layer and loss function module; an optical target domain fusion feature module, configured to input the optical image dataset into the retrieval model, generate a radar target domain based on the cross-modal image generation module, and perform feature fusion on the radar target domain and the optical image dataset based on the feature fusion module to obtain an optical target domain fusion feature; a radar target domain fusion feature module, configured to input the radar image dataset into the retrieval model, generate an optical target domain based on the cross-modal image generation module, and perform feature fusion on the optical target domain and the radar image dataset based on the feature fusion module to obtain a radar target domain fusion feature; a semantic alignment module, configured to semantically align the optical target domain fusion feature and the radar target domain fusion feature according to the alignment module to obtain an aligned radar target domain fusion feature, comprising: calculating the similarity between the optical target domain fusion feature vector and the radar target domain fusion feature vector based on a similarity calculation formula; adjusting the radar feature weight of the radar target domain fusion feature according to the similarity; and aligning the radar target domain fusion features according to the radar feature weight to obtain the aligned radar target domain fusion feature; a target retrieval model determination module, wherein the hash layer calculates an optical hash code and a radar hash code according to the optical target domain fusion feature and the aligned radar target domain fusion feature, respectively, calculates a loss value between the optical hash code and the radar hash code according to the loss function module, and iteratively trains to determine the retrieval model with the minimum loss value as the target retrieval model; The retrieval module is used to retrieve the optical image to be retrieved / the radar image to be retrieved based on the target retrieval model to obtain corresponding radar image data / optical image data.

8. An electronic device, characterized in that: comprising a memory and a processor, wherein, The memory is used to store programs; The processor is coupled to the memory and is configured to execute the program stored in the memory to implement the steps of the cross-modal remote sensing image retrieval method described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that Used to store computer-readable programs or instructions, which, when executed by a processor, can implement the steps of the cross-modal remote sensing image retrieval method described in any one of claims 1 to 6.

Images