Cross-modal remote sensing image retrieval method and device, electronic equipment and storage medium

Abstract

The invention discloses a cross-modal remote sensing image retrieval method and device, electronic equipment and a storage medium, and belongs to the technical field of remote sensing image processing.The method comprises the steps that a remote sensing image data set is obtained; constructing a retrieval model; inputting the optical image data set into a retrieval model to obtain optical target domain fusion features; inputting the radar image data set into a retrieval model to obtain radar target domain fusion features; performing semantic alignment on the optical target domain fusion features and the radar target domain fusion features according to an alignment module to obtain aligned radar target domain fusion features; calculating a loss value between the optical hash code and the radar hash code according to a loss function module, and determining the retrieval model with the minimum loss value as a target retrieval model; and performing retrieval based on the target retrieval model. And cross-modal processing is performed on the image through the cross-modal image generation module, and the features of the two modals are fused to realize retrieval based on the fused features, so that cross-modal remote sensing image retrieval is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing image processing, and in particular to a cross-modal remote sensing image retrieval method, device, electronic equipment and storage medium. Background Art

[0002] The rapid development of sensing technology has provided people with a large number of optical, hyperspectral, and synthetic aperture radar images, among which optical images and synthetic aperture radar images have shown significant advantages in cross-modal retrieval. However, due to the different imaging mechanisms of optical images and synthetic aperture radar images, there are significant differences in target representation, which increases the difficulty of retrieval. Specifically, optical images are electromagnetic reflection imaging, which can directly observe the brightness, texture structure, etc. of targets and scenes. However, this imaging method is easily affected by weather. When clouds, fog or water vapor form a certain scale on the signal propagation route, it will cause inaccurate imaging or simply be covered. Synthetic aperture radar images are microwave imaging, using synthetic aperture radar satellites that can overcome the limitations of cloud, fog, rain, snow and dark night conditions, and can be used for ground-to-ground observations all day and all weather. However, since the synthetic aperture radar imaging process undergoes complex signal processing, synthetic aperture radar images will have problems such as speckle noise pollution, thereby reducing the image interpretation ability.

[0003] Optical images and synthetic aperture radar images are two different modal data in different image spaces. In the process of image retrieval, the existing technology targets images of the same image space and the same modality. The retrieval results are relatively simple and cannot meet the requirements of cross-modal remote sensing image retrieval.

[0004] Therefore, in the process of cross-modal remote sensing image retrieval, the existing technology has the problem of difficulty in realizing cross-modal remote sensing image retrieval. Summary of the invention

[0005] In view of this, it is necessary to provide a cross-modal remote sensing image retrieval method, device, electronic device and storage medium to solve the problem that cross-modal remote sensing image retrieval is difficult to achieve in the process of cross-modal remote sensing image retrieval in the prior art.

[0006] In order to solve the above problems, the present invention provides a cross-modal remote sensing image retrieval method, comprising: Acquire a remote sensing image dataset, where the remote sensing image dataset includes an optical image dataset and a radar image dataset; Build a retrieval model, including a cross-modal image generation module, a feature fusion module, an alignment module, a hash layer, and a loss function module; The optical image dataset is input into the retrieval model, the radar target domain is generated based on the cross-modal image generation module, and the radar target domain and the optical image dataset are fused based on the feature fusion module to obtain the optical target domain fusion feature; The radar image dataset is input into the retrieval model, the optical target domain is generated based on the cross-modal image generation module, and the optical target domain and the radar image dataset are fused based on the feature fusion module to obtain the radar target domain fusion feature; According to the alignment module, the optical target domain fusion features and the radar target domain fusion features are semantically aligned to obtain the aligned radar target domain fusion features; The hash layer calculates the optical hash code and the radar hash code respectively according to the optical target domain fusion features and the aligned radar target domain fusion features, calculates the loss value between the optical hash code and the radar hash code according to the loss function module, iterates the training, and determines the retrieval model with the smallest loss value as the target retrieval model; The optical image to be retrieved / the radar image to be retrieved is retrieved based on the target retrieval model to obtain the corresponding radar image data / optical image data.

[0007] In a possible implementation, the cross-modal image generation module includes a denoising network submodule and an autoencoder submodule; generating a radar target domain based on the cross-modal image generation module includes: Based on the denoising network submodule, the optical image dataset is denoised to obtain a denoised optical image; In the latent space, the denoised optical image is encoded and decoded based on the autoencoder sub-module to reconstruct the radar target domain of the optical image dataset.

[0008] In a possible implementation, the feature fusion module includes a deep learning module and a fusion submodule; based on the feature fusion module, the radar target domain and the optical image data set are feature fused to obtain the optical target domain fusion feature, including: Based on the deep learning module, key features of the radar target domain and optical image data set are extracted to obtain the key features of the radar target domain and the key features of the optical image data set respectively; Based on the fusion submodule, the key features of the radar target domain and the key features of the optical image dataset are fused to obtain the optical target domain image fusion features.

[0009] In a possible implementation, the deep learning module includes a multi-scale residual module and a context attention module; based on the deep learning module, key features of the radar target domain and the optical image data set are extracted to obtain key features of the radar target domain and key features of the optical image data set, respectively, including: Adaptively detect the radar target domain and optical image data set according to the multi-scale residual module to obtain multi-scale residual radar features and multi-scale residual optical features respectively; According to the contextual attention module, the multi-scale residual radar features and the multi-scale residual optical features are targeted and the key features of the radar target domain and the key features of the optical image dataset are obtained respectively.

[0010] In a possible implementation, semantic alignment is performed on the optical target domain fusion features and the radar target domain fusion features according to the alignment module to obtain aligned radar target domain fusion features, including: The similarity between the fusion feature vector in the optical target domain and the fusion feature vector in the radar target domain is calculated based on the similarity calculation formula; Adjust radar feature weights of radar target domain fusion features according to similarity; The radar target domain fusion features are aligned according to the radar feature weights to obtain the aligned radar target domain fusion features.

[0011] In a possible implementation, the loss function module includes a semantic alignment loss function, an orthogonal constraint loss function, and a hash quantization loss function; and the loss value between the optical hash code and the radar hash code is calculated according to the loss function module, including: Calculate the semantic alignment loss between the optical hash code and the radar hash code according to the semantic alignment loss function; The orthogonal constraint loss between the optical target domain fusion features and the radar target domain fusion features is calculated according to the orthogonal constraint loss function; Calculating a hash quantization loss between an optical hash code and a radar hash code according to a hash quantization loss function; The loss value is determined based on semantic alignment loss, orthogonality constraint loss, and hash quantization loss.

[0012] In a possible implementation, the hash quantization loss between the optical hash code and the radar hash code is calculated according to the hash quantization loss function, including: The initial hash quantization loss function is updated by a smoothing function to obtain a hash quantization loss function; Among them, the calculation formula of the hash quantization loss function is:

[0013] Quantize the loss for hashing, l is the dimension of the feature vector, i For the i samples, K is the number of hash code bits, t For the t hash code, cosh x expressx The hyperbolic cosine function, ∑ represents the sum, Indicates i Samples and t The feature vector corresponding to the hash code.

[0014] In order to solve the above problems, the present invention also provides a cross-modal remote sensing image retrieval device, comprising: A data set acquisition module is used to acquire a remote sensing image data set, where the remote sensing image data set includes an optical image data set and a radar image data set; A retrieval model building module is used to build a retrieval model, including a cross-modal image generation module, a feature fusion module, an alignment module, a hash layer, and a loss function module; An optical target domain fusion feature module is used to input the optical image data set into the retrieval model, generate the radar target domain based on the cross-modal image generation module, and fuse the radar target domain and the optical image data set based on the feature fusion module to obtain the optical target domain fusion feature; The radar target domain fusion feature module is used to input the radar image data set into the retrieval model, generate the optical target domain based on the cross-modal image generation module, and fuse the optical target domain and the radar image data set based on the feature fusion module to obtain the radar target domain fusion feature; A semantic alignment module is used to semantically align the optical target domain fusion features and the radar target domain fusion features according to the alignment module to obtain an aligned radar target domain fusion feature; Target retrieval model determination module: The hash layer calculates the optical hash code and the radar hash code according to the optical target domain fusion features and the aligned radar target domain fusion features, respectively. The loss value between the optical hash code and the radar hash code is calculated according to the loss function module. Iterative training is performed to determine the retrieval model with the smallest loss value as the target retrieval model. The retrieval module is used to retrieve the optical image to be retrieved / the radar image to be retrieved based on the target retrieval model to obtain the corresponding radar image data / optical image data.

[0015] In order to solve the above problem, the present invention further provides an electronic device, including a memory and a processor, wherein: Memory, used to store programs; The processor is coupled to the memory and is used to execute the program stored in the memory to implement the steps in the cross-modal remote sensing image retrieval method as described above.

[0016] In order to solve the above problems, the present invention also provides a computer-readable storage medium for storing computer-readable programs or instructions, which, when executed by a processor, can implement the steps in the cross-modal remote sensing image retrieval method as described above.

[0017] The beneficial effect of adopting the above embodiment is as follows: the present invention provides a cross-modal remote sensing image retrieval method, first, a radar target domain / optical target domain is generated by a cross-modal image generation module, thereby realizing cross-modal processing of input data; then, the radar target domain / optical target domain and the optical image data set / radar image data set are feature fused by a feature fusion module, thereby obtaining an optical target domain fusion feature / radar target domain fusion feature that includes both optical and radar features; next, the optical target domain fusion feature and the radar target domain fusion feature are semantically aligned, thereby realizing full capture of the complex semantics of the cross-modal remote sensing image, thereby ensuring that the features of images of different modalities are more consistent at the feature level; finally, by taking the loss value as a training quantitative indicator of the retrieval model, it is ensured that the model can accurately represent the cross-modal image similarity, thereby ensuring the reliability of the cross-modal remote sensing image retrieval, and since the retrieval model can process image data of two modalities and output the required remote sensing image retrieval results accordingly, cross-modal remote sensing image retrieval is realized. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A schematic diagram of a process flow of an embodiment of a cross-modal remote sensing image retrieval method provided by the present invention; Figure 2 A schematic diagram of the structure of an embodiment of a retrieval model provided by the present invention; Figure 3 A schematic diagram of the structure of an alignment module structure according to an embodiment of the present invention; Figure 4 A result schematic diagram of an embodiment of the intra-class similarity classification result provided by the present invention; Figure 5 A structural block diagram of an embodiment of a cross-modal remote sensing image retrieval device provided by the present invention; Figure 6 This is a structural block diagram of an embodiment of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0019] The preferred embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of the present invention and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.

[0020] Remote sensing images are images of the earth's surface acquired at long distances through remote sensing platforms such as aircraft and satellites, using different types of sensors such as optical cameras, multispectral cameras, infrared cameras, radars, etc.

[0021] Target domain (TD) images refer to test sample images to be classified or identified in a dataset with few or even no labeled samples. These images have different data distributions from source domain images, but usually belong to the same type of task or problem domain. Target domain images play an important role in machine learning, deep learning, and computer vision. By rationally utilizing the knowledge and information in source domain images, we can effectively address challenges such as data scarcity, domain differences, and labeling difficulties, and further improve the accuracy and efficiency of target domain image classification or recognition.

[0022] Source Domain Images is an important concept in machine learning fields such as transfer learning and domain adaptation. It refers to the labeled or unlabeled image datasets used in the training phase that have different data distributions from the target domain images but belong to the same type of task or problem domain.

[0023] Domain alignment refers to connecting the source domain and the target domain through certain methods so that the model can achieve good performance in both the source domain and the target domain. In machine learning, the source domain and the target domain usually refer to different data sets, which may have different data distributions or feature representations. Due to the difference in data distribution, it may not be ideal to directly generalize the performance of the model in the source domain to the target domain. Therefore, domain alignment is needed to reduce the difference between the source domain and the target domain.

[0024] The loss function, also known as the cost function, is a non-negative real-valued function whose input is the model's predicted value and the actual observed value, and whose output is a scalar that represents the degree of difference between the predicted value and the actual value. The smaller the difference, the more accurate the model's prediction results. During the model training process, by minimizing the value of the loss function, the model's parameters can be continuously optimized, thereby improving the model's prediction performance.

[0025] Backpropagation is one of the most core and commonly used optimization algorithms in deep learning and is widely used in the training process of neural networks. It is based on the gradient descent method and is a learning algorithm suitable for multi-layer neural networks. The basic principle of the backpropagation algorithm is to calculate the gradient of the loss function with respect to the network parameters in order to update the parameters and minimize the loss function.

[0026] A hash code is an algorithm that allows objects of the same class to have different hash codes according to their different characteristics. However, different objects may have the same hash code, which depends on the specific implementation of the hash code algorithm. In programming languages ​​such as Java, hash codes are often used for fast lookup and deduplication operations, such as in hash tables and collection classes (such as HashMap, HashSet, etc.).

[0027] Remote sensing image retrieval technology achieves fast and accurate image retrieval through vision, object and spatial relationships, and semantic levels, and is used for tasks such as environmental and disaster monitoring. The retrieval scope directly determines the breadth of the information obtained. However, in the process of image retrieval, the existing technology targets images of the same image space and the same modality, and the retrieval results are relatively simple, which cannot meet the requirements of cross-modal remote sensing image retrieval.

[0028] Therefore, in the process of cross-modal remote sensing image retrieval, the existing technology has the problem of difficulty in realizing cross-modal remote sensing image retrieval.

[0029] In order to solve the above problems, the present invention provides a cross-modal remote sensing image retrieval method, device, electronic device and storage medium, which are described in detail below.

[0030] Figure 1 A schematic diagram of a flow chart of an embodiment of a cross-modal remote sensing image retrieval method provided by the present invention is shown in FIG. Figure 1 As shown, the cross-modal remote sensing image retrieval method includes: S101: Acquire a remote sensing image dataset, where the remote sensing image dataset includes an optical image dataset and a radar image dataset; S102: constructing a retrieval model, including a cross-modal image generation module, a feature fusion module, an alignment module, a hash layer, and a loss function module; S103: inputting the optical image data set into the retrieval model, generating a radar target domain based on the cross-modal image generation module, and fusing features of the radar target domain and the optical image data set based on the feature fusion module to obtain optical target domain fusion features; S104: inputting the radar image dataset into the retrieval model, generating an optical target domain based on the cross-modal image generation module, and fusing features of the optical target domain and the radar image dataset based on the feature fusion module to obtain radar target domain fusion features; S105: semantically aligning the optical target domain fusion feature and the radar target domain fusion feature according to the alignment module to obtain an aligned radar target domain fusion feature; S106: The hash layer calculates the optical hash code and the radar hash code respectively according to the optical target domain fusion feature and the aligned radar target domain fusion feature, calculates the loss value between the optical hash code and the radar hash code according to the loss function module, iterates the training, and determines the retrieval model with the smallest loss value as the target retrieval model; S107: Retrieve the optical image to be retrieved / the radar image to be retrieved based on the target retrieval model to obtain corresponding radar image data / optical image data.

[0031] In this embodiment, first, a radar target domain / optical target domain is generated by a cross-modal image generation module, thereby realizing cross-modal processing of input data; then, the radar target domain / optical target domain and the optical image data set / radar image data set are feature fused by a feature fusion module, thereby obtaining an optical target domain fusion feature / radar target domain fusion feature that includes both optical and radar features; next, the optical target domain fusion feature and the radar target domain fusion feature are semantically aligned, thereby fully capturing the complex semantics of the cross-modal remote sensing image and ensuring that the features of images of different modalities are more consistent at the feature level; finally, by using the loss value as a training quantitative indicator for the retrieval model, it is ensured that the model can accurately represent the cross-modal image similarity and ensure the reliability of the cross-modal remote sensing image retrieval. Since the retrieval model can process image data of two modalities and output the required remote sensing image retrieval results accordingly, cross-modal remote sensing image retrieval is realized.

[0032] It should be noted that the radar image in this application refers to a synthetic aperture radar image (Synthetic Aperture Radar Image, SAR image).

[0033] As a preferred embodiment, in S101, the remote sensing image data set comes from a public remote sensing image data set, and the collected remote sensing image data can also be manually screened and grouped according to actual needs to meet the personalized needs of subsequent model training.

[0034] As a preferred embodiment, in S102, in order to clearly represent the relationship between the cross-modal image generation module, the feature fusion module, the alignment module, the hash layer and the loss function module, as shown in FIG. Figure 2 As shown, Figure 2 A schematic diagram of the structure of an embodiment of a retrieval model provided by the present invention.

[0035] As a preferred embodiment, in S103, the cross-modal image generation module includes a denoising network submodule and an autoencoder submodule; in order to generate a radar target domain based on the cross-modal image generation module, first, the optical image dataset is denoised based on the denoising network submodule to obtain a denoised optical image; then, in the latent space, the denoised optical image is encoded and decoded based on the autoencoder submodule to reconstruct the radar target domain of the optical image dataset.

[0036] Similarly, in order to generate the optical target domain based on the cross-modal image generation module, first, the radar image dataset is denoised based on the denoising network sub-module to obtain the denoised radar image; then, in the latent space, the denoised radar image is encoded and decoded based on the autoencoder sub-module to reconstruct the optical target domain of the radar image dataset.

[0037] It should be noted that the structure of the cross-modal image generation module is based on the diffusion model. The cross-modal image generation module can generate images through the optical images in the dataset. Generate synthetic aperture radar images , and also through synthetic aperture radar images Generating optical images .

[0038] The core of the diffusion model is to use the Markov chain principle. The model is divided into forward diffusion process and backward diffusion process. Forward diffusion is the process of gradually adding noise to a real image until it becomes a pure noise image. In the forward diffusion process, Gaussian noise is T step gradually adds to the input image In the example above, we generate noisy image samples. ，…， ,when T When it is big enough, Approximately Gaussian distribution. The overall noise diffusion process is as follows: (1) Among them, the diffusion transfer variance of each step is as follows: N( ) (2) In the formula, is the prior distribution, N is a Gaussian distribution, For time t The corresponding noisy data, I is a matrix of all 1s, ∈(0，1) is satisfied < < is the variance hyperparameter of the Gaussian distribution. Let =1- ， , then the data distribution at any time can be deduced from formula (2): (3) in is standard Gaussian noise. t As the noise ratio increases, the original data ratio decreases. The distribution of N(0,I) , when the original image is transformed into a completely noisy image, the diffusion process of the model has been completed.

[0039] The reverse diffusion process of the diffusion model starts from a completely noisy image and uses a noise estimation network to infer the true distribution , until a real image is generated. This process is shown in formula (4): ( ) (4) in, is the model parameter Below, the joint probability distribution from time step T to time step 0. represents the original real image, Represents a pure noise image after T-step diffusion process. ( ) is the marginal probability distribution at time step T, that is, the probability distribution of pure noise images generated by the model after T steps. is a continuous multiplication term, which represents the product of the conditional probability distribution from time step T to 1. t Given the current noise image Next Image Conditional probability.

[0040] (5) in, Given the current noise image Under the condition of The conditional probability of this probability distribution is a Gaussian distribution, whose parameters are determined by the model parameters Decide. yesThe representation of Gaussian distribution, where represents a Gaussian distribution, is a Gaussian distributed random variable, is the mean of the Gaussian distribution, is the covariance matrix of the Gaussian distribution.

[0041] By Decompose into and noise, the approximate value of the mean can be obtained as: (6) in, is the model at time step t Given the current noisy image When , the predicted image of the previous time step The mean of , which is the expected value of the Gaussian distribution, is used to guide the process of restoring the real image from the noisy image. Indicates that at time step t When , the ratio of the real image signal to the current noise image is a value between 0 and 1, which is usually determined by a predefined variance scheduling strategy. is the sum of all previous time steps The cumulative result of the value indicates the proportion of the real image signal accumulated in the noise image up to the current time step. is the noise term predicted by the model.

[0042] The variance Set to Related constants , then the trainable parameters only exist in the mean, and the generation process can be expressed as: (7) in, represents a neural network with the same input and output, where The predicted noise at each step is used for the back diffusion process. Indicates the current time step t Noise image. Indicates that at time step t-1 of the image. express The complement of , indicating that at the time step t , the variance part of the real signal.

[0043] It should be noted that the cross-modal image generation module diffuses in the latent space rather than in the pixel space, thus saving computational resources.

[0044] Image reconstruction is performed through the autoencoder model. The encoder E maps the input image encoding to the latent space z, and the decoder D reconstructs the image from z. Therefore, the entire network consists of a denoising network and an autoencoder, and the total loss function can be expressed as follows: (8) in, t is the diffusion step, It is a denoising Unet network. is real noise and network prediction noise The mean square error between is used to measure the performance of the denoising network. is added to the original image x The noise in is assumed to be of standard normal distribution.

[0045] The target domain image generated by the cross-modal image generation module has the same content as the original image, and the color is very close. The generated synthetic aperture radar image is similar to the original image in texture. This shows the effectiveness of the cross-modal image generation module in dealing with the visual gap between optical images and synthetic aperture radar images, and can effectively narrow the difference between the optical image space and the synthetic aperture radar image space.

[0046] Furthermore, since remote sensing images contain some useless background information, this information is not helpful for image analysis and processing. Therefore, in order to extract the required key features, for optical images, it is also necessary to fuse the radar target domain and optical image data sets based on the feature fusion module to obtain the optical target domain fusion features, where the feature fusion module includes a deep learning module and a fusion submodule.

[0047] Specifically, based on the deep learning module, key features of the radar target domain and the optical image dataset are extracted to obtain the key features of the radar target domain and the key features of the optical image dataset, respectively; based on the fusion submodule, the key features of the radar target domain and the key features of the optical image dataset are fused to obtain the optical target domain image fusion features.

[0048] Correspondingly, for radar images, in order to fuse the features of the optical target domain and the radar image dataset based on the feature fusion module to obtain the radar target domain fusion features, firstly, the key features of the optical target domain and the radar image dataset are extracted based on the deep learning module to obtain the key features of the optical target domain and the key features of the radar image dataset respectively; based on the fusion submodule, the key features of the optical target domain and the key features of the radar image dataset are fused to obtain the radar target domain image fusion features.

[0049] It should be noted that the deep learning module includes a multi-scale residual module and a contextual attention module. In order to extract key features of the radar target domain and the optical image dataset based on the deep learning module, the key features of the radar target domain and the key features of the optical image dataset are obtained respectively, including: Adaptively detect the radar target domain and optical image data set according to the multi-scale residual module to obtain multi-scale residual radar features and multi-scale residual optical features respectively; According to the contextual attention module, the multi-scale residual radar features and the multi-scale residual optical features are targeted and the key features of the radar target domain and the key features of the optical image dataset are obtained respectively.

[0050] Correspondingly, for radar images, in order to extract key features of the optical target domain and radar image dataset based on the deep learning module, the key features of the optical target domain and the key features of the radar image dataset are obtained respectively, including: Adaptively detect the optical target domain and radar image data set according to the multi-scale residual module, and obtain multi-scale residual optical features and multi-scale residual radar features respectively; According to the contextual attention module, the multi-scale residual optical features and multi-scale residual radar features are focused on the target to obtain the key features of the optical target domain and the key features of the radar image dataset respectively.

[0051] In this embodiment, a contextual attention module is embedded into the network to focus on important target information and suppress useless information expression.

[0052] The image features are extracted through the multi-scale residual module, and the feature representation of the optical image is: , the optical target domain feature generated by the cross-modal image generation module is represented as , the synthetic aperture radar image features are expressed as , the generated synthetic aperture radar target domain features can be expressed as , where C represents the number of channels, H is the height, and W is the width.

[0053] In a specific embodiment, in order to fuse the two relevant features extracted from the optical image (synthetic aperture radar image) and the generated synthetic aperture radar target domain (optical target domain) into a single feature vector, so that it has more discriminative information. and ( and ) into a single vector in four ways: 1. Combine the two features; 2. Superimpose the two features by pixels to increase the feature information of the image; 3. Multiply the two features point by point to enhance the common features and weaken the unimportant features; 4. Perform differential fusion on the feature values ​​to obtain the differential information of the two features, capturing the relationship between the features and the different degrees of response to the data.

[0054] The fusion function is defined as follows:

[0055] in, The feature vector representing the optical image, is the processed optical image feature, represents the original synthetic aperture radar image features, is the processed synthetic aperture radar image feature, is the fusion function, is the concatenation function.

[0056] Two eigenvectors are obtained by fusion function u and v , which captures information from optical remote sensing images (synthetic aperture radar images) and generated synthetic aperture radar images (optical images). This module can further fuse the feature vectors of the two modalities to achieve cross-modal information interaction, thereby generating feature vectors with richer semantics and stronger expressiveness, and improving the performance of the model.

[0057] In this embodiment, feature fusion helps reduce the differences between modalities, allowing the model to better understand and match the image content of different modalities. The model can learn a unified feature space, which helps improve the accuracy and robustness of cross-modal retrieval. It should be noted that although the remote sensing images from different sources have been mapped to a latent space through the cross-domain image generation module and the feature fusion module, and the global correspondence between the optical image and the synthetic aperture radar image has been mined, such global information ignores the semantic structure.

[0058] As a preferred embodiment, in S105, in order to better mine and establish semantic consistency correspondences between different data domains, the optical target domain fusion features and the radar target domain fusion features are semantically aligned according to the alignment module to obtain aligned radar target domain fusion features, specifically including: The similarity between the fusion feature vector in the optical target domain and the fusion feature vector in the radar target domain is calculated based on the similarity calculation formula; Adjust radar feature weights of radar target domain fusion features according to similarity; The radar target domain fusion features are aligned according to the radar feature weights to obtain the aligned radar target domain fusion features.

[0059] The alignment module mines the correspondence between optical image segments and synthetic aperture radar image segments, not only performing cross-domain alignment on the image domain, but also aligning the semantic structure in the cross-domain shared space.

[0060] In a specific embodiment, although the optical image and the synthetic aperture radar image in the industrial category are different descriptions of the same scene, there are significant visual differences. In order to better mine and establish the semantic consistency correspondence between the optical image and the synthetic aperture radar image, an alignment module is used to align the target representation at the semantic level, strengthen the semantic correspondence between the optical image fragment and the synthetic aperture radar image fragment, and reduce the difference between the hash codes calculated subsequently.

[0061] Specifically, the alignment module is used to further constrain the relationship between samples, that is, industrial optical images and synthetic aperture radar images belonging to the same category are brought closer together, and synthetic aperture radar images of different categories are pushed further away. The optical image and synthetic aperture radar image are not only aligned in the image space, but also further refined in the feature space.

[0062] like Figure 3 As shown, Figure 3 This is a schematic diagram of the structure of an embodiment of the alignment module structure provided by the present invention, which guides the radar target domain fusion feature through the optical target domain fusion feature. Given the optical target domain fusion feature 𝑢 , 𝑢 Fusion of features with radar target domain 𝑣 The similarity between the two can be calculated using the following formula: (11) in, express u and v The similarity of represents the dot product operation, ‖ ‖ represents the vector modulus operation. Then according to u , v The similarity between them is calculated by using the sigmoid function to construct different weights. Specifically, the cosine similarity is mapped to the range of (0, 1) by the sigmoid function, and the weights in the alignment process are dynamically adjusted according to the degree of similarity. The process can be formulated as: (S(u , v)) (12) in, represents the weight parameter, ( ) represents the sigmoid function, The higher the similarity between the optical target domain fusion feature and the radar target domain fusion feature, the greater the corresponding weight, and vice versa. Alignment of radar target domain fusion features The radar target domain fusion feature V can be obtained by The formula is as follows: V=v× (13) Therefore, the alignment module can enhance the semantic consistency relationship between optical images and SAR images. u , V is input to the hashing layer, which consists of tanh Activation function implementation to obtain the hash code of the optical image and hash codes of synthetic aperture radar images , where K represents the number of hash code bits.

[0063] The alignment module fully aligns the optical image features and the synthetic aperture radar image features, and then optimizes the data distribution of the two, making the feature distribution of the source domain and the target domain as close as possible. Therefore, the alignment module can ensure that the features of images of different modalities are consistent in the feature space. In cross-modal image retrieval, the alignment module optimizes the distribution of the feature space through semantic alignment and hash quantization loss, so that image features of the same category are closer in the feature space, and image features of different categories remain separated, which helps the model to more accurately identify and match cross-modal image content in retrieval tasks.

[0064] As a preferred embodiment, in S106, the total model loss function value L includes: semantic alignment loss function and As well as the orthogonal constraint loss function and hash quantization loss function of the category.

[0065] and We learn semantic structures between optical and synthetic aperture radar images, which can be compressed into hash codes with shared semantics in Hamming space.

[0066] The cross-modal hash loss function needs to effectively handle the visual semantic differences when mapping the optical image and the synthetic aperture radar image into hash codes in the common Hamming space, that is, to maintain the intra-class similarity. Due to the different imaging technologies of optical images and synthetic aperture radar images, the visual semantic differences are large, and the intra-class similarity is not easy to learn. Therefore, this application directly uses the class label as the supervisory signal, and directly uses the class label to guide the hash learning through the semantic consistency loss function, so as to fully capture the complex semantics of the cross-modal remote sensing image. Specifically, Figure 4 As shown, Figure 4This is a schematic diagram of the result of an embodiment of the intra-class similarity classification result provided by the present invention. In the Hamming space, the optical image and synthetic aperture radar images belong In order to maintain the intra-class similarity between optical images and SAR images, the semantic consistency loss function constrains and Around the class vector Distribution, can be and Mapped into similar hash codes, effectively eliminating the differences between modalities, where is a binary vector of categories. In addition, the optical image and synthetic aperture radar images do not belong to the same category, the semantic consistency loss function increases and The Hamming distance between them is calculated to maintain the distance between classes, where * { ，y} .

[0067] The semantic information provided by the category-level labels in the dataset can help the network to correctly classify the samples. It can learn hash codes with shared semantics in the Hamming space and make images of the same category close together and images of different categories far apart, that is, it maintains the intra-class similarity and inter-class differences of optical images and synthetic aperture radar images, and maintains the semantic similarity between different modalities. The hash network is optimized using class labels and the softmax function. The output of the softmax function is a probability distribution of multi-class labels, which is defined as follows: (14) in, Indicates The normalized probability of samples is Represents the category score vector The j elements, among which j∈{1，2，…，l} , Represents the class embedding vector and the eigenvector h Optimizing the loss is equivalent to optimizing In the retrieval task, It can be expressed as: (15) In the formula in j represents a class, and express or Based on formula (15), if is approximately a constant, then and There is a positive correlation, and the formula can be obtained as follows: (16) In order to make Close to a constant, let and Orthogonal, we can get: 0(17) According to formulas (16) and (17), we can get the maximum The optimal solution for classification can be approximated, namely and Parallel and in the same direction, the formula can be obtained as follows: ⊥ ，k {1, 2, …, l}∧k≠j (18) in, j represents a certain category. The cross-modal retrieval task of optical images and synthetic aperture radar images requires optimizing all category labels, which means that the columns of 𝐸 are orthogonal to each other. In addition, the Hamming distance Ham ( , ) is linearly related to the inner product as follows: (19) In the formula is the length of the hash code. Based on the above formula, the softmax loss can be expressed as follows: (20) From formula (20), we can know that the maximum log Classification and hash learning can be optimized simultaneously, that is, the hash codes of optical images and synthetic aperture radar images of the same category are close to the common class vector, thereby aligning the semantic information.

[0068] Therefore, the total loss function is obtained according to formulas (14), (16), (18), and (19). Specifically, it includes the semantic alignment loss and Class Orthogonality Constraint Loss and hash quantization loss . and The semantic structure between optical images and synthetic aperture radar images is learned, and the two can be compressed into hash codes with shared semantics in the Hamming space. And these two loss functions can maintain the intra-class similarity and inter-class distinction between optical images and synthetic aperture radar images. Semantic alignment loss function and The formula is as follows: (twenty one) (twenty two) yes The orthogonality constraint loss constrains two vectors to be orthogonal by making their inner product close to zero. The definition is as follows: (twenty three) in represents the identity matrix, and is the vector of the jth and kth categories in the class embedding vector E, yes The Euclidean norm between and the identity matrix I is used to measure the difference between the two matrices.

[0069] Hash quantization loss The definition is as follows: (twenty four) in Calculate the absolute value of an element, represents L1 normalization. Reduces the quantization error between continuous features and binary features. However, for generating and of sign (∙) The function is not differentiable at zero, which affects the optimization of the backpropagation algorithm and the update of the hash network. To solve this problem, a smooth function is used instead: , formula (24) can be rewritten as follows: (25) Quantize the loss for hashing, l is the dimension of the feature vector, i For the i samples, K is the number of hash code bits, t For the t hash code, cosh x express x The hyperbolic cosine function, ∑ represents the sum, Indicates i Samples andt The feature vector corresponding to the hash code.

[0070] To sum up, the total loss function is: (26) This application uses a deep learning model to learn cross-modal feature representations between optical images and synthetic aperture radar images, thereby improving the accuracy of cross-modal image retrieval. This cross-modal feature learning is particularly important for remote sensing image analysis, because remote sensing image data usually contains multiple modalities and needs to be effectively fused and utilized.

[0071] Through the above method, firstly, the radar target domain / optical target domain is generated by the cross-modal image generation module, so as to realize the cross-modal processing of the input data; then, the feature fusion module is used to fuse the radar target domain / optical target domain and the optical image dataset / radar image dataset, so as to obtain the optical target domain fusion features and radar target domain fusion features which contain both optical and radar features; next, the optical target domain fusion features and the radar target domain fusion features are semantically aligned, so as to fully capture the complex semantics of the cross-modal remote sensing images and ensure that the features of images of different modalities are more consistent at the feature level; by taking the loss value as the training quantitative indicator of the retrieval model, it can be ensured that the model accurately represents the cross-modal image similarity, thereby ensuring the reliability of the cross-modal remote sensing image retrieval.

[0072] In order to solve the above problems, the present invention also provides a cross-modal remote sensing image retrieval device, such as Figure 5 As shown, Figure 5 The cross-modal remote sensing image retrieval device 500 is a structural block diagram of an embodiment of the cross-modal remote sensing image retrieval device provided by the present invention. The cross-modal remote sensing image retrieval device 500 includes: The data set acquisition module 501 is used to acquire a remote sensing image data set, where the remote sensing image data set includes an optical image data set and a radar image data set; A retrieval model construction module 502 is used to construct a retrieval model, including a cross-modal image generation module, a feature fusion module, an alignment module, a hash layer and a loss function module; The optical target domain fusion feature module 503 is used to input the optical image data set into the retrieval model, generate the radar target domain based on the cross-modal image generation module, and fuse the radar target domain and the optical image data set based on the feature fusion module to obtain the optical target domain fusion feature; The radar target domain fusion feature module 504 is used to input the radar image data set into the retrieval model, generate the optical target domain based on the cross-modal image generation module, and fuse the optical target domain and the radar image data set based on the feature fusion module to obtain the radar target domain fusion feature; A semantic alignment module 505 is used to semantically align the optical target domain fusion feature and the radar target domain fusion feature according to the alignment module to obtain an aligned radar target domain fusion feature; The target retrieval model determination module 506, the hash layer calculates the optical hash code and the radar hash code according to the optical target domain fusion feature and the aligned radar target domain fusion feature, respectively, calculates the loss value between the optical hash code and the radar hash code according to the loss function module, iterates the training, and determines the retrieval model with the smallest loss value as the target retrieval model; The retrieval module 507 is used to retrieve the optical image to be retrieved / the radar image to be retrieved based on the target retrieval model to obtain the corresponding radar image data / optical image data.

[0073] like Figure 6 As shown, the present invention also provides an electronic device 600. The electronic device 600 includes a processor 601, a memory 602 and a display 603. Figure 6 Only some components of the electronic device 600 are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0074] In some embodiments, the processor 601 may be a central processing unit (CPU), a microprocessor or other data processing chip, used to run program codes or process data stored in the memory 602, such as the cross-modal remote sensing image retrieval method of the present invention.

[0075] In some embodiments, the processor 601 may be a single server or a server group. The server group may be centralized or distributed. In some embodiments, the processor 601 may be local or remote. In some embodiments, the processor 601 may be implemented in a cloud platform. In one embodiment, the cloud platform may include a private cloud, a public cloud, a hybrid cloud, a community cloud, a distributed cloud, an internal cloud, a multi-cloud, etc., or any combination thereof.

[0076] In some embodiments, the memory 602 may be an internal storage unit of the electronic device 600, such as a hard disk or memory of the electronic device 600. In other embodiments, the memory 602 may also be an external storage device of the electronic device 600, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc., equipped on the electronic device 600.

[0077] Furthermore, the memory 602 may include both an internal storage unit of the electronic device 600 and an external storage device. The memory 602 is used to store application software installed in the electronic device 600 and various data.

[0078] In some embodiments, the display 603 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, etc. The display 603 is used to display information of the electronic device 600 and to display a visual user interface. The components 601-603 of the electronic device 600 communicate with each other through a system bus.

[0079] In one embodiment, when the processor 601 executes the cross-modal remote sensing image retrieval program in the memory 602, the following steps may be implemented: Acquire a remote sensing image dataset, where the remote sensing image dataset includes an optical image dataset and a radar image dataset; Build a retrieval model, including a cross-modal image generation module, a feature fusion module, an alignment module, a hash layer, and a loss function module; The optical image dataset is input into the retrieval model, the radar target domain is generated based on the cross-modal image generation module, and the radar target domain and the optical image dataset are fused based on the feature fusion module to obtain the optical target domain fusion feature; The radar image dataset is input into the retrieval model, the optical target domain is generated based on the cross-modal image generation module, and the optical target domain and the radar image dataset are fused based on the feature fusion module to obtain the radar target domain fusion feature; According to the alignment module, the optical target domain fusion features and the radar target domain fusion features are semantically aligned to obtain the aligned radar target domain fusion features; The hash layer calculates the optical hash code and the radar hash code respectively according to the optical target domain fusion features and the aligned radar target domain fusion features, calculates the loss value between the optical hash code and the radar hash code according to the loss function module, iterates the training, and determines the retrieval model with the smallest loss value as the target retrieval model; The optical image to be retrieved / the radar image to be retrieved is retrieved based on the target retrieval model to obtain the corresponding radar image data / optical image data.

[0080] It should be understood that: when the processor 601 executes the cross-modal remote sensing image retrieval program in the memory 602, in addition to the above functions, other functions may also be implemented, and details may refer to the description of the corresponding method embodiments above.

[0081] Furthermore, the embodiment of the present invention does not specifically limit the type of the electronic device 600 mentioned, and the electronic device 600 may be a portable electronic device such as a mobile phone, a tablet computer, a personal digital assistant (PDA), a wearable device, a laptop computer, etc. Exemplary embodiments of portable electronic devices include but are not limited to portable electronic devices equipped with IOS, Android, Microsoft or other operating systems. The above-mentioned portable electronic device may also be other portable electronic devices, such as a laptop computer with a touch-sensitive surface (e.g., a touch panel). It should also be understood that in some other embodiments of the present invention, the electronic device 600 may not be a portable electronic device, but a desktop computer with a touch-sensitive surface (e.g., a touch panel).

[0082] Accordingly, an embodiment of the present invention also provides a computer-readable storage medium, which is used to store computer-readable programs or instructions. When the program or instructions are executed by a processor, the steps or functions in the cross-modal remote sensing image retrieval method provided by the above-mentioned method embodiments can be implemented.

[0083] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware (such as a processor, a controller, etc.) through a computer program, and the computer program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0084] The cross-modal remote sensing image retrieval method, device, electronic device and storage medium provided by the present invention are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A cross-modal remote sensing image retrieval method, characterized in that: include: Acquire a remote sensing image dataset, wherein the remote sensing image dataset includes an optical image dataset and a radar image dataset; Build a retrieval model, including a cross-modal image generation module, a feature fusion module, an alignment module, a hash layer, and a loss function module; Inputting the optical image data set into the retrieval model, generating a radar target domain based on the cross-modal image generation module, and fusing features of the radar target domain and the optical image data set based on the feature fusion module to obtain an optical target domain fusion feature; Inputting the radar image dataset into the retrieval model, generating an optical target domain based on the cross-modal image generation module, and fusing features of the optical target domain and the radar image dataset based on the feature fusion module to obtain radar target domain fusion features; Semantically aligning the optical target domain fusion feature and the radar target domain fusion feature according to the alignment module to obtain an aligned radar target domain fusion feature; The hash layer calculates an optical hash code and a radar hash code respectively according to the optical target domain fusion feature and the aligned radar target domain fusion feature, calculates a loss value between the optical hash code and the radar hash code according to the loss function module, iterates training, and determines that the retrieval model with the smallest loss value is the target retrieval model; The optical image to be retrieved / the radar image to be retrieved is retrieved based on the target retrieval model to obtain corresponding radar image data / optical image data.

2. The cross-modal remote sensing image retrieval method according to claim 1, characterized in that: The cross-modal image generation module includes a denoising network submodule and an autoencoder submodule; The generating a radar target domain based on the cross-modal image generation module includes: Performing denoising processing on the optical image data set based on the denoising network submodule to obtain a denoised optical image; In the latent space, the denoised optical image is encoded and decoded based on the autoencoder submodule to reconstruct the radar target domain of the optical image dataset.

3. The cross-modal remote sensing image retrieval method according to claim 1, characterized in that: The feature fusion module includes a deep learning module and a fusion submodule; The step of fusing the radar target domain and the optical image data set based on the feature fusion module to obtain an optical target domain fusion feature includes: Based on the deep learning module, key features of the radar target domain and the optical image data set are extracted to obtain key features of the radar target domain and key features of the optical image data set respectively; Based on the fusion submodule, feature fusion is performed on the radar target domain key features and the optical image data set key features to obtain optical target domain image fusion features.

4. The cross-modal remote sensing image retrieval method according to claim 3, characterized in that: The deep learning module includes a multi-scale residual module and a context attention module; the key feature extraction of the radar target domain and the optical image data set based on the deep learning module to obtain the key features of the radar target domain and the key features of the optical image data set respectively includes: Adaptively detecting the radar target domain and the optical image data set according to the multi-scale residual module to obtain multi-scale residual radar features and multi-scale residual optical features respectively; The multi-scale residual radar features and the multi-scale residual optical features are targeted and focused according to the contextual attention module to obtain radar target domain key features and optical image data set key features respectively.

5. The cross-modal remote sensing image retrieval method according to claim 1, characterized in that: The semantically aligning the optical target domain fusion feature and the radar target domain fusion feature according to the alignment module to obtain the aligned radar target domain fusion feature includes: Calculating the similarity between the optical target domain fusion feature vector and the radar target domain fusion feature vector based on a similarity calculation formula; adjusting the radar feature weight of the radar target domain fusion feature according to the similarity; The radar target domain fusion features are aligned according to the radar feature weights to obtain the aligned radar target domain fusion features.

6. The cross-modal remote sensing image retrieval method according to claim 1, characterized in that: The loss function module includes a semantic alignment loss function, an orthogonal constraint loss function and a hash quantization loss function; and the loss value between the optical hash code and the radar hash code is calculated according to the loss function module, including: Calculating a semantic alignment loss between the optical hash code and the radar hash code according to the semantic alignment loss function; Calculating the orthogonal constraint loss between the optical target domain fusion feature and the radar target domain fusion feature according to the orthogonal constraint loss function; Calculating a hash quantization loss between the optical hash code and the radar hash code according to the hash quantization loss function; The loss value is determined according to the semantic alignment loss, the orthogonality constraint loss and the hash quantization loss.

7. The cross-modal remote sensing image retrieval method according to claim 6, characterized in that: The calculating the hash quantization loss between the optical hash code and the radar hash code according to the hash quantization loss function includes: The initial hash quantization loss function is updated by a smoothing function to obtain the hash quantization loss function; The calculation formula of the hash quantization loss function is: Quantify the loss for the hash, l is the dimension of the feature vector, i For the i samples, K is the number of hash code bits, t For the t hash code, cosh x express x The hyperbolic cosine function, ∑ represents the sum, Indicates i Samples and t The feature vector corresponding to the hash code.

8. A cross-modal remote sensing image retrieval device, characterized in that: include: A data set acquisition module, used to acquire a remote sensing image data set, wherein the remote sensing image data set includes an optical image data set and a radar image data set; A retrieval model building module is used to build a retrieval model, including a cross-modal image generation module, a feature fusion module, an alignment module, a hash layer, and a loss function module; An optical target domain fusion feature module is used to input the optical image data set into the retrieval model, generate a radar target domain based on the cross-modal image generation module, and perform feature fusion on the radar target domain and the optical image data set based on the feature fusion module to obtain an optical target domain fusion feature; A radar target domain fusion feature module is used to input the radar image data set into the retrieval model, generate an optical target domain based on the cross-modal image generation module, and perform feature fusion on the optical target domain and the radar image data set based on the feature fusion module to obtain a radar target domain fusion feature; A semantic alignment module, used for semantically aligning the optical target domain fusion feature and the radar target domain fusion feature according to the alignment module to obtain an aligned radar target domain fusion feature; A target retrieval model determination module, wherein the hash layer calculates an optical hash code and a radar hash code according to the optical target domain fusion feature and the aligned radar target domain fusion feature, respectively, calculates a loss value between the optical hash code and the radar hash code according to the loss function module, iterates training, and determines the retrieval model with the smallest loss value as the target retrieval model; The retrieval module is used to retrieve the optical image to be retrieved / the radar image to be retrieved based on the target retrieval model to obtain the corresponding radar image data / optical image data.

9. An electronic device, characterized in that: comprising a memory and a processor, wherein: The memory is used to store programs; The processor is coupled to the memory and is used to execute the program stored in the memory to implement the steps in the cross-modal remote sensing image retrieval method described in any one of claims 1 to 7 above.

10. A computer-readable storage medium, characterized in that: Used to store computer-readable programs or instructions, which, when executed by a processor, can implement the steps in the heterogeneous brain network classification method described in any one of claims 1 to 7.

Images