Cross-modal Remote Sensing Image Retrieval Method and Device Based on Maximum Correlation of HGR

Abstract

In this application, a cross-modal remote sensing image retrieval method and device based on the maximum HGR correlation are provided. The method includes: extracting features from a query remote sensing image to obtain a query Euclidean distance feature and a query HGR feature; calculating Euclidean distance scores between the query Euclidean distance feature and each retrieval Euclidean distance feature in the retrieval feature set, and calculating the maximum HGR correlation scores between the query HGR feature and each retrieval HGR feature in the retrieval feature set; based on the DS evidence theory, fusing the Euclidean distance scores and the maximum HGR correlation scores to obtain a fused mass function; sorting the retrieval remote sensing images in descending order according to the fused mass function to obtain the final retrieval result. By using the maximum HGR correlation between different modalities, the heterogeneity gap caused by modal differences can be better eliminated, and the Euclidean distance feature and the HGR feature between different modality remote sensing images are combined, enhancing the reliability of the retrieval result.

Description

Technical Field

[0001] This application relates to the field of information processing and mining of remote sensing big data, and particularly relates to a cross-modal remote sensing image retrieval method and device based on the maximum correlation of HGR. Background Art

[0002] With the explosive growth of remote sensing data, it is of great importance to obtain relevant data from large-scale remote sensing data sets. Cross-modal remote sensing image retrieval refers to retrieving remote sensing images with similar semantic information from remote sensing images of different modalities (for example, optical remote sensing images, infrared remote sensing images, etc.). Due to the differences in image resolution, image expression, optical characteristics, etc. of remote sensing images of different modalities, cross-modal remote sensing image retrieval is a very challenging problem.

[0003] In related technologies, multi-deep learning strategies are used to bridge the heterogeneity gap between different modalities. Currently, most cross-modal remote sensing image retrieval methods based on deep neural networks project data samples from different modalities into a low-dimensional common space and optimize the projection features under the Euclidean distance constraint during the training phase. Although this method has achieved results significantly better than traditional methods, only optimizing the feature representation according to the Euclidean distance may eventually result in information redundancy, that is, a few feature terms may dominate in the similarity evaluation, leading to unreliable retrieval results and reducing the retrieval accuracy.

[0004] Therefore, how to better evaluate the relationship between remote sensing images of different modalities and design a more reliable cross-modal remote sensing image retrieval method is a technical problem that needs to be urgently solved by those skilled in the art. Summary of the Invention

[0005] In view of the above problems, embodiments of this application provide a cross-modal remote sensing image retrieval method and device based on the maximum correlation of HGR, so as to overcome the above problems or at least partially solve the above problems.

[0006] In the first aspect of the embodiments of this application, a cross-modal remote sensing image retrieval method based on the maximum correlation of HGR is disclosed. The method includes:

[0007] Extract features from the query remote sensing image to obtain a query Euclidean distance feature and a query HGR feature;

[0008] Calculate the Euclidean distance scores between the query Euclidean distance feature and each retrieval Euclidean distance feature in the retrieval feature set, and calculate the maximum HGR correlation scores between the query HGR feature and each retrieval HGR feature in the retrieval feature set. The retrieval feature set is a retrieval feature set obtained by pre-extracting features from a retrieval remote sensing image set. The retrieval remote sensing images in the retrieval remote sensing image set and the query remote sensing image are remote sensing images of different modalities;

[0009] Based on the DS evidence theory, fuse and calculate the Euclidean distance scores and the maximum HGR correlation scores to obtain a fused mass function, which characterizes the relevance between the query remote sensing image and the retrieval remote sensing image;

[0010] Sort the retrieval remote sensing images in the retrieval remote sensing image set in descending order according to the fused mass function to obtain the final retrieval result.

[0011] Optionally, the step of fusing and calculating the Euclidean distance scores and the maximum HGR correlation scores based on the DS evidence theory to obtain a fused mass function includes:

[0012] Define the score hypothesis cases, which include: reliable score, unreliable score, uncertain score reliability, and empty set;

[0013] According to the score hypothesis cases, define the mass function of the Euclidean distance score in different score hypothesis cases and the mass function of the maximum HGR correlation score in different score hypothesis cases;

[0014] According to the DS evidence theory fusion rule, fuse the Euclidean distance mass function and the maximum HGR correlation mass function with an intersection in the score hypothesis cases to obtain a fused mass function.

[0015] Optionally, the step of fusing the Euclidean distance mass function and the maximum HGR correlation mass function with an intersection in the score hypothesis cases according to the DS evidence theory fusion rule includes:

[0016] For the hypothesis case of reliable score, fuse the Euclidean distance mass function with reliable score, the maximum HGR correlation mass function with reliable score, the Euclidean distance mass function with uncertain score reliability, and the maximum HGR correlation mass function with uncertain score reliability to obtain a fused mass function with reliable score;

[0017] For the hypothetical situation where the scores are unreliable, fuse the Euclidean distance quality function with unreliable scores, the HGR maximum correlation quality function with unreliable scores, the Euclidean distance quality function with uncertain score reliability, and the HGR maximum correlation quality function with uncertain score reliability to obtain a fused quality function with unreliable scores;

[0018] For the hypothetical situation where the score reliability is uncertain, fuse the Euclidean distance quality function with uncertain score reliability and the HGR maximum correlation quality function with uncertain score reliability to obtain a fused quality function with uncertain score reliability;

[0019] For the hypothetical situation where the score is an empty set, set the fused quality function of the score empty set to 0.

[0020] Optionally, sort the retrieved remote sensing images in the retrieved remote sensing image set in descending order according to the fused quality function to obtain the final retrieval result, including:

[0021] Based on the maximization plausibility criterion in the DS evidence theory, fuse the reliable score fused quality function and the unreliable score fused quality function to obtain the decision quality function of each retrieved remote sensing image;

[0022] Sort the retrieved remote sensing images in the retrieved remote sensing image set in descending order according to the decision quality function to obtain the final retrieval result.

[0023] Optionally, the cross-modal remote sensing image retrieval method based on HGR maximum correlation is implemented by a pre-trained retrieval model, and the retrieval model is trained in the following manner:

[0024] Construct a training data set, which includes: paired remote sensing images composed of first-modal remote sensing images and second-modal remote sensing images, and class labels for each pair of remote sensing images;

[0025] Input the paired remote sensing images into the retrieval model for processing to obtain the Euclidean distance features and HGR features corresponding to each modal remote sensing image, and process the Euclidean distance features and the HGR features to obtain predicted features;

[0026] Calculate the Euclidean distance loss according to the Euclidean distance features, calculate the HGR maximum correlation loss according to the HGR features, and calculate the prediction loss according to the predicted features and the corresponding class labels;

[0027] Update the parameters of the retrieval model based on the Euclidean distance loss, the HGR maximum correlation loss, and the prediction loss. After meeting the training end condition, obtain the trained retrieval model.

[0028] Optionally, the Euclidean distance loss is used to measure the difference between the Euclidean distance features corresponding to remote sensing images of different modalities. The Euclidean distance loss includes: similarity loss and ranking loss;

[0029] The similarity loss J I is expressed as:

[0030]

[0031] where N represents the total amount of a batch of training data pairs, v i , s i respectively represent the first-modal remote sensing image and the second-modal remote sensing image in the paired remote sensing images, and f E (·), g E (·) respectively represent the Euclidean distance features of the first-modal remote sensing image and the second-modal remote sensing image in the paired remote sensing images;

[0032] The ranking loss J R is expressed as:

[0033]

[0034] where d(·,·) represents the Euclidean distance, respectively represent the positive samples sampled from the first-modal remote sensing image and the second-modal remote sensing image in this batch of training data, respectively represent the negative samples sampled from the first-modal remote sensing image and the second-modal remote sensing image in this batch of training data, and margin represents the optimization margin in the triplet loss.

[0035] Optionally, the HGR maximum correlation loss is used to maximize the HGR feature correlation of remote sensing images of different modalities. The HGR maximum correlation loss includes: correlation loss and correlation ranking loss;

[0036] The correlation loss J C is expressed as:

[0037]

[0038] where tr(·) represents the trace of a matrix, cov(·) represents the covariance matrix, v i , s i respectively represent the first-modal remote sensing image and the second-modal remote sensing image in the paired remote sensing images, and f H (·), g H (·) respectively represent the HGR features of the first-modal remote sensing image and the second-modal remote sensing image in the paired remote sensing images;

[0039] The correlation sorting loss J CR is expressed as:

[0040]

[0041] where v i and s i respectively represent the first-modal remote sensing image and the second-modal remote sensing image in the paired remote sensing images, respectively represent the positive samples obtained by sampling the first-modal remote sensing image and the second-modal remote sensing image in this batch of training data, respectively represent the negative samples obtained by sampling the first-modal remote sensing image and the second-modal remote sensing image in this batch of training data.

[0042] Optionally, the prediction loss is calculated using the mean squared error loss, and the prediction loss J P is expressed as:

[0043]

[0044] where v i and s i respectively represent the first-modal remote sensing image and the second-modal remote sensing image in the paired remote sensing images, f E (·), g E (·) respectively represent the Euclidean distance features of the first-modal remote sensing image and the second-modal remote sensing image in the paired remote sensing images, f H (·), g H (·) respectively represent the HGR features of the first-modal remote sensing image and the second-modal remote sensing image in the paired remote sensing images, y i represents the class label of the paired remote sensing images.

[0045] Optionally, calculating the Euclidean distance score between the query Euclidean distance feature and each retrieval Euclidean distance feature in the retrieval feature set includes:

[0046] Calculating the Euclidean distance between the query Euclidean distance feature and each retrieval Euclidean distance feature in the retrieval feature set;

[0047] Normalizing the Euclidean distance between the query Euclidean distance feature and each retrieval Euclidean distance feature in the retrieval feature set according to the maximum and minimum values of the Euclidean distance to obtain the Euclidean distance score.

[0048] Optionally, calculating the HGR maximum correlation score between the query HGR feature and each retrieval HGR feature in the retrieval feature set includes:

[0049] Calculating the HGR maximum correlation coefficient between the query HGR feature and each retrieval HGR feature in the retrieval feature set;

[0050] According to the maximum and minimum values of the correlation coefficient, normalize the maximum HGR correlation coefficient between the query HGR feature and each retrieval HGR feature in the retrieval feature set to obtain the maximum HGR correlation score.

[0051] In a second aspect of the embodiments of the present application, a cross-modal remote sensing image retrieval device based on the maximum HGR correlation is disclosed. The device includes:

[0052] A feature extraction module, configured to extract features from a query remote sensing image to obtain a query Euclidean distance feature and a query HGR feature;

[0053] A score calculation module, configured to calculate the Euclidean distance score between the query Euclidean distance feature and each retrieval Euclidean distance feature in the retrieval feature set, and calculate the maximum HGR correlation score between the query HGR feature and each retrieval HGR feature in the retrieval feature set. The retrieval feature set is a retrieval feature set obtained by pre-extracting features from a retrieval remote sensing image set, and the retrieval remote sensing images in the retrieval remote sensing image set and the query remote sensing image are remote sensing images of different modalities;

[0054] A score fusion module, configured to fuse and calculate the Euclidean distance score and the maximum HGR correlation score based on the DS evidence theory to obtain a fused mass function, where the fused mass function characterizes the relevance between the query remote sensing image and the retrieval remote sensing image;

[0055] A result sorting module, configured to sort the retrieval remote sensing images in the retrieval remote sensing image set in descending order according to the fused mass function to obtain a final retrieval result.

[0056] The embodiments of the present application have the following advantages:

[0057] In the embodiments of the present application, the Euclidean distance correlation relationship and the maximum HGR correlation between remote sensing images of different modalities are considered simultaneously to better eliminate the influence caused by modality differences, solve the problem of information redundancy, and achieve high-precision cross-modal remote sensing image retrieval.

[0058] When performing cross-modal remote sensing image retrieval, by extracting features from the query remote sensing image, the query Euclidean distance feature and the query HGR feature are obtained. Then, the Euclidean distance scores between the query Euclidean distance feature and each retrieval Euclidean distance feature in the retrieval feature set, and the maximum HGR correlation scores between the query HGR feature and each retrieval HGR feature in the retrieval feature set are calculated respectively. Furthermore, using the DS evidence theory, the Euclidean distance scores and the maximum HGR correlation scores are fused and calculated to obtain a fusion quality function that can represent the relevance between the query remote sensing image and the retrieval remote sensing image, and the final retrieval result is obtained by sorting in descending order according to the fusion quality function.

[0059] Compared with the existing methods, the present application further utilizes the maximum HGR correlation between different modalities to better eliminate the heterogeneity gap caused by modality differences, combines the Euclidean distance features and HGR features between different modality remote sensing images, and enhances the reliability of the retrieval results. At the same time, the final retrieval result is determined based on the fusion quality function calculated by the DS evidence theory, rather than simply sorting the Euclidean distance or cosine similarity, which further ensures the reliability of the final detection result. Brief Description of the Drawings

[0060] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the accompanying drawings required for the description of the embodiments of the present application will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0061] Figure 1 It is a flowchart of the steps of a cross-modal remote sensing image retrieval method based on the maximum HGR correlation provided by the embodiments of the present application;

[0062] Figure 2 It is a schematic flowchart of a cross-modal remote sensing image retrieval method based on the maximum HGR correlation provided by the embodiments of the present application;

[0063] Figure 3 It is a schematic flowchart of the training process of a retrieval model for realizing cross-modal remote sensing image retrieval provided by the embodiments of the present application;

[0064] Figure 4 It is a schematic structural diagram of a cross-modal remote sensing image retrieval device based on the maximum HGR correlation provided by the embodiments of the present application. Detailed Embodiments

[0065] To make the above objects, features, and advantages of the present application more apparent and understandable, the following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of them. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present application.

[0066] Referring to Figure 1 as shown, Figure 1 FIG. shows a flowchart of the steps of a cross-modal remote sensing image retrieval method based on HGR maximum correlation provided by an embodiment of the present application. As Figure 1 shown, a cross-modal remote sensing image retrieval method based on HGR maximum correlation provided by an embodiment of the present application may specifically include step S110 and step S140:

[0067] Step S110: Extract features from the query remote sensing image to obtain a query Euclidean distance feature and a query HGR feature.

[0068] In the embodiments of the present application, the goal of content-based cross-modal remote sensing image retrieval is to give a query remote sensing image of a certain modality (such as an optical remote sensing image) and search for the most semantically relevant remote sensing image in a remote sensing image set of another modality (such as a SAR synthetic aperture radar remote sensing image). Among them, semantically relevant remote sensing images refer to those with relevant content represented in the remote sensing images. For example, if two remote sensing images of different modalities both correspond to a farmland area, then these two remote sensing images are semantically relevant; if one modality remote sensing image corresponds to a farmland area and the other modality remote sensing image corresponds to a highway area, then these two remote sensing images are not semantically relevant.

[0069] The query remote sensing image can be any modality of remote sensing image. For example, the query remote sensing image can be any one of an optical remote sensing image, a SAR remote sensing image, and an infrared remote sensing image. The Euclidean distance feature is a feature optimized based on the Euclidean distance, and the HGR feature is a feature optimized based on the HGR maximum correlation. The HGR feature represents the orthogonal correlation information between different modalities.

[0070] In specific implementation, the query remote sensing image is input into the corresponding modal convolutional neural network for feature extraction to obtain the Euclidean distance feature and HGR feature corresponding to the query remote sensing image, that is, the query Euclidean distance feature and the query HGR feature. For example, the optical remote sensing image is input into the convolutional neural network of the optical remote sensing image for feature extraction to obtain the Euclidean distance feature and HGR feature of the optical remote sensing image. Among them, the convolutional neural network for feature extraction (for example, the AlexNet network) can be a network pre-trained independently of the retrieval model in the embodiment of the present application, or a sub-network integrated in the retrieval model in the embodiment of the present application.

[0071] Step S120: Calculate the Euclidean distance score between the query Euclidean distance feature and each retrieval Euclidean distance feature in the retrieval feature set, and calculate the HGR maximum correlation score between the query HGR feature and each retrieval HGR feature in the retrieval feature set. The retrieval feature set is a retrieval feature set obtained by pre-extracting features from a retrieval remote sensing image set. The retrieval remote sensing images in the retrieval remote sensing image set and the query remote sensing image are remote sensing images of different modalities.

[0072] In the embodiment of the present application, the retrieval remote sensing image set contains a large number of other modal remote sensing images different from the query remote sensing image. For example, if the retrieval remote sensing image is an optical remote sensing image, the images in the retrieval remote sensing image set are all other modal remote sensing images different from the optical remote sensing image, such as SAR remote sensing images, infrared remote sensing images, etc.

[0073] The retrieval feature set includes an Euclidean distance feature set and an HGR feature set. Each Euclidean distance feature in the Euclidean distance feature set corresponds to the HGR feature in the HGR feature set. The corresponding Euclidean distance feature and HGR feature jointly represent the feature of a retrieval remote sensing image. Specifically, the retrieval remote sensing images in the retrieval remote sensing image set are input into the corresponding convolutional neural network for feature extraction to obtain the retrieval Euclidean distance feature and retrieval HGR feature corresponding to each retrieval remote sensing image. All the retrieval Euclidean distance features form the Euclidean distance feature set, and all the retrieval HGR features form the retrieval HGR feature set.

[0074] Among them, the Euclidean distance score represents the Euclidean distance correlation relationship between the query remote sensing image and the retrieval remote sensing image, and the HGR maximum correlation score represents the maximum HGR correlation between the query remote sensing image and the retrieval remote sensing image. By calculating the Euclidean distance score and the HGR maximum correlation score, the relationship between different modal remote sensing images is evaluated to obtain a more reliable retrieval result in subsequent steps.

[0075] In an alternative embodiment, the calculating the Euclidean distance score between the query Euclidean distance feature and each retrieval Euclidean distance feature in the retrieval feature set includes:

[0076] Calculate the Euclidean distance between the query Euclidean distance feature and each retrieval Euclidean distance feature in the retrieval feature set; normalize the Euclidean distance between the query Euclidean distance feature and each retrieval Euclidean distance feature in the retrieval feature set according to the maximum and minimum values of the Euclidean distance to obtain the Euclidean distance score.

[0077] Specifically, the calculation of the Euclidean distance between the query Euclidean distance feature and the retrieval Euclidean distance feature can be expressed as:

[0078]

[0079] where, v q represents the query remote sensing image, v j represents the j-th retrieval remote sensing image in the retrieval feature set, f E (v q ) represents the Euclidean distance feature of the query remote sensing image, f E (v q ) k represents the k-th element of the Euclidean distance feature of the query remote sensing image, g E (v j ) represents the Euclidean distance feature of the j-th retrieval remote sensing image, g E (v j ) k represents the k-th element of the Euclidean distance feature of the j-th retrieval remote sensing image.

[0080] Moreover, for the convenience of unified comparison, after calculating the Euclidean distance between each retrieval Euclidean distance feature in the retrieval feature set and the query Euclidean distance feature, the maximum and minimum values of the Euclidean distance are determined, and then normalization is performed according to the maximum and minimum values to process each Euclidean distance into a Euclidean distance score between 0 and 1.

[0081] In an alternative embodiment, the calculation of the HGR maximum correlation score between the query HGR feature and each retrieval HGR feature in the retrieval feature set includes:

[0082] Calculate the correlation coefficient between the query HGR feature and each retrieval HGR feature in the retrieval feature set; normalize the correlation coefficient between the query HGR feature and each retrieval HGR feature in the retrieval feature set according to the maximum and minimum values of the correlation coefficient to obtain the HGR maximum correlation score.

[0083] Specifically, the correlation coefficient between the query HGR feature and each retrieval HGR feature in the retrieval feature set can be expressed as:

[0084]

[0085] Among them, f H (v q ) represents the HGR feature of the query remote sensing image, and g H (s j ) represents the Euclidean HGR feature of the j-th retrieved remote sensing image.

[0086] Similarly, for the sake of unified comparison, after calculating the correlation coefficient between each retrieved HGR feature and the query HGR feature in the retrieved feature set, the maximum and minimum values of the correlation coefficient are determined, and then normalization processing is performed according to the maximum and minimum values to process each correlation coefficient into an HGR maximum correlation score between 0 and 1.

[0087] Step S130: Based on the DS evidence theory, fuse and calculate the Euclidean distance score and the HGR maximum correlation score to obtain a fused mass function, and the fused mass function characterizes the relevance between the query remote sensing image and the retrieved remote sensing image.

[0088] In the embodiment of the present application, the Euclidean distance score and the HGR maximum correlation score can reflect the relevance between the query remote sensing image and the retrieved remote sensing image from different perspectives. In order to obtain a reliable retrieval result, the Euclidean distance score and the HGR maximum correlation score are fused using the DS evidence theory to obtain a fused mass function that can more comprehensively reflect the relevance between the query remote sensing image and the retrieved remote sensing image.

[0089] In an alternative embodiment, the fusing and calculating the Euclidean distance score and the HGR maximum correlation score based on the DS evidence theory to obtain a fused mass function includes steps S130-1 to S130-3:

[0090] Step S130-1: Define the score hypothesis cases, and the score hypothesis cases include: score reliable, score unreliable, score reliability uncertain, and empty set.

[0091] In the embodiment of the present application, for each Euclidean distance score and each HGR maximum correlation score, there are four hypotheses: score reliable, score unreliable, score reliability uncertain, and empty set. Among them, the empty set refers to the situation where the score does not belong to the three hypothesis cases of score reliable, score unreliable, and score reliability uncertain.

[0092] Specifically, the score hypothesis cases can be expressed as:

[0093] A1: Score reliable

[0094] A2: Score unreliable

[0095] A3: Score reliability is uncertain

[0096] A4: Empty set

[0097] Step S130-2: According to the score hypothesis situation, define the quality function of the Euclidean distance score in different score hypothesis situations and the quality function of the HGR maximum correlation score in different score hypothesis situations.

[0098] Specifically, the Euclidean distance quality function is expressed as:

[0099] m1(A1) = b1 s EUC

[0100] m1(A2) = b1(1 - s EUC )

[0101] m1(A3) = 1 - b1

[0102] m1(A4) = 0

[0103] Among them, m1(A1) represents the Euclidean distance quality function with reliable scores, m1(A2) represents the Euclidean distance quality function with unreliable scores, m1(A3) represents the Euclidean distance quality function with uncertain score reliability, m1(A4) represents the Euclidean distance quality function of the empty set, s EUC represents the Euclidean distance score, b1 represents the hyperparameter that controls the proportion of the Euclidean distance score, and b1 is set according to the actual situation. When the hyperparameter that controls the proportion of the Euclidean distance score is determined, the Euclidean distance quality function is a function that only relates to the Euclidean distance score. Substitute the Euclidean distance score into the functions of different score hypothesis situations to obtain the Euclidean distance quality function corresponding to each Euclidean distance score.

[0104] The HGR maximum correlation quality function is expressed as:

[0105] m2(A1) = b2s HGR

[0106] m2(A2) = b2(1 - s HGR )

[0107] m2(A3) = 1 - b2

[0108] m2(A4) = 0

[0109] Among them, m2(A1) represents the HGR maximum correlation quality function with reliable scores, m2(A2) represents the HGR maximum correlation quality function with unreliable scores, m2(A3) represents the HGR maximum correlation quality function with uncertain score reliability, m2(A4) represents the HGR maximum correlation quality function of the empty set, s HGRLet \(HGR\) denote the maximum correlation score, and \(b_2\) denote the hyperparameter that controls the proportion of the \(HGR\) maximum correlation score. \(b_2\) is set according to the actual situation. When the hyperparameter that controls the proportion of the \(HGR\) maximum correlation score is determined, the \(HGR\) maximum correlation quality function is a function that only depends on the \(HGR\) maximum correlation score. Substitute the \(HGR\) maximum correlation score into the functions of different score hypothesis cases to obtain the \(HGR\) maximum correlation quality function corresponding to each \(HGR\) maximum correlation score.

[0110] Step S130-3: According to the DS evidence theory fusion rule, fuse the Euclidean distance quality function and the \(HGR\) maximum correlation quality function where the score hypothesis cases have an intersection to obtain a fused quality function.

[0111] In the embodiments of the present application, the situations where the score hypothesis cases have an intersection specifically include: the intersection between reliable scores is a reliable score, the intersection between a reliable score and an uncertain score reliability is a reliable score, the intersection between unreliable scores is an unreliable score, and the intersection between uncertain score reliabilities is an uncertain score reliability.

[0112] Exemplarily, the fused quality function can be expressed as:

[0113]

[0114] Among them, \(m\) represents the fused quality function, \(K\) represents the sum of the quality functions of all score hypotheses with non-empty intersections, \(A\) is the hypothesis situation of the score, including reliable score \(A_1\), unreliable score \(A_2\), uncertain score reliability \(A_3\), and empty set \(A_4\).

[0115] Furthermore, the fusing of the Euclidean distance quality function and the \(HGR\) maximum correlation quality function where the score hypothesis cases have an intersection according to the DS evidence theory fusion rule includes four cases of \(A_1\) to \(A_4\):

[0116] A1: For the hypothesis situation of a reliable score, fuse the Euclidean distance quality function of the reliable score, the \(HGR\) maximum correlation quality function of the reliable score, the Euclidean distance quality function of the uncertain score reliability, and the \(HGR\) maximum correlation quality function of the uncertain score reliability to obtain a reliable score fused quality function.

[0117] Specifically, there is an intersection between the score-reliable Euclidean distance quality function m1(A1) and the score-reliable HGR maximum correlation quality function m2(A1), an intersection between the score-reliable Euclidean distance quality function m1(A1) and the HGR maximum correlation quality m2(A3) with uncertain score reliability, and an intersection between the Euclidean distance quality function m1(A3) with uncertain score reliability and the score-reliable HGR maximum correlation quality function m2(A1). Therefore, the score-reliable fusion quality function m(A1) is expressed as:

[0118]

[0119] A2: For the hypothetical situation of unreliable scores, fuse the unreliable Euclidean distance quality function, the unreliable HGR maximum correlation quality function, the Euclidean distance quality function with uncertain score reliability, and the HGR maximum correlation quality function with uncertain score reliability to obtain an unreliable score fusion quality function.

[0120] Specifically, there is an intersection between the unreliable Euclidean distance quality function m1(A2) and the unreliable HGR maximum correlation quality function m2(A2), an intersection between the unreliable Euclidean distance quality function m1(A2) and the HGR maximum correlation quality m2(A3) with uncertain score reliability, and an intersection between the Euclidean distance quality function m1(A3) with uncertain score reliability and the unreliable HGR maximum correlation quality function m2(A2). Therefore, the unreliable score fusion quality function m(A2) is expressed as:

[0121]

[0122] A3: For the hypothetical situation of uncertain score reliability, fuse the Euclidean distance quality function with uncertain score reliability and the HGR maximum correlation quality function with uncertain score reliability to obtain a fusion quality function with uncertain score reliability.

[0123] Specifically, the fusion quality function m(A3) with uncertain score reliability is expressed as:

[0124]

[0125] A4: For the hypothetical situation of an empty score set, set the fusion quality function of the empty score set to 0. That is:

[0126] m(A4) = 0

[0127] Step S140: Sort the retrieved remote sensing images in the retrieved remote sensing image set in descending order according to the fusion quality function to obtain the final retrieval result.

[0128] In the embodiments of the present application, the fusion quality function characterizes the relevance between the query remote sensing image and the retrieved remote sensing image. The larger the fusion quality function, the greater the relevance between the query remote sensing image and the retrieved remote sensing image. Sorting is performed according to the order of the fusion quality function from large to small. Therefore, in the final retrieval result, the retrieved remote sensing images ranked higher are the images with stronger relevance to the query remote sensing image. Since the final retrieval result is determined based on the fusion quality function calculated by the DS evidence theory, rather than simply sorting the Euclidean distance or cosine similarity, the reliability of the final retrieval result is ensured.

[0129] In an alternative embodiment, the quality functions corresponding to two hypothetical situations of score reliability and score unreliability are used for decision-making. Specifically, sorting is performed according to the order of the fusion quality function from large to small to obtain the final retrieval result, including:

[0130] Based on the maximization plausibility criterion in the DS evidence theory, the score-reliable fusion quality function and the score-unreliable fusion quality function are fused to obtain the decision quality function of each retrieved remote sensing image; sorting the retrieved remote sensing images in the retrieved remote sensing image set is performed according to the order of the decision quality function from large to small to obtain the final retrieval result.

[0131] Among them, fusing the score-reliable fusion quality function m(A1) and the score-unreliable fusion quality function m(A3) means adding the score-reliable fusion quality function and the score-unreliable fusion quality function to obtain the decision quality function of each retrieved remote sensing image. Specifically, the decision quality function is expressed as:

[0132] PLB(A) = m(A1) + m(A3)

[0133] For example, for a query remote sensing image retrieved in a retrieved remote sensing image set containing 1000 other-modal remote sensing images, based on the processing of step S120, an Euclidean distance score containing 1000 values (i.e., obtaining an Euclidean distance score vector with a length of 1000) and another HGR maximum correlation score containing 1000 values (i.e., obtaining an HGR maximum correlation score vector with a length of 1000) are obtained. Then, based on the method of step S130, these two vectors with a length of 1000 are fused. Finally, 1000 fusion quality functions containing 4 score hypotheses are fused. When sorting, the score-reliable fusion quality function and the score-unreliable fusion quality function are fused to obtain 1000 PLB(A) values, and the 1000 PLB(A) values are sorted in descending order to sort the retrieved remote sensing images.

[0134] Figure 2The flow diagram of the cross-modal remote sensing image retrieval method based on the maximum HGR correlation in the embodiment of the present application is shown. First, feature extraction is performed on the query remote sensing image to obtain the query Euclidean distance feature and the query HGR feature, and the Euclidean distance score between the query Euclidean distance feature and each retrieval Euclidean distance feature in the retrieval feature set, and the maximum HGR correlation score between the query HGR feature and each retrieval HGR feature in the retrieval feature set are calculated. Based on the DS evidence theory, the Euclidean distance score and the maximum HGR correlation score are fused and calculated to obtain the fused mass function. The retrieval remote sensing images in the retrieval remote sensing image set are sorted in descending order of the fused mass function to obtain the final retrieval result.

[0135] In the embodiment of the present application, the cross-modal remote sensing image retrieval method based on the maximum HGR correlation is implemented by a pre-trained retrieval model. Specifically, the remote sensing images in the retrieval remote sensing image set are input into the corresponding convolutional neural network, and the corresponding convolutional neural network is selected for feature extraction. Specifically, the AlexNet structure is used as the feature extraction network for the remote sensing image, and the feature output layer and the label output layer are represented by fully connected layers. The lengths of the finally output Euclidean distance feature and the maximum HGR correlation feature are both 512, and the length of the label output layer is 10, obtaining the retrieval feature set. Then, the query remote sensing image is input into the convolutional neural network of the corresponding modality, and the second-to-last layer of the network is the obtained Euclidean distance feature and the maximum HGR correlation feature. Furthermore, the Euclidean distance score between the query Euclidean distance feature and each retrieval Euclidean distance feature in the retrieval feature set, and the maximum HGR correlation score between the query HGR feature and each retrieval HGR feature in the retrieval feature set are calculated, and the fused mass function is calculated based on the DS evidence theory, and sorted in descending order of the fused mass function to obtain the final retrieval result.

[0136] In an alternative embodiment, the retrieval model is trained in the following manner, specifically including steps B1 to B4:

[0137] Step B1: Construct a training data set, which includes: paired remote sensing images composed of first-modal remote sensing images and second-modal remote sensing images, and the class labels of each pair of remote sensing images.

[0138] Among them, the first-modal remote sensing image and the second-modal remote sensing image respectively represent two different modalities of remote sensing images. For example, the first-modal remote sensing image is an optical remote sensing image, and the second-modal remote sensing image is a SAR remote sensing image. The remote sensing images in the training data set exist in pairs, and the paired remote sensing images belong to the same category. Specifically, the training data set can be expressed as: Among them, the first-modal remote sensing image set is expressed as The second-modal remote sensing image set is denoted as Each pair of the first-modal remote sensing image and the second-modal remote sensing image corresponds to a class label, which is denoted as y i =[y i1 ,…,y ic ∈R c .

[0139] Step B2: Input the paired remote sensing images into a retrieval model for processing to obtain the Euclidean distance feature and the HGR feature corresponding to each modal remote sensing image, and process the Euclidean distance feature and the HGR feature to obtain a prediction feature.

[0140] In the embodiment of the present application, for remote sensing images of different modalities, training is performed in a manner of mutual retrieval between the first-modal remote sensing image and the second-modal remote sensing image. The paired remote sensing images in the training dataset are input into a retrieval model for processing. Specifically, the deep neural network in the retrieval model respectively extracts the high-order features (i.e., the Euclidean distance feature and the HGR feature) of the first-modal remote sensing image and the second-modal remote sensing image, and then inputs the high-order features of each modal remote sensing image into a fully connected network for processing to obtain a prediction feature.

[0141] For each pair of modal remote sensing images in the training data, a set of features optimized based on the Euclidean distance (Euclidean distance feature) will be obtained, that is, the Euclidean distance feature f E (v i ) of the first-modal remote sensing image and the Euclidean distance feature g E (s i ) of the second-modal remote sensing image; a set of features optimized based on the maximum correlation of HGR (i.e., the HGR feature), that is, the HGR feature f H (v i ) of the first-modal remote sensing image and the HGR feature g H (s i ) of the second-modal remote sensing image. To accelerate the training speed, the backbone neural network of the retrieval model can select a model pre-trained on a large-scale dataset, and at the same time, the fully connected network for outputting label features adopts a weight sharing strategy to obtain a more accurate retrieval result.

[0142] Step B3: Calculate the Euclidean distance loss according to the Euclidean distance feature, calculate the HGR maximum correlation loss according to the HGR feature, and calculate the prediction loss according to the prediction feature and the corresponding class label.

[0143] In the embodiment of the present application, the loss function for training the retrieval model consists of the Euclidean distance loss, the HGR maximum correlation loss, and the prediction loss. Specifically:

[0144] (1) The Euclidean distance loss is used to measure the differences between the Euclidean distance features corresponding to remote sensing images of different modalities. The Euclidean distance loss includes: similarity loss and ranking loss. Among them, the similarity loss directly measures the similarity between the Euclidean distance features of paired data, and optimizes the minimization of the distance between the Euclidean distance features of paired data in the Euclidean space. The similarity loss J I is expressed as:

[0145]

[0146] where N represents the total amount of a batch of training data pairs, v i , s i respectively represent the first-modal remote sensing image and the second-modal remote sensing image in the paired remote sensing images, and f E (·), g E (·) respectively represent the Euclidean distance features of the first-modal remote sensing image and the second-modal remote sensing image in the paired remote sensing images.

[0147] The ranking loss is a commonly used loss function. By performing positive sampling and negative sampling on a batch of training data, positive samples and negative samples are obtained to train a deep neural network to learn the similarity or distance metric between images. Among them, the positive sample of a remote sensing image refers to another-modal remote sensing image collected in the same batch of training data that belongs to the same type of area as this remote sensing image, and the negative sample of a remote sensing image refers to another-modal remote sensing image collected in the same batch of training data that does not belong to the same type of area as this remote sensing image. For example, taking optical remote sensing images and SAR remote sensing images as an example, for the optical image corresponding to the farmland area, the SAR remote sensing image of the farmland area is a positive sample, and the SAR remote sensing image of the highway area is a negative sample.

[0148] Since the remote sensing image retrieval task is essentially a ranking problem, the training process of the Euclidean distance features is optimized by directly learning the ranking relationship of samples through the ranking loss. Specifically, the ranking loss aims to minimize the distance between the query remote sensing image and the relevant remote sensing images in the retrieval remote sensing image set, and at the same time maximize the distance between the query remote sensing image and the irrelevant remote sensing images. Specifically, the ranking loss is respectively composed of two-modal triplet losses. The ranking loss J R is expressed as:

[0149]

[0150] where d(·,·) represents the Euclidean distance, respectively represent the positive samples obtained by sampling the first-modal remote sensing image and the second-modal remote sensing image in this batch of training data, respectively represent negative samples obtained by sampling the first - modality remote - sensing image and the second - modality remote - sensing image in this batch of training data, and margin represents the optimization margin in the triplet loss.

[0151] Finally, the Euclidean distance loss function J EUC is expressed as:

[0152] J EUC = J I + J R

[0153] (2) The HGR maximum - correlation loss is used to maximize the HGR - feature correlation of remote - sensing images of different modalities. The HGR maximum - correlation loss includes: correlation loss and correlation - ranking loss. The HGR maximum - correlation is a tool for detecting complex, non - linear correlations between variables, aiming to obtain a set of features carrying the maximum amount of relevant information. For random variables X, Y ∈ R K , solving the HGR maximum - correlation can be described as the following optimization problem:

[0154]

[0155] where cov(·) represents the covariance matrix, I represents the identity matrix, and E(·) represents the mathematical expectation. represents the maximum - correlation transformation. Due to the orthogonality constraint of the HGR maximum - correlation, directly solving the above problem requires matrix factorization. In the case of high - dimensional continuous data, the calculation of correlation is complex and there is a problem of numerical instability. To solve this problem, the embodiment of this application adopts an approximation of the HGR maximum - correlation, soft - HGR (HGR s ) as the optimization objective, and the specific expression is:

[0156]

[0157] where tr(·) represents the trace of the matrix.

[0158] Therefore, the correlation loss is used to directly measure the correlation degree of the HGR features of remote - sensing images of different modalities. The correlation loss J C is expressed as:

[0159]

[0160] where tr(·) represents the trace of the matrix, cov(·) represents the covariance matrix, v i , s i respectively represent the first - modality remote - sensing image and the second - modality remote - sensing image in the paired remote - sensing images, and f H (·), g H(·) represent the HGR features of the first-modal remote sensing image and the second-modal remote sensing image in the paired remote sensing images respectively.

[0161] The correlation ranking loss is for each pair of remote sensing images. Positive sampling and negative sampling are performed in the training data of the same batch, so that the correlation between each pair of remote sensing images and the positive samples obtained by sampling in the remote sensing images of the other modality is as large as possible, and the correlation with the negative samples is as small as possible, so as to learn the ranking relationship of the correlation between remote sensing images of different modalities. The correlation ranking loss J CR is expressed as:

[0162]

[0163] where v i , s i represent the first-modal remote sensing image and the second-modal remote sensing image in the paired remote sensing images respectively. represent the positive samples obtained by sampling the first-modal remote sensing image and the second-modal remote sensing image in the training data of this batch respectively. represent the negative samples obtained by sampling the first-modal remote sensing image and the second-modal remote sensing image in the training data of this batch respectively.

[0164] Finally, the expression of the HGR maximum correlation loss is:

[0165] J HGR = J C + J CR

[0166] (3) The prediction loss can be used to train the retrieval model supervised by making full use of the label information. The prediction loss is calculated using the mean squared error loss. The prediction loss is expressed as:

[0167]

[0168] where v i , s i represent the first-modal remote sensing image and the second-modal remote sensing image in the paired remote sensing images respectively. f E (·), g E (·) represent the Euclidean distance features of the first-modal remote sensing image and the second-modal remote sensing image in the paired remote sensing images respectively. f H (·), g H (·) represent the HGR features of the first-modal remote sensing image and the second-modal remote sensing image in the paired remote sensing images respectively. y i represents the class label of the paired remote sensing images.

[0169] Therefore, the overall loss function in the training process of the retrieval model is expressed as:

[0170] J = J P + αJ EUC + βJ HGR

[0171] Wherein, α is a hyperparameter for controlling the proportion of the Euclidean distance loss, and β is a hyperparameter for controlling the proportion of the maximum correlation loss of HGR.

[0172] Step B4: Update the parameters of the retrieval model based on the Euclidean distance loss, the maximum correlation loss of HGR, and the prediction loss. After meeting the training end condition, a trained retrieval model is obtained.

[0173] Specifically, according to the Euclidean distance loss, the maximum correlation loss of HGR, and the prediction loss, appropriate step sizes and hyperparameters are selected to train the model, and the Adam optimizer is used to optimize the neural network until convergence. For example, the step size is set to 10 -4 , and the hyperparameters α and β in the training stage are both set to 10 -2 for training.

[0174] In the embodiments of the present application, the problem of information redundancy is solved by jointly learning two features under the Euclidean distance constraint and the maximum correlation constraint of HGR. The Euclidean distance constraint and the maximum correlation constraint of HGR can better measure the relationship between different modalities by tools, and thus the obtained retrieval model has better reliability.

[0175] Figure 3 Taking optical remote sensing images and SAR remote sensing images as examples, the training process of the retrieval model in the embodiments of the present application is illustrated. The optical remote sensing images are input into the convolutional neural network of the optical remote sensing images of the retrieval model for processing to obtain the Euclidean distance features and HGR features corresponding to the optical remote sensing images. The SAR remote sensing images are input into the convolutional neural network of the SAR remote sensing images of the retrieval model for processing to obtain the Euclidean distance features and HGR features corresponding to the SAR remote sensing images; the Euclidean distance features and HGR features corresponding to the optical remote sensing images and the Euclidean distance features and HGR features corresponding to the SAR remote sensing images are respectively processed to obtain prediction features; by performing positive sampling and negative sampling in a batch of training data, positive samples and negative samples are obtained, the Euclidean distance loss is calculated based on the positive samples, negative samples, the Euclidean distance features of the optical remote sensing images, and the Euclidean distance features of the SAR remote sensing images, the maximum correlation loss of HGR is calculated based on the positive samples, negative samples, the HGR features of the optical remote sensing images, and the HGR features of the SAR remote sensing images, and the prediction loss is calculated based on the prediction features and the class labels of the remote sensing images; finally, the network parameters of the retrieval model are updated based on the Euclidean distance loss, the maximum correlation loss of HGR, and the prediction loss.

[0176] Table 1 shows the comparison of the retrieval performance between the method proposed in this embodiment and four current mainstream cross-modal retrieval methods (CCA, Corr-AE, ACMR, CMIR-Net).

[0177] Table 1 Comparison of the retrieval performance between the method proposed in this embodiment and existing methods

[0178]

[0179] The average precision of the whole class, a mainstream evaluation index of the retrieval model, is used to compare the retrieval accuracy of different methods. It can be seen from Table 1 that the method based on the maximum correlation enhancement of HGR proposed in this embodiment significantly improves the retrieval accuracy. Therefore, this method has high application value in practice.

[0180] In the embodiment of this application, the Euclidean distance correlation relationship and the maximum correlation of HGR between remote sensing images of different modalities are considered simultaneously to better eliminate the differences caused by modalities and solve the problem of information redundancy. The Euclidean distance features and HGR features between remote sensing images of different modalities are combined to enhance the reliability of the retrieval results. At the same time, the final retrieval result is determined based on the fusion quality function calculated by the DS evidence theory, rather than simply sorting the Euclidean distance or cosine similarity, which further ensures the reliability of the final detection result.

[0181] Refer to Figure 4 As shown, a schematic structural diagram of a cross-modal remote sensing image retrieval device based on the maximum correlation of HGR in the embodiment of this application is shown. As Figure 4 shown, the device includes:

[0182] A feature extraction module 41, configured to extract features from a query remote sensing image to obtain a query Euclidean distance feature and a query HGR feature;

[0183] A score calculation module 42, configured to calculate the Euclidean distance score between the query Euclidean distance feature and each retrieval Euclidean distance feature in the retrieval feature set, and calculate the maximum correlation score of HGR between the query HGR feature and each retrieval HGR feature in the retrieval feature set. The retrieval feature set is a retrieval feature set obtained by pre-extracting features from a retrieval remote sensing image set. The retrieval remote sensing images in the retrieval remote sensing image set and the query remote sensing image are remote sensing images of different modalities;

[0184] A score fusion module 43, configured to fuse and calculate the Euclidean distance score and the maximum correlation score of HGR based on the DS evidence theory to obtain a fusion quality function, where the fusion quality function characterizes the relevance between the query remote sensing image and the retrieval remote sensing image;

[0185] A result sorting module 44, configured to sort in descending order according to the fusion quality function to obtain a final retrieval result.

[0186] In an alternative embodiment, the score fusion module includes:

[0187] A hypothesis definition module, configured to define score hypothesis cases, where the score hypothesis cases include: reliable score, unreliable score, uncertain score reliability, and empty set;

[0188] A score calculation module, configured to define a quality function of the Euclidean distance score in different score hypothesis cases and a quality function of the HGR maximum correlation score in different score hypothesis cases according to the score hypothesis cases;

[0189] A fusion calculation module, configured to fuse the Euclidean distance quality function and the HGR maximum correlation quality function with an intersection in the score hypothesis cases according to the DS evidence theory fusion rule to obtain a fusion quality function.

[0190] In an alternative embodiment, the fusion calculation module includes:

[0191] A first fusion calculation sub-module, configured to fuse the Euclidean distance quality function with reliable score, the HGR maximum correlation quality function with reliable score, the Euclidean distance quality function with uncertain score reliability, and the HGR maximum correlation quality function with uncertain score reliability for the hypothesis case of reliable score to obtain a reliable score fusion quality function;

[0192] A second fusion calculation sub-module, configured to fuse the Euclidean distance quality function with unreliable score, the HGR maximum correlation quality function with unreliable score, the Euclidean distance quality function with uncertain score reliability, and the HGR maximum correlation quality function with uncertain score reliability for the hypothesis case of unreliable score to obtain an unreliable score fusion quality function;

[0193] A third fusion calculation sub-module, configured to fuse the Euclidean distance quality function with uncertain score reliability and the HGR maximum correlation quality function with uncertain score reliability for the hypothesis case of uncertain score reliability to obtain an uncertain score reliability fusion quality function;

[0194] A fourth fusion calculation sub-module, configured to set the score empty set fusion quality function to 0 for the hypothesis case of an empty set score.

[0195] In an alternative embodiment, the result sorting module includes:

[0196] A decision function module, which is used to fuse the score reliable fusion quality function and the score unreliable fusion quality function based on the maximization plausibility criterion in the DS evidence theory to obtain the decision quality function of each retrieved remote sensing image;

[0197] A function sorting module, which is used to sort the retrieved remote sensing images in the retrieved remote sensing image set in descending order according to the decision quality function to obtain the final retrieval result.

[0198] In an alternative embodiment, the cross-modal remote sensing image retrieval method based on HGR maximum correlation is implemented by a pre-trained retrieval model. The device further includes a model training module, and the model training module is used to train the retrieval model. The model training module includes:

[0199] A data set module, which is used to construct a training data set. The training data set includes: paired remote sensing images composed of first-modal remote sensing images and second-modal remote sensing images, and class labels of each pair of remote sensing images;

[0200] A feature prediction module, which is used to input the paired remote sensing images into the retrieval model for processing to obtain the Euclidean distance feature and HGR feature corresponding to each modal remote sensing image, and process the Euclidean distance feature and the HGR feature to obtain a predicted feature;

[0201] A loss calculation module, which is used to calculate the Euclidean distance loss according to the Euclidean distance feature, calculate the HGR maximum correlation loss according to the HGR feature, and calculate the prediction loss according to the predicted feature and the corresponding class label;

[0202] A parameter optimization module, which is used to update the parameters of the retrieval model based on the Euclidean distance loss, the HGR maximum correlation loss, and the prediction loss. After meeting the training end condition, a trained retrieval model is obtained.

[0203] In an alternative embodiment, the score calculation module includes:

[0204] A distance module, which is used to calculate the Euclidean distance between the query Euclidean distance feature and each retrieval Euclidean distance feature in the retrieval feature set;

[0205] A distance score module, which is used to normalize the Euclidean distance between the query Euclidean distance feature and each retrieval Euclidean distance feature in the retrieval feature set according to the maximum and minimum values of the Euclidean distance to obtain a Euclidean distance score.

[0206] In an alternative embodiment, the score calculation module includes:

[0207] A correlation module for calculating a correlation coefficient between the query HGR feature and each retrieved HGR feature in the retrieved feature set;

[0208] A correlation score module for normalizing the correlation coefficient between the query HGR feature and each retrieved HGR feature in the retrieved feature set according to the maximum and minimum values of the correlation coefficient to obtain the maximum HGR correlation score.

[0209] Each embodiment in this specification is described in a progressive manner. Each embodiment focuses on the differences from other embodiments. For the same or similar parts among the embodiments, reference can be made to each other.

[0210] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods and devices according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, and the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal devices generate a device for implementing the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0211] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing terminal device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device, and the instruction device implements the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0212] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device, so that a series of operation steps are executed on the computer or other programmable terminal device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable terminal device provide steps for implementing the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0213] Although the preferred embodiments of the embodiments of the present application have been described, those skilled in the art can make additional changes and modifications once they learn the basic creative concept. Therefore, the appended claims are intended to be construed to include the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0214] Finally, it should also be noted that in this text, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or terminal device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or terminal device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or terminal device comprising the element.

[0215] The above has introduced in detail a cross-modal remote sensing image retrieval method and device based on HGR maximum correlation provided by the present application. Specific examples are used in this text to elaborate on the principle and implementation manner of the present application. The description of the above embodiments is only used to help understand the method and its core idea of the present application; at the same time, for those of ordinary skill in the art, according to the idea of the present application, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation to the present application.

Claims

1. A cross-modal remote sensing image retrieval method based on the maximum correlation of HGR, characterized in that The method includes: Performing feature extraction on the query remote sensing image to obtain a query Euclidean distance feature and a query HGR feature; Calculating the Euclidean distance score between the query Euclidean distance feature and each retrieval Euclidean distance feature in the retrieval feature set, and calculating the HGR maximum correlation score between the query HGR feature and each retrieval HGR feature in the retrieval feature set. The retrieval feature set is a retrieval feature set obtained by pre-performing feature extraction on a retrieval remote sensing image set. The retrieval remote sensing images in the retrieval remote sensing image set and the query remote sensing image are remote sensing images of different modalities; Based on the DS evidence theory, fusing and calculating the Euclidean distance score and the HGR maximum correlation score to obtain a fused mass function, where the fused mass function characterizes the relevance between the query remote sensing image and the retrieval remote sensing image; Sorting the retrieval remote sensing images in the retrieval remote sensing image set in descending order according to the fused mass function to obtain a final retrieval result; The fusing and calculating the Euclidean distance score and the HGR maximum correlation score based on the DS evidence theory to obtain a fused mass function includes: Defining score hypothesis cases, where the score hypothesis cases include: score reliable, score unreliable, score reliability uncertain, and empty set; According to the score hypothesis cases, defining the mass function of the Euclidean distance score in different score hypothesis cases and the mass function of the HGR maximum correlation score in different score hypothesis cases; According to the DS evidence theory fusion rule, fusing the Euclidean distance mass function and the HGR maximum correlation mass function with an intersection in the score hypothesis cases to obtain a fused mass function.

2. The method according to claim 1, characterized in that, The fusing the Euclidean distance mass function and the HGR maximum correlation mass function with an intersection in the score hypothesis cases according to the DS evidence theory fusion rule includes: For the hypothesis case of score reliable, fusing the Euclidean distance mass function of score reliable, the HGR maximum correlation mass function of score reliable, the Euclidean distance mass function of score reliability uncertain, and the HGR maximum correlation mass function of score reliability uncertain to obtain a score reliable fused mass function; For the hypothesis case of score unreliable, fusing the Euclidean distance mass function of score unreliable, the HGR maximum correlation mass function of score unreliable, the Euclidean distance mass function of score reliability uncertain, and the HGR maximum correlation mass function of score reliability uncertain to obtain a score unreliable fused mass function; For the hypothesis case of score reliability uncertain, fusing the Euclidean distance mass function of score reliability uncertain and the HGR maximum correlation mass function of score reliability uncertain to obtain a score reliability uncertain fused mass function; For the hypothesis case of score being an empty set, setting the score empty set fused mass function equal to 0.

3. The method according to claim 2, characterized in that, Sorting the retrieval remote sensing images in the retrieval remote sensing image set in descending order according to the fused mass function to obtain a final retrieval result, including: Based on the maximization plausibility criterion in the DS evidence theory, fuse the score reliable fusion quality function and the score unreliable fusion quality function to obtain the decision quality function of each retrieved remote sensing image; Sort the retrieved remote sensing images in the retrieved remote sensing image set in descending order according to the decision quality function to obtain the final retrieval result.

4. The method according to any one of claims 1-3, characterized in that The cross-modal remote sensing image retrieval method based on HGR maximum correlation is implemented through a pre-trained retrieval model, and the retrieval model is trained in the following manner: Construct a training data set, which includes: paired remote sensing images composed of first-modal remote sensing images and second-modal remote sensing images, and class labels of each pair of remote sensing images; Input the paired remote sensing images into the retrieval model for processing to obtain the Euclidean distance feature and HGR feature corresponding to each modal remote sensing image, and process the Euclidean distance feature and the HGR feature to obtain a prediction feature; Calculate the Euclidean distance loss according to the Euclidean distance feature, calculate the HGR maximum correlation loss according to the HGR feature, and calculate the prediction loss according to the prediction feature and the corresponding class label; Update the parameters of the retrieval model based on the Euclidean distance loss, the HGR maximum correlation loss, and the prediction loss. After meeting the training end condition, obtain the trained retrieval model.

5. The method according to claim 4, characterized in that, The Euclidean distance loss is used to measure the difference between the Euclidean distance features corresponding to different modal remote sensing images, and the Euclidean distance loss includes: similarity loss and ranking loss; The similarity loss is expressed as: Among them, N represents the total amount of a batch of training data pairs, v i , s i respectively represent the first-modal remote sensing image and the second-modal remote sensing image in the paired remote sensing images, f E (·), g E (·) respectively represent the Euclidean distance features of the first-modal remote sensing image and the second-modal remote sensing image in the paired remote sensing images; The sorting loss is expressed as: where d(·, ·) represents the Euclidean distance, respectively represent the positive samples sampled from the first-modal remote sensing image and the second-modal remote sensing image in this batch of training data, respectively represent the negative samples sampled from the first-modal remote sensing image and the second-modal remote sensing image in this batch of training data, and margin represents the optimization margin in the triplet loss.

6. The method according to claim 4, characterized in that, The HGR maximum correlation loss is used to maximize the correlation of the HGR features of different modal remote sensing images, and the HGR maximum correlation loss includes: correlation loss and correlation ranking loss; The correlation loss is expressed as: Among them, tr(·) represents the trace of a matrix, cov(·) represents the covariance matrix, and v i , s i respectively represent the first-modal remote sensing image and the second-modal remote sensing image in the paired remote sensing images, and f H (·), g H (·) respectively represent the HGR features of the first-modal remote sensing image and the second-modal remote sensing image in the paired remote sensing images; The relevance ranking loss is expressed as: Among them, v i , s i respectively represent the first-modal remote sensing image and the second-modal remote sensing image in the paired remote sensing images, respectively represent the positive samples sampled from the first-modal remote sensing image and the second-modal remote sensing image in this batch of training data, respectively represent the negative samples sampled from the first-modal remote sensing image and the second-modal remote sensing image in this batch of training data.

7. The method according to claim 4, characterized in that, The prediction loss is calculated using the mean squared error loss, and the prediction loss is expressed as: Among them, v i and s i respectively represent the first-modal remote sensing image and the second-modal remote sensing image in the paired remote sensing images, f E (·) and g E (·) respectively represent the Euclidean distance features of the first-modal remote sensing image and the second-modal remote sensing image in the paired remote sensing images, f H (·) and g H (·) respectively represent the HGR features of the first-modal remote sensing image and the second-modal remote sensing image in the paired remote sensing images, y i represents the class label of the paired remote sensing images.

8. The method according to claim 1, characterized in that The calculation of the Euclidean distance score between the query Euclidean distance feature and each retrieval Euclidean distance feature in the retrieval feature set includes: Calculate the Euclidean distance between the query Euclidean distance feature and each retrieval Euclidean distance feature in the retrieval feature set; Normalize the Euclidean distance between the query Euclidean distance feature and each retrieval Euclidean distance feature in the retrieval feature set according to the maximum and minimum values of the Euclidean distance to obtain the Euclidean distance score.

9. The method according to claim 1, characterized in that The calculation of the HGR maximum correlation score between the query HGR feature and each retrieval HGR feature in the retrieval feature set includes: Calculate the HGR maximum correlation coefficient between the query HGR feature and each retrieval HGR feature in the retrieval feature set; Normalize the HGR maximum correlation coefficient between the query HGR feature and each retrieval HGR feature in the retrieval feature set according to the maximum and minimum values of the correlation coefficient to obtain the HGR maximum correlation score.

10. A cross-modal remote sensing image retrieval device based on the maximum correlation of HGR, characterized in that, The device includes: A feature extraction module for extracting features from the query remote sensing image to obtain a query Euclidean distance feature and a query HGR feature; A score calculation module, which is used to calculate the Euclidean distance scores between the query Euclidean distance feature and each retrieval Euclidean distance feature in the retrieval feature set, and calculate the HGR maximum correlation scores between the query HGR feature and each retrieval HGR feature in the retrieval feature set. The retrieval feature set is a retrieval feature set obtained by pre-extracting features from a retrieval remote sensing image set, and the retrieval remote sensing images in the retrieval remote sensing image set and the query remote sensing image are remote sensing images of different modalities; A score fusion module, which is used to fuse and calculate the Euclidean distance score and the HGR maximum correlation score based on the DS evidence theory to obtain a fused mass function, and the fused mass function characterizes the relevance between the query remote sensing image and the retrieval remote sensing image; includes: defining score hypothesis cases, where the score hypothesis cases include: reliable score, unreliable score, uncertain score reliability, and empty set; according to the score hypothesis cases, defining the mass functions of the Euclidean distance score in different score hypothesis cases and the mass functions of the HGR maximum correlation score in different score hypothesis cases; fusing the Euclidean distance mass function and the HGR maximum correlation mass function with intersections in the score hypothesis cases according to the DS evidence theory fusion rule to obtain a fused mass function; A result sorting module, which is used to sort the retrieval remote sensing images in the retrieval remote sensing image set in descending order according to the fused mass function to obtain a final retrieval result.

Images