A multi-modal hash retrieval method for massive remote sensing target fast association

Abstract

The application belongs to the technical field of artificial intelligence. The application provides a multi-modal hash retrieval method for massive remote sensing target fast association. The disclosure embodiment introduces an extensible large-scale unsupervised similarity graph generation strategy based on anchor points, and utilizes the Laplace regularization term of the similarity graph to limit the hash coding to contain more semantic information. Without sacrificing the discrete constraint, high-quality coding is directly learned, solving the problem of coding precision loss caused by relaxation processing. By learning fine-grained multi-modal feature information, the association between cross-modal data is mined, effectively improving the accuracy of multi-modal remote sensing target retrieval in theory and practice.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to a multimodal hash retrieval method for rapid association of massive remote sensing targets. Background Technology

[0002] Multimodal retrieval is an important research direction in computer vision and information retrieval. It solves the problem of how to formulate a query in one data modality (such as text) and retrieve relevant content in another different modality (such as images, videos, or audio). In the context of remote sensing technology, multimodal data refers to multiple data types acquired through different sensors. Although these data vary in form, they are often related and describe the same event or object. This technology can not only improve the efficiency of comprehensive data utilization but also significantly enhance the efficiency and accuracy of information retrieval. Multimodal retrieval technology has important application value in fields such as military reconnaissance, environmental monitoring, and disaster early warning, and it also promotes modern urban management and development such as intelligent transportation and urban planning.

[0003] However, the implementation of multimodal retrieval technology still faces multiple challenges, including how to effectively extract and transform features between modalities, how to build models that can understand and match complex relationships between different modalities, and how to handle and optimize the retrieval efficiency of massive multimodal datasets. How to quickly and accurately retrieve the required information from multimodal data has become an urgent problem to be solved.

[0004] Hash encoding techniques are commonly used in fast association retrieval to achieve low storage costs and high retrieval efficiency. Common methods mainly consist of two modules: feature learning and hash encoding learning. The input is usually a pair of remote sensing image-text pairs, where the text is multiple sentences describing the remote sensing image. The image and text are encoded using image encoders and text encoders respectively to obtain corresponding modal features. Then, a similarity map is constructed based on the feature or label data. Next, a loss function is designed based on the similarity map and model characteristics to guide the hash function in generating the desired hash code. Finally, the hash code is optimized. Since the feature spaces of different modal samples are different, it is impossible to directly calculate the association between cross-modal samples. A common unsupervised approach is to first calculate the similarity between samples in a single modal space, and construct a graph based on a similarity threshold or the top-k nearest neighbors. Then, the single-modal graph is weighted to obtain the overall similarity map. In hash encoding optimization, continuous encoding is often learned first, and then transformed into a discrete form.

[0005] The above approach has two main drawbacks: First, the similarity graph does not take into account the association between cross-modal samples, lacks finer-grained semantic information, and often does not consider the imbalance of the number of samples of different modalities in the dataset; Second, hash coding optimization first learns continuous codes, and the gap between the continuous codes in the learning process and the final expected codes is large, which will lead to a loss of accuracy.

[0006] Therefore, it is necessary to improve one or more of the problems existing in the above-mentioned related technical solutions.

[0007] It should be noted that this section is intended to provide background or context for the technical solutions of this disclosure as set forth in the claims. The description herein does not constitute an admission that it is prior art simply because it is included in this section. Summary of the Invention

[0008] The purpose of this disclosure is to provide a multimodal hash retrieval method for rapid association of massive remote sensing targets, thereby overcoming, to at least to some extent, one or more problems caused by the limitations and defects of related technologies.

[0009] According to embodiments of this disclosure, a multimodal hash retrieval method for rapid association of massive remote sensing targets is provided, the method comprising:

[0010] Acquire multimodal remote sensing sample data; wherein, the multimodal remote sensing sample data includes remote sensing image data and its corresponding text information;

[0011] Construct a multimodal hash retrieval model; wherein the multimodal hash retrieval model includes at least a unimodal encoder, and the unimodal encoder includes an image encoder and a text encoder;

[0012] The remote sensing image data and the text information are preprocessed, and the image encoder and the text encoder are used to extract features from the preprocessed remote sensing image data and the text information to obtain image features and text features.

[0013] Based on the image features and the text features, a large-scale unsupervised similarity graph generation strategy based on anchor points is used to obtain a joint cross-modal graph of the remote sensing image data and the text information.

[0014] Based on the image features, the text features, and the joint cross-modal graph, a discrete hash code is generated using the factorization method;

[0015] The discrete hash code is optimized using the ADMM optimization algorithm to obtain the optimal model parameters, and a trained multimodal hash retrieval model is obtained based on the optimal model parameters.

[0016] The data to be retrieved is input into the trained multimodal hash retrieval model to obtain the retrieval results.

[0017] Further, the step of preprocessing the remote sensing image data and the text information, and extracting features from the preprocessed remote sensing image data and the text information using the image encoder and the text encoder respectively to obtain image features and text features, includes:

[0018] Image enhancement, denoising, and alignment are performed on the remote sensing image data to ensure the consistency and quality of the remote sensing image data;

[0019] The image encoder is used to extract features from the preprocessed remote sensing image data to obtain image features;

[0020] The text encoder is used to extract features from the preprocessed text information to obtain the text features.

[0021] Further, the step of obtaining a joint cross-modal graph of the remote sensing image data and the text information by utilizing an anchor-based large-scale unsupervised similarity graph generation strategy based on the image features and the text features includes:

[0022] From the image features and the text features Select For anchor points As an anchor set, based on n sample pairs and Anchor point pair To obtain the relationship between samples and anchor points within the image and text modalities. and the mapping matrix that maps images and text to a public space. ;in, v Represents mode, v =1 indicates an image. v =2 represents text;

[0023] Based on the mapping matrix within the image and text modalities The relationship diagram between cross-modal anchor points is obtained. , For the dimension of public space;

[0024] Based on the relationship diagram between the image and text modal samples and anchor points. The relationship graph between samples within the image and text modalities is obtained. ;

[0025] Based on the relationship diagram between the image and text cross-modal anchor points QThe relationship between the samples and anchor points within the image and text modalities. The relationship graph between cross-modal samples was obtained. ;

[0026] Based on the relationship diagram between the samples within the image and text modalities The relationship between the cross-modal samples and the graph The joint cross-modal graph is obtained; wherein, the t-step transition matrix is ​​used. To describe the relationships between samples that are reachable in multiple steps.

[0027] Furthermore, the expression for the sample-anchor relationship graph within the modality is:

[0028]

[0029] The expression for the sample-anchor relationship diagram within the modality can be simplified to:

[0030]

[0031] in, For image features or text features, For anchor points corresponding to modal features, As the first hyperparameter, This is the second hyperparameter. For the first The image feature or text feature and the first Similarity relationships between text features or image features Indicates the number of anchor points. It is a picture The row degree matrix, It is a picture The column-degree matrix, yes The row vector, The optimal solution is the intermediate term. The matrix consisting of eigenvectors corresponding to the smallest eigenvalues;

[0032] The expression for the relationship between the intramodal sample and the anchor point is:

[0033]

[0034] in, This is a graph showing the relationship between samples and anchor points within image and text modalities. For the first i Image features or text features For the corresponding mode, the first j Anchor point features, This is the third hyperparameter of the corresponding mode;

[0035] The expression for the relationship graph between the samples within the image and text modalities is as follows:

[0036]

[0037] in, This is a graph showing the relationships between samples within an image modality. This is a graph showing the relationships between samples within a text modality.

[0038] The expression for the relationship diagram between the cross-modal anchor points is:

[0039]

[0040] in, For image modal anchor points, For the anchor point of the text modality, For the first The first image anchor point and the first Similarity relationships between text anchors This is the fourth hyperparameter;

[0041] The expression for the relationship graph between the cross-modal samples is:

[0042]

[0043] The expression for the joint cross-modal graph is:

[0044]

[0045] Using the t-step transition matrix The expression describing the relationship between samples that can be reached in multiple steps is as follows:

[0046]

[0047] in, for of Power of 1 To establish the relationships between image samples after multiple steps of reachability. To establish the relationships between text samples after multiple steps of accessibility. It is cross-modal similarity information obtained through an unsupervised similarity generation strategy. for The transpose of .

[0048] Further, the step of generating discrete hash codes using a factorization method based on the image features, the text features, and the joint cross-modal graph includes:

[0049] Construct the hash coding loss function for the hash coding factorization method;

[0050] The image features and text features are uniformly mapped to a common Hamming space to generate a unified initial hash code;

[0051] Based on the hash encoding loss function, the Laplace matrix of the joint cross-modal graph is introduced into the initial hash encoding to obtain the discrete hash encoding.

[0052] Furthermore, the expression for the hash encoding loss function is:

[0053]

[0054] in, This is the fifth hyperparameter. This is the sixth hyperparameter. This is the seventh hyperparameter. This is the eighth hyperparameter. and These represent the features obtained from the original image and text after being encoded, respectively. and transpose, d 1 represents the dimension of image features. d 2 represents the dimension of text features. Hash encoding, and These are the decomposition matrices for image features and text features, respectively. r The length of the hash code. The image mapping matrix represents the mapping from the feature space to the common space. The text mapping matrix representing the feature space to the common space. Represents the Laplace matrix of the joint cross-modal graph. for A diagonal matrix formed by the sum of the elements in each row. This indicates that the square of the corresponding Frobenius norm is calculated for each matrix in parentheses.

[0055] Further, the step of optimizing the discrete hash code using the ADMM optimization algorithm to obtain optimal model parameters, and obtaining the trained single-modal encoder based on the optimal model parameters, includes:

[0056] The matrices that need to be solved by alternating optimization include , , , and ;

[0057] The discrete constraints in the discrete hash code are equivalently transformed into the intersection of two continuous spaces, and the ADMM optimization algorithm is used to transform it into a problem of the intersection of two continuous spaces.

[0058] Define an augmented Lagrangian function, decompose the problem into multiple subproblems, and iteratively solve the subproblems until convergence to obtain the optimal model parameters.

[0059] The multimodal hash retrieval model is updated using the optimal model parameters to obtain the trained multimodal hash retrieval model.

[0060] Furthermore, the problem is:

[0061]

[0062] in, For the first Image features, For the first Text features, For the first A hash code, It is an r-dimensional space similar to a cube. It is an r-dimensional sphere;

[0063] The expression for the augmented Lagrange function is:

[0064]

[0065] in, As the first Lagrange multiplier, For the second Lagrange multiplier, It is the ninth hyperparameter. This is the tenth hyperparameter.

[0066] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects:

[0067] In the embodiments of this disclosure, the multimodal hash retrieval method for rapid association of massive remote sensing targets, as described above, introduces a scalable, anchor-based, large-scale unsupervised similarity graph generation strategy and utilizes the Laplacian regularization term of the similarity graph to restrict the encoding to include more semantic information. High-quality encoding is learned directly without sacrificing discrete constraints, solving the problem of encoding accuracy loss caused by relaxation processing. Furthermore, it can learn fine-grained multimodal feature information and mine the associations between cross-modal data, effectively improving the accuracy of multimodal remote sensing target retrieval both theoretically and practically. Attached Figure Description

[0068] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0069] Figure 1 This diagram illustrates the steps of a multimodal hash retrieval method for rapid association of massive remote sensing targets according to an exemplary embodiment of this disclosure.

[0070] Figure 2 This diagram illustrates the model training flowchart in an exemplary embodiment of this disclosure.

[0071] Figure 3 A schematic diagram illustrating a large-scale unsupervised similarity graph generation strategy in an exemplary embodiment of this disclosure is shown.

[0072] Figure 4 This diagram illustrates the hash encoding technique used in an exemplary embodiment of this disclosure.

[0073] Figure 5 A visual schematic diagram illustrating the discrete constraint equivalent transformation in an exemplary embodiment of this disclosure is shown;

[0074] Figure 6 A flowchart illustrating the model retrieval process in an exemplary embodiment of this disclosure is shown. Detailed Implementation

[0075] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0076] Furthermore, the accompanying drawings are merely illustrative diagrams of embodiments of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities.

[0077] This example implementation provides a multimodal hash retrieval method for rapidly associating massive amounts of remote sensing targets. (Reference) Figure 1 As shown, the multimodal hash retrieval method for rapid association of massive remote sensing targets may include steps S101 to S107.

[0078] Step S101: Acquire multimodal remote sensing sample data; wherein, the multimodal remote sensing sample data includes remote sensing image data and its corresponding text information;

[0079] Step S102: Construct a multimodal hash retrieval model; wherein the multimodal hash retrieval model includes at least a single-modal encoder, and the single-modal encoder includes an image encoder and a text encoder;

[0080] Step S103: Preprocess the remote sensing image data and the text information, and use the image encoder and the text encoder to extract features from the preprocessed remote sensing image data and the text information to obtain image features and text features;

[0081] Step S104: Based on the image features and the text features, use a large-scale unsupervised similarity graph generation strategy based on anchor points to obtain a joint cross-modal graph of the remote sensing image data and the text information;

[0082] Step S105: Based on the image features, the text features, and the joint cross-modal graph, generate discrete hash codes using the factorization method;

[0083] Step S106: Optimize the discrete hash code using the ADMM optimization algorithm to obtain the optimal model parameters, and obtain the trained multimodal hash retrieval model based on the optimal model parameters;

[0084] Step S107: Input the data to be retrieved into the trained multimodal hash retrieval model to obtain the retrieval results.

[0085] The multimodal hashing retrieval method described above for rapid association of massive remote sensing targets introduces a scalable, anchor-based, large-scale unsupervised similarity graph generation strategy. It utilizes the Laplacian regularization term of the similarity graph to constrain the encoding to include more semantic information. High-quality encoding is learned directly without sacrificing discrete constraints, thus resolving the accuracy loss caused by relaxation processing. Furthermore, by learning fine-grained multimodal feature information, it mines the correlations between cross-modal data, effectively improving the accuracy of multimodal remote sensing target retrieval both theoretically and practically.

[0086] Below, we will refer to Figures 1 to 6 The steps of the multimodal hash retrieval method for rapid association of massive remote sensing targets described in this example embodiment will be explained in more detail.

[0087] In step S101, multimodal remote sensing data is acquired, including remote sensing images and their corresponding descriptive information. The remote sensing images are acquired through different sensors such as optics and radar, and the descriptive information is provided by experts or an automatic generation system.

[0088] like Figure 2 The diagram shown is a flowchart of the training process for a multimodal retrieval model.

[0089] In step S102, the single-modal encoder includes an image encoder and a text encoder, wherein the image encoder and the text encoder are existing encoders, such as the image encoder using a convolutional neural network and the text encoder using a bag-of-words model, without specific limitations here.

[0090] In step S103, the collected multimodal remote sensing data is preprocessed, including image enhancement, denoising, and alignment, to ensure data consistency and quality. A suitable encoder is selected to convert the image and text data into corresponding embedding vectors.

[0091] In step S104, for the scalable anchor-based large-scale unsupervised similarity graph generation strategy, addressing the issue that common similarity graphs do not adequately consider intra-modal and inter-modal associations, this strategy utilizes anchors to comprehensively consider intra-modal and inter-modal associations as well as multi-step reachability relationships between samples. By analyzing the hyperparameter impact of this strategy and comparing it with other common construction methods, the results show that this strategy can effectively guide the generation of hash codes. Figure 3 The diagram shown illustrates a strategy for generating large-scale unsupervised similarity graphs.

[0092] Specifically, a scalable, anchor-based, large-scale unsupervised similarity map generation strategy is proposed, where the input to the similarity map is remote sensing image features obtained through an image encoder and a text encoder. and related text features Choose from Using image text features as anchors Given a sample pair and anchor point pair (v represents modality, 1 represents image, 2 represents text), the mathematical expression for the sample-anchor relationship graph within a modality is defined as follows:

[0093]

[0094] The above expression can be transformed into the following expression:

[0095]

[0096] Where, mapping matrix Map image and text features into the same space. It is an intramodal sparse graph. Indicates the number of anchor points. It is a picture The row degree matrix, It is a picture The column-degree matrix, yes The row vector. The optimal solution is the intermediate term. A matrix consisting of the eigenvectors corresponding to the smallest eigenvalues.

[0097] calculate The anchor points need to be sorted from smallest to largest according to their Euclidean distance from the samples. Find the nearest neighbors, and then dynamically update the mapping matrix using the learned matrix. The weights of the nearest neighbors are used. Since the image and text modal mapping methods are consistent, this is used here. The unified representation is shown in the following mathematical expression:

[0098]

[0099] Next, we further calculate the relationship graph between samples within the text and image modalities. As shown in the following formula:

[0100]

[0101] Obtain the mapping matrix within the image and text modalities. and Then, the relationship diagram between cross-modal anchor points. The mathematical expression is as follows:

[0102]

[0103] picture and For a sparse graph, only There are three non-zero vectors, each row of which sums to 1, meaning the sum of the similarities between the sample and other samples is 1. Calculate... First, the image and text anchor points need to be mapped to the same space, and then the Euclidean distance needs to be calculated to determine the location. Next nearest neighbor, then directly use the following formula to calculate Element.

[0104]

[0105] The relationship between cross-modal anchor points is obtained. and the relationship between intramodal samples and anchor points and Then, a graph showing the relationships between samples across modalities can be calculated. It is the affinity matrix defined in the bipartite graph between the image and the text, as shown in the following equation:

[0106]

[0107] The joint crossmodal graph consists of three parts, and its expression is defined as follows:

[0108]

[0109] Using the t-step transition matrix To describe the relationships between samples that are reachable in multiple steps, its mathematical expression is defined as follows:

[0110]

[0111] For homogeneous Markov chains, yes t to the power of It can be divided into four parts. and The figures represent similarity information within each modality. This is then used to strengthen the cross-modal correlation through Laplace regularization terms.

[0112] In step S105, for hash code generation, matrix factorization is used to uniformly map image and text features to a common Hamming space to generate a unified and efficient hash code, and a similarity map is introduced. The Laplace regularization term enhances the semantic information of the encoding. The optimization process includes the derivation of the closed-form solution, the application of an alternating update strategy, and the calculation of discrete hash codes. Figure 4 The diagram shown is a schematic of discrete coding optimization techniques.

[0113] Specifically, for hash encoding, the loss function of the factorization method can be defined as follows:

[0114]

[0115] in, and These represent the features obtained from the original image and text after being encoded, respectively. and The transpose of the matrix and Decomposed into public Hamming space , where r represents the length of the hash code. and The matrix representing the mapping from the feature space to the common space. This represents the graph obtained in the previous step. The Laplace matrix, for A diagonal matrix formed by the sum of the elements in each row.

[0116] In step S106, for discrete coding optimization techniques, in order to solve the problem of coding accuracy loss caused by relaxation processing, the ADMM optimization algorithm is introduced while retaining discrete constraints, to directly learn high-quality hash codes, thereby effectively improving the accuracy and efficiency of cross-modal retrieval in theory and practice.

[0117] Specifically, the discrete coding optimization technique employs an alternating optimization method, where only one variable is updated while other variables are kept constant. The matrix to be solved includes... , , , , ,Apart from All matrices except those in the loss function can be directly solved by taking the derivative of the loss function, and The discrete constraints require further processing.

[0118] (1) Update

[0119] First, analyze The update process, The update process is similar. The relevant loss function can be simplified as follows:

[0120]

[0121] By calculating the derivative and setting it to 0, the optimal solution to the problem can be obtained as follows:

[0122]

[0123] The accumulated data is defined as shown in the following formula. At the end of each iteration, the product of the relevant variables is updated for subsequent calculations.

[0124]

[0125]

[0126] Therefore, considering cumulative data It can be expressed as follows:

[0127]

[0128] The optimal solution can be obtained similarly:

[0129]

[0130] The cumulative data is defined as follows:

[0131]

[0132] (2) Update

[0133] Fix other variables and only update or First, let's analyze The update process, The update process is similar. The relevant loss function can be simplified as follows:

[0134]

[0135] By calculating that the derivative is 0, we can obtain The analytical solution is shown in the following equation:

[0136]

[0137] The cumulative product of variables is defined as follows:

[0138]

[0139]

[0140] thus It can be represented as:

[0141]

[0142] Similarly, we can obtain Analytical solution:

[0143]

[0144] in, .

[0145] (3) Update

[0146] Hash encoding The relevant loss function can be simplified as follows:

[0147]

[0148] Discrete constraints in hash coding are equivalently transformed into the intersection of two continuous spaces. Using the ADMM optimization algorithm to compute discrete hash codes, the integer programming problem can be first transformed into a problem of finding the intersection of two continuous spaces. Introducing additional variables decomposes the original problem into subproblems that are easier to solve. For example... Figure 5 The image shown is a visualization of the discrete constraint equivalent transformation of the optimization method.

[0149]

[0150] The ADMM optimization method is used to solve this non-convex optimization problem. The problem is decomposed into more easily solvable subproblems, which are then iteratively solved until convergence. The augmented Lagrangian function is defined as follows:

[0151]

[0152] ①Update

[0153] For the last term of the similarity matrix, the diagonal elements are set to 0, and the gradient is calculated to obtain the result. The closed-form solution is expressed mathematically as follows:

[0154]

[0155] in, This represents the i-th row of the similarity matrix, i.e., the similarity between sample i and other samples, after removing the negative values. element.

[0156] ②Update

[0157] renew The mathematical expressions for the subproblems are as follows:

[0158]

[0159] Its closed-form solution has the following mathematical expression:

[0160]

[0161] Perform element-level operations to map the elements to an r-dimensional space similar to a cube (when r=3). It is an r-dimensional vector whose elements are all -1. It is an r-dimensional vector with all elements being 1.

[0162] ③Update

[0163] renew The mathematical expressions for the subproblems are as follows:

[0164]

[0165] The optimal solution to this problem is obtained by calculating the Euclidean projection onto a d-dimensional sphere. The formula for this calculation is as follows:

[0166]

[0167] ④ Update and

[0168] For Lagrange multipliers and Update using the gradient descent algorithm, where and It's a hyperparameter.

[0169]

[0170]

[0171] In step S107, after training, the hash codes of the images and text are obtained as the retrieval set, along with the hash functions for the images and text. and During retrieval, the encoder is used to calculate the features of the query sample. After that, you can use Obtain the corresponding hash code, and retrieve the most relevant cross-modal information by calculating the cosine similarity between the hash code and the code in the retrieval set. The function is defined as follows:

[0172]

[0173] like Figure 6 The diagram shown is a flowchart of the model retrieval process.

[0174] In a specific embodiment, to verify the effectiveness of the designed fast association retrieval technology for multimodal remote sensing targets, the multimodal hash retrieval method for fast association of massive remote sensing targets proposed in this application and existing methods (specifically including: IMH-b, CMFH-b, FSH-b and OCMH, with processing similar to other common methods, -b indicating that common offline methods are used for online scenarios of fast retrieval, i.e., the data is only trained once) were tested on the MIRFlickr and NUS-WIDE datasets respectively.

[0175] Table 1. Performance on the MIRFlickr dataset

[0176]

[0177] Table 2 Performance on the NUS-WIDE dataset

[0178]

[0179] Table 1 shows the performance of the proposed method and existing methods on the MIRFlickr dataset. Table 2 shows the performance of the proposed method and existing methods on the NUS-WIDE dataset. The results in Tables 1 and 2 clearly demonstrate that the proposed method significantly improves performance compared to existing methods.

[0180] Furthermore, taking the MIRFlickr dataset as an example, the training time of this application and other methods is shown in Table 3:

[0181] Table 3 Training time (in seconds) on the MIRFlickr dataset

[0182]

[0183] As can be seen from the results in Table 3, compared with other existing methods (specifically IMH and FSH), the method in this application has a shorter training time and can better meet the requirements of large-scale data retrieval.

[0184] The multimodal hashing retrieval method described above for rapid association of massive remote sensing targets introduces a scalable, anchor-based, large-scale unsupervised similarity graph generation strategy. It utilizes the Laplacian regularization term of the similarity graph to constrain the encoding to include more semantic information. High-quality encoding is learned directly without sacrificing discrete constraints, thus resolving the accuracy loss caused by relaxation processing. Furthermore, by learning fine-grained multimodal feature information, it mines the correlations between cross-modal data, effectively improving the accuracy of multimodal remote sensing target retrieval both theoretically and practically.

[0185] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of this disclosure, "a plurality of" means two or more, unless otherwise explicitly specified.

[0186] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. In addition, those skilled in the art can combine and integrate the different embodiments or examples described in this specification.

[0187] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the appended claims.

Claims

1. A multimodal hash retrieval method for rapid association of massive remote sensing targets, characterized in that, The method includes: Acquire multimodal remote sensing sample data; wherein, the multimodal remote sensing sample data includes remote sensing image data and its corresponding text information; Construct a multimodal hash retrieval model; wherein the multimodal hash retrieval model includes at least a unimodal encoder, and the unimodal encoder includes an image encoder and a text encoder; The remote sensing image data and the text information are preprocessed, and the image encoder and the text encoder are used to extract features from the preprocessed remote sensing image data and the text information to obtain image features and text features. Based on the image features and the text features, a large-scale unsupervised similarity graph generation strategy based on anchor points is used to obtain a joint cross-modal graph of the remote sensing image data and the text information. Based on the image features, the text features, and the joint cross-modal graph, a discrete hash code is generated using the factorization method; The discrete hash code is optimized using the ADMM optimization algorithm to obtain the optimal model parameters, and a trained multimodal hash retrieval model is obtained based on the optimal model parameters. The data to be retrieved is input into the trained multimodal hash retrieval model to obtain the retrieval results; The step of obtaining a joint cross-modal graph of the remote sensing image data and the text information by using an anchor-based large-scale unsupervised similarity graph generation strategy based on the image features and the text features includes: From the image features and the text features Select For anchor points As an anchor set, based on n sample pairs and Anchor point pair To obtain the relationship between samples and anchor points within the image and text modalities. and the mapping matrix that maps images and text to a public space. ;in, v Represents mode, v =1 indicates an image. v =2 represents text; Based on the mapping matrix within the image and text modalities The relationship diagram between cross-modal anchor points is obtained. , For the dimension of public space; Based on the relationship diagram between the image and text modal samples and anchor points. The relationship graph between samples within the image and text modalities is obtained. ; Based on the relationship diagram between the image and text cross-modal anchor points Q The relationship between the samples and anchor points within the image and text modalities. The relationship graph between cross-modal samples was obtained. ; Based on the relationship diagram between the samples within the image and text modalities The relationship between the cross-modal samples and the graph The joint cross-modal graph is obtained; wherein, the t-step transition matrix is ​​used. To describe the relationships between samples that are reachable in multiple steps.

2. The multimodal hash retrieval method for rapid association of massive remote sensing targets according to claim 1, characterized in that, The steps of preprocessing the remote sensing image data and the text information, and extracting features from the preprocessed remote sensing image data and the text information using the image encoder and the text encoder respectively to obtain image features and text features, include: Image enhancement, denoising, and alignment are performed on the remote sensing image data to ensure the consistency and quality of the remote sensing image data; The image encoder is used to extract features from the preprocessed remote sensing image data to obtain image features; The text encoder is used to extract features from the preprocessed text information to obtain the text features.

3. The multimodal hash retrieval method for rapid association of massive remote sensing targets according to claim 2, characterized in that, The expression for the sample-anchor relationship graph within the modality is: The expression for the sample-anchor relationship diagram within the modality can be simplified to: in, For image features or text features, For anchor points corresponding to modal features, As the first hyperparameter, This is the second hyperparameter. For the first The image feature or text feature and the first Similarity relationships between text features or image features Indicates the number of anchor points. It is a picture The row degree matrix, It is a picture The column-degree matrix, yes The row vector, The optimal solution is the intermediate term. The matrix consisting of eigenvectors corresponding to the smallest eigenvalues; The expression for the relationship between the intramodal sample and the anchor point is: in, This is a graph showing the relationship between samples and anchor points within image and text modalities. For the first i Image features or text features For the corresponding mode, the first j Anchor point features, This is the third hyperparameter of the corresponding mode; The expression for the relationship graph between the samples within the image and text modalities is as follows: in, This is a graph showing the relationships between samples within an image modality. This is a graph showing the relationships between samples within a text modality. The expression for the relationship diagram between the cross-modal anchor points is: in, For image modal anchor points, For the anchor point of the text modality, For the first The first image anchor point and the first Similarity relationships between text anchors This is the fourth hyperparameter; The expression for the relationship graph between the cross-modal samples is: The expression for the joint cross-modal graph is: Using the t-step transition matrix The expression describing the relationship between samples that can be reached in multiple steps is as follows: in, for of Power of 1 To establish the relationships between image samples after multiple steps of reachability. To establish the relationships between text samples after multiple steps of accessibility. It is cross-modal similarity information obtained through an unsupervised similarity generation strategy. for The transpose of .

4. The multimodal hash retrieval method for rapid association of massive remote sensing targets according to claim 3, characterized in that, The step of generating discrete hash codes using factorization based on the image features, the text features, and the joint cross-modal graph includes: Construct the hash coding loss function for the hash coding factorization method; The image features and text features are uniformly mapped to a common Hamming space to generate a unified initial hash code; Based on the hash encoding loss function, the Laplace matrix of the joint cross-modal graph is introduced into the initial hash encoding to obtain the discrete hash encoding.

5. The multimodal hash retrieval method for rapid association of massive remote sensing targets according to claim 4, characterized in that, The expression for the hash coding loss function is: in, This is the fifth hyperparameter. This is the sixth hyperparameter. This is the seventh hyperparameter. This is the eighth hyperparameter. and These represent the features obtained from the original image and text after being encoded, respectively. and transpose, d 1 represents the dimension of image features. d 2 represents the dimension of text features. Hash encoding, and These are the decomposition matrices for image features and text features, respectively. r The length of the hash code. The image mapping matrix represents the mapping from the feature space to the common space. The text mapping matrix representing the feature space to the common space. Represents the Laplace matrix of the joint cross-modal graph. for A diagonal matrix formed by the sum of the elements in each row. This indicates that the square of the corresponding Frobenius norm is calculated for each matrix in parentheses.

6. The multimodal hash retrieval method for rapid association of massive remote sensing targets according to claim 5, characterized in that, The step of optimizing the discrete hash code using the ADMM optimization algorithm to obtain optimal model parameters, and then obtaining the trained multimodal hash retrieval model based on the optimal model parameters, includes: The matrices that need to be solved by alternating optimization include , , , and ; The discrete constraints in the discrete hash code are equivalently transformed into the intersection of two continuous spaces, and the ADMM optimization algorithm is used to transform it into a problem of the intersection of two continuous spaces. Define an augmented Lagrangian function, decompose the problem into multiple subproblems, iteratively solve the subproblems until convergence, and obtain the optimal model parameters. The multimodal hash retrieval model is updated using the optimal model parameters to obtain the trained multimodal hash retrieval model.

7. The multimodal hash retrieval method for rapid association of massive remote sensing targets according to claim 6, characterized in that, The problem is: in, For the first Image features, For the first Text features, For the first A hash code, It is an r-dimensional space similar to a cube. It is an r-dimensional sphere; The expression for the augmented Lagrange function is: in, As the first Lagrange multiplier, For the second Lagrange multiplier, It is the ninth hyperparameter. This is the tenth hyperparameter.

Images