Multi-modal Hash retrieval method for rapid association of massive remote sensing targets

Abstract

The invention belongs to the technical field of artificial intelligence. The invention provides a multi-modal Hash retrieval method for rapid association of massive remote sensing targets. According to the embodiment of the invention, an extensible large-scale unsupervised similarity graph generation strategy based on anchor points is introduced, and the Laplacian regular term of the similarity graph is utilized to limit the Hash coding to contain more semantic information. On the premise of not sacrificing discrete constraints, high-quality coding is directly learned, and the problem of coding precision loss caused by relaxation processing is solved. By learning fine-grained multi-modal feature information, association among cross-modal data is mined, and the accuracy of multi-modal remote sensing target retrieval is effectively improved theoretically and practically.

Description

Technical Field

[0001] The disclosed embodiments relate to the field of artificial intelligence technology, and in particular to a multimodal hash retrieval method for rapid association of massive remote sensing targets. Background Art

[0002] Multimodal retrieval is an important research direction in the field of computer vision and information retrieval. It solves the problem of how to make a query in one data modality (such as text) and retrieve related content in another different modality (such as image, video or audio). In the context of remote sensing technology, multimodal data refers to multiple data types obtained by different sensors. Although these data are in different forms, they are often related to each other and describe the same events or objects. This technology can not only improve the comprehensive utilization efficiency of data, but also significantly improve the efficiency and accuracy of information retrieval. Multimodal retrieval technology not only has important application value in military reconnaissance, environmental monitoring and disaster warning, but also promotes modern urban management and development such as intelligent transportation and urban planning.

[0003] However, the implementation of multimodal retrieval technology still faces multiple challenges, including how to effectively extract and transform features between modalities, how to build models that can understand and match the complex relationships between different modalities, and how to process and optimize the retrieval efficiency of massive multimodal data sets. How to quickly and accurately retrieve the required information from multimodal data has become an urgent problem that needs to be solved.

[0004] Hash coding technology is often used in fast association retrieval to achieve low storage cost and high retrieval efficiency. Common methods are mainly divided into two modules: feature learning and hash coding learning. The input is usually a pair of remote sensing image text pairs, and the text is multiple sentences describing the remote sensing image. The image and text obtain the corresponding modal features through the image encoder and text encoder respectively, and then build a similarity graph based on the feature or label data. Then, the loss function is designed according to the similarity graph and model characteristics to guide the hash function to generate the desired hash code, and finally the hash code is optimized. Due to the different feature spaces of different modal samples, the association between cross-modal samples cannot be directly calculated. A common approach in unsupervised scenarios is to first calculate the similarity between samples in a single modal space, and build a graph based on the similarity threshold or top-k nearest neighbor set, and then weight the single modal graph to obtain the total similarity graph. When optimizing hash coding, continuous encoding is often learned first and then converted into discrete form.

[0005] The above approach has two major flaws: first, the similarity map does not take into account the correlation between cross-modal samples, lacks more fine-grained semantic information, and often does not consider the imbalance in the number of samples of different modalities in the dataset; second, the hash coding optimization first learns continuous coding, and there is a large gap between the continuous coding in the learning process and the final expected coding, which will result in a loss of accuracy.

[0006] Therefore, it is necessary to improve one or more problems existing in the above-mentioned related technical solutions.

[0007] It should be noted that this section is intended to provide background or context for the technical solutions of the present disclosure stated in the claims. The description herein is not admitted to be prior art by virtue of being included in this section. Summary of the invention

[0008] The purpose of the embodiments of the present disclosure is to provide a multimodal hash retrieval method for rapid association of massive remote sensing targets, thereby overcoming one or more problems caused by limitations and defects of related technologies at least to a certain extent.

[0009] According to an embodiment of the present disclosure, a multimodal hash retrieval method for rapid association of massive remote sensing targets is provided, the method comprising: Acquire multimodal remote sensing sample data; wherein the multimodal remote sensing sample data includes remote sensing image data and corresponding text information; Constructing a multimodal hash retrieval model; wherein the multimodal hash retrieval model at least includes a unimodal encoder, and the unimodal encoder includes an image encoder and a text encoder; Preprocessing the remote sensing image data and the text information, and extracting features from the preprocessed remote sensing image data and the text information using the image encoder and the text encoder, respectively, to obtain image features and text features; Based on the image features and the text features, a large-scale unsupervised similarity graph generation strategy based on anchor points is used to obtain a joint cross-modal graph of the remote sensing image data and the text information; Based on the image features, the text features and the joint cross-modal graph, a discrete hash code is generated using a factor decomposition method; Optimizing the discrete hash code by using the ADMM optimization algorithm to obtain optimal model parameters, and obtaining a trained multimodal hash retrieval model according to the optimal model parameters; The data to be retrieved is input into the trained multimodal hash retrieval model to obtain the retrieval result.

[0010] Further, the steps of preprocessing the remote sensing image data and the text information, respectively extracting features from the preprocessed remote sensing image data and the text information using the image encoder and the text encoder to obtain image features and text features include: Performing image enhancement, denoising and alignment on the remote sensing image data to ensure the consistency and quality of the remote sensing image data; Using the image encoder to perform feature extraction on the preprocessed remote sensing image data to obtain image features; The text encoder is used to perform feature extraction on the preprocessed text information to obtain the text features.

[0011] Furthermore, based on the image features and the text features, a large-scale unsupervised similarity graph generation strategy based on anchor points is used to obtain a joint cross-modal graph of the remote sensing image data and the text information, including: From the image features and the text features Select Anchor point As an anchor point set, based on n sample pairs and Anchor point pairs , to obtain the relationship diagram between samples and anchor points in image and text modalities , and the mapping matrix that maps images and text to a common space ;in, v Indicates the mode, v =1 means image, v =2 means text; According to the mapping matrix within the image and text modalities , and obtain the relationship diagram between cross-modal anchor points , for the dimension of public space; According to the relationship diagram between samples and anchor points in the image and text modalities , we get the relationship graph between samples in image and text modalities ; According to the relationship diagram between the image and text cross-modal anchor points Q and the relationship diagram between samples and anchor points in the image and text modalities , and obtain the relationship diagram between cross-modal samples ; According to the relationship diagram between samples in the image and text modalities and the relationship diagram between the cross-modal samples , and obtain the joint cross-modal graph; wherein, using the t-step transition matrix To depict the relationship between samples that can be reached in multiple steps.

[0012] Furthermore, the expression of the sample-anchor relationship graph in the modality is:

[0013] The expression of the sample-anchor relationship graph in the modality can be obtained by sorting:

[0014] in, is an image feature or a text feature, is the anchor point corresponding to the modal feature, is the first hyperparameter, is the second hyperparameter, For the The image feature or text feature is The similarity relationship between text features or image features, represents the number of anchor points, It is a picture The row degree matrix of It is a picture The column degree matrix of yes The row vector of The optimal solution is the intermediate term The matrix composed of the eigenvectors corresponding to the smallest eigenvalues; The expression of the relationship graph between samples and anchor points in the modality is:

[0015] in, is the relationship diagram between samples and anchor points in image and text modalities, For the i image features or text features, is the first j Anchor features, is the third hyperparameter of the corresponding mode; The expression of the relationship graph between samples in the image and text modalities is:

[0016] in, is the relationship diagram between samples in the image modality, It is a relationship diagram between samples in the text modality; The expression of the relationship graph between the cross-modal anchor points is:

[0017] in, is the anchor point of the image modality, is the anchor point of the text modal, For the image anchors and The similarity relationship between text anchors, is the fourth hyperparameter; The expression of the relationship graph between the cross-modal samples is:

[0018] The expression of the joint cross-modal graph is:

[0019] Using the t-step transition matrix The expression to describe the relationship between samples that can be reached in multiple steps is:

[0020] in, for of Power, is the relationship between image samples after multi-step reachability, is the relationship between text samples after multi-step reachability, It is the cross-modal similarity information obtained through the unsupervised similarity generation strategy. for The transpose of .

[0021] Furthermore, the step of generating discrete hash codes using a factor decomposition method based on the image features, the text features and the joint cross-modal graph includes: Construct the hash code loss function of the hash code factorization method; The image features and the text features are uniformly mapped to a common Hamming space to generate a uniform initial hash code; Based on the hash coding loss function, a Laplacian matrix of the joint cross-modal graph is introduced into the initial hash coding to obtain the discrete hash coding.

[0022] Furthermore, the expression of the hash coding loss function is:

[0023] in, is the fifth hyperparameter, is the sixth hyperparameter, is the seventh hyperparameter, is the eighth hyperparameter, and Respectively represent the features of the original image and text after passing through the encoder, that is and The transpose of d 1 is the dimension of image features, d 2 is the dimension of text features, is a hash code, and are the decomposition matrices of image features and text features respectively, r is the length of the hash code, represents the image mapping matrix from feature space to common space, represents the text mapping matrix from feature space to common space, represents the Laplacian matrix of the joint cross-modal graph, for A diagonal matrix consisting of the sum of the elements in each row, It means to calculate the square of the corresponding Frobenius norm for each matrix in the brackets.

[0024] Further, the discrete hash code is optimized by using an ADMM optimization algorithm to obtain optimal model parameters, and the step of obtaining a trained unimodal encoder according to the optimal model parameters includes: The matrices to be solved by alternating optimization include , , , and ; The discrete constraints in the discrete hash code are equivalently changed into the intersection of two continuous spaces, and the ADMM optimization algorithm is used to convert it into the intersection of two continuous spaces; Define an augmented Lagrangian function, decompose the problem into multiple sub-problems, and iteratively solve the sub-problems until convergence to obtain the optimal model parameters The multimodal hash retrieval model is updated using the optimal model parameters to obtain the trained multimodal hash retrieval model.

[0025] Furthermore, the problem is:

[0026] in, For the image features, For the text features, For the hash codes, is a cube-like r-dimensional space, is an r-dimensional sphere; The expression of the augmented Lagrangian function is:

[0027] in, is the first Lagrange multiplier, is the second Lagrange multiplier, is the ninth hyperparameter, is the tenth hyperparameter.

[0028] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects: In the embodiments of the present disclosure, through the above-mentioned multimodal hash retrieval method for rapid association of massive remote sensing targets, on the one hand, a scalable large-scale unsupervised similarity graph generation strategy based on anchor points is introduced, and the Laplace regularization term of the similarity graph is used to restrict the encoding to contain more semantic information. Without sacrificing discrete constraints, high-quality encoding is directly learned, which solves the problem of loss of encoding accuracy caused by relaxation processing. On the other hand, it is possible to learn fine-grained multimodal feature information and explore the association between cross-modal data, effectively improving the accuracy of multimodal remote sensing target retrieval in theory and practice. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0030] Figure 1 A step diagram showing a multi-modal hash retrieval method for rapid association of massive remote sensing targets in an exemplary embodiment of the present disclosure; Figure 2 A flow chart of model training in an exemplary embodiment of the present disclosure is shown; Figure 3 A schematic diagram showing a large-scale unsupervised similarity graph generation strategy in an exemplary embodiment of the present disclosure; Figure 4 A schematic diagram showing a hash coding technique in an exemplary embodiment of the present disclosure; Figure 5 A visual schematic diagram showing the equivalent transformation of discrete constraints in an exemplary embodiment of the present disclosure; Figure 6 A model retrieval flow chart in an exemplary embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0031] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0032] In addition, the accompanying drawings are only schematic illustrations of the embodiments of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated descriptions will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities.

[0033] This example implementation provides a multi-modal hash retrieval method for rapid association of massive remote sensing targets. Figure 1 As shown in , the multimodal hash retrieval method for rapid association of massive remote sensing targets may include: steps S101 to S107.

[0034] Step S101: Acquire multimodal remote sensing sample data; wherein the multimodal remote sensing sample data includes remote sensing image data and corresponding text information; Step S102: constructing a multimodal hash retrieval model; wherein the multimodal hash retrieval model at least includes a unimodal encoder, and the unimodal encoder includes an image encoder and a text encoder; Step S103: preprocessing the remote sensing image data and the text information, respectively extracting features from the preprocessed remote sensing image data and the text information using the image encoder and the text encoder to obtain image features and text features; Step S104: Based on the image features and the text features, a large-scale unsupervised similarity graph generation strategy based on anchor points is used to obtain a joint cross-modal graph of the remote sensing image data and the text information; Step S105: generating a discrete hash code using a factor decomposition method based on the image features, the text features and the joint cross-modal graph; Step S106: optimizing the discrete hash code by using the ADMM optimization algorithm to obtain optimal model parameters, and obtaining a trained multimodal hash retrieval model according to the optimal model parameters; Step S107: input the data to be retrieved into the trained multimodal hash retrieval model to obtain retrieval results.

[0035] Through the above multimodal hash retrieval method for rapid association of massive remote sensing targets, on the one hand, a scalable large-scale unsupervised similarity graph generation strategy based on anchor points is introduced, and the Laplace regularization term of the similarity graph is used to restrict the encoding to contain more semantic information. Without sacrificing discrete constraints, high-quality encoding is directly learned, solving the problem of encoding accuracy loss caused by relaxation processing. On the other hand, by learning fine-grained multimodal feature information and mining the association between cross-modal data, the accuracy of multimodal remote sensing target retrieval is effectively improved in theory and practice.

[0036] Next, we will refer to Figures 1 to 6 The various steps of the multimodal hash retrieval method for rapid association of massive remote sensing targets in this example implementation are described in more detail.

[0037] In step S101, multimodal remote sensing data are obtained, which include remote sensing images and their corresponding description information. The remote sensing images are obtained through different sensors such as optics and radar, and the description information is provided by experts or an automatic generation system.

[0038] like Figure 2 As shown, this is the training flow chart of the multimodal retrieval model.

[0039] In step S102, the unimodal encoder includes an image encoder and a text encoder, wherein the image encoder and the text encoder are existing encoders, for example, the image encoder uses a convolutional neural network, and the text encoder uses a bag-of-words model, which is not specifically limited here.

[0040] In step S103, the collected multimodal remote sensing data is preprocessed, including image enhancement, denoising, alignment and other steps to ensure the consistency and quality of the data. An appropriate encoder is selected to convert the image and text data into corresponding embedding vectors.

[0041] In step S104, for the scalable large-scale unsupervised similarity graph generation strategy based on anchor points, in order to address the problem that common similarity graphs do not fully consider the intra-modal and inter-modal associations, anchor points are used to comprehensively consider the intra-modal and inter-modal associations and the multi-step reachable relationships between samples. By analyzing the influence of the hyperparameters of the strategy and comparing it with other common construction methods, the results show that the strategy can effectively guide the generation of hash codes. Figure 3 As shown in Figure 2, it is a schematic diagram of the strategy for generating large-scale unsupervised similarity graphs.

[0042] A specific and scalable large-scale unsupervised similarity graph generation strategy based on anchor points. The input of the similarity graph is the remote sensing image features obtained by the image encoder and the text encoder. and related text features , choose from Using image text features as anchors Given a sample pair and anchor pair (v represents the modality, 1 represents image, and 2 represents text). The mathematical expression of the sample-anchor relationship graph within the modality is defined as follows:

[0043] The above expression can be converted into the following expression:

[0044] Among them, the mapping matrix Map image and text features into the same space. is the intra-modal sparse graph, represents the number of anchor points, It is a picture The row degree matrix of It is a picture The column degree matrix of yes The row vector of . The optimal solution is the intermediate term The matrix consists of the eigenvectors corresponding to the smallest eigenvalues.

[0045] calculate It needs to be determined by sorting the Euclidean distance between the anchor point and the sample from small to large. nearest neighbors, and then dynamically update them using the learned mapping matrix The weight of the nearest neighbor. Since the composition method in image and text modalities is the same, we use The unified representation is shown in the following mathematical expression:

[0046] Next, we further calculate the relationship graph between samples in the text and image modalities , as shown below:

[0047] Get the mapping matrix between image and text modalities and After that, the relationship diagram between cross-modal anchor points The mathematical expression is as follows:

[0048] picture and For a sparse graph, only non-zero vectors, the sum of the elements in each row is 1, that is, the sum of the similarities between the sample and other samples is 1. Calculation It is necessary to first map the image and text anchor points to the same space, and then calculate the Euclidean distance to determine Nearest neighbor, then directly use the following formula to calculate elements.

[0049]

[0050] Getting the relationship between cross-modal anchor points And the relationship between samples and anchor points within the modality and After that, the relationship graph between cross-modal samples can be calculated. is the affinity matrix of the bipartite graph defined between the image and the text, as shown below:

[0051] The joint cross-modal graph consists of three parts, and its expression is defined as follows:

[0052] Using the t-step transition matrix To describe the relationship between samples that can be reached in multiple steps, its mathematical expression is defined as follows:

[0053] where, for a homogeneous Markov chain, yes The t power, It can be divided into four parts, and Respectively represent the similarity information within the modality, Fig. It is used to strengthen the cross-modal association through the Laplace regularization term in the future.

[0054] In step S105, for hash code generation, matrix factorization technology is used to uniformly map image and text features to a common Hamming space to generate a unified and efficient hash code, and similarity graph is introduced to generate a uniform and efficient hash code. The Laplace regularization term of enhances the semantic information of the code. The optimization process includes the derivation of the closed-form solution, the application of the alternating update strategy, and the calculation of the discrete hash code. Figure 4 As shown, it is a schematic diagram of discrete coding optimization technology.

[0055] Specifically, for hash coding, the loss function of the factorization method can be defined as follows:

[0056] in, and Respectively represent the features of the original image and text after passing through the encoder, that is and The transpose of the matrix and Decompose into a common Hamming space , r represents the length of the hash code. and represents the mapping matrix from feature space to common space, Represents the graph obtained in the previous step The Laplace matrix of for A diagonal matrix consisting of the sum of the elements in each row.

[0057] In step S106, for discrete coding optimization technology, in order to solve the problem of coding accuracy loss caused by relaxation processing, the ADMM optimization algorithm is introduced while retaining discrete constraints to directly learn high-quality hash codes, thereby effectively improving the accuracy and efficiency of cross-modal retrieval in theory and practice.

[0058] Specifically, the discrete coding optimization technique uses an alternating optimization method, where only one variable is updated each time while other variables are fixed. The matrix to be solved includes , , , , ,Apart from The closed-form solution of matrices other than can be directly obtained by differentiating the loss function, and Discrete constraints require further processing.

[0059] (1) Update

[0060] Analyze first The update process, The update process is similar to that of The relevant loss function can be simplified as follows:

[0061] By calculating the derivative and setting it to 0, we can get the optimal solution to the problem as follows:

[0062] The accumulated data is defined as shown in the following formula. At the end of each iteration, the product of the relevant variables is updated for subsequent calculations.

[0063]

[0064]

[0065] Therefore, considering the cumulative data It can be expressed as follows:

[0066] The optimal solution of can be obtained in the same way:

[0067] Among them, the cumulative data is defined as follows:

[0068] (2) Update

[0069] Fixed other variables to only update or , first analyze The update process, The update process is similar to that of The relevant loss function can be simplified as follows:

[0070] By calculating the derivative to 0, we can get The analytical solution is shown below:

[0071] The definition of the cumulative variable product is as follows:

[0072]

[0073] thus It can be expressed as:

[0074] The same can be said The analytical solution is:

[0075] in, .

[0076] (3) Update

[0077] Hash Encoding The relevant loss function can be simplified as follows:

[0078] The discrete constraints in hash coding are equivalent to the intersection of two continuous spaces. Using the ADMM optimization algorithm to calculate discrete hash coding, we can first convert the integer programming problem into the intersection problem of two continuous spaces, and introduce additional variables to decompose the original problem into sub-problems that are easier to solve. Figure 5 As shown, it is a visualization diagram of the equivalent transformation of discrete constraints of the optimization method.

[0079]

[0080] The ADMM optimization method is used to solve the non-convex optimization problem, which is decomposed into sub-problems that are easier to solve, and the sub-problems are solved iteratively until convergence. The definition of the augmented Lagrangian function is as follows:

[0081] ① Update

[0082] For the last similarity matrix, the diagonal elements are set to 0 and the gradient is calculated to obtain The closed-form solution of is as follows:

[0083] in, Represents the i-th row of the similarity matrix, that is, the similarity between sample i and other samples, and removes element.

[0084] ② Update

[0085] renew The mathematical expression of the sub-problem is as follows:

[0086] The mathematical expression of its closed-form solution is as follows:

[0087] Perform element-wise operations, mapping the elements into an r-dimensional space similar to a cube (when r=3), is an r-dimensional vector whose elements are all -1. is an r-dimensional vector whose elements are all 1.

[0088] ③ Update

[0089] renew The mathematical expression of the sub-problem is as follows:

[0090] This problem is solved optimally by calculating the Euclidean projection on a d-dimensional sphere, and the calculation formula is:

[0091] ④Update and

[0092] For the Lagrange multiplier and Update using the gradient descent algorithm, where and is a hyperparameter.

[0093]

[0094]

[0095] In step S107, after the training is completed, the hash codes of the image and text are obtained as the retrieval set, and the hash functions for the image and text are obtained. and . The encoder is used to calculate the features of the query sample during retrieval After that, you can use Obtain the corresponding hash code and retrieve the most relevant cross-modal information by calculating the cosine similarity with the code in the retrieval set, where The function definition is as follows:

[0096] like Figure 6 The figure shows the model retrieval flow chart.

[0097] In a specific embodiment, in order to verify the effectiveness of the designed fast association retrieval technology for multimodal remote sensing targets, the multimodal hash retrieval method for fast association of massive remote sensing targets proposed in this application is tested on the MIRFlickr and NUS-WIDE datasets respectively with existing methods (specifically including: IMH-b, CMFH-b, FSH-b and OCMH, following the processing of other common methods, -b means that the common offline method is used for online scenarios of fast retrieval, that is, the data is trained only once).

[0098] Table 1 Performance on the MIRFlickr dataset

[0099] Table 2 Performance on the NUS-WIDE dataset

[0100] As shown in Table 1, the performance of the method of the present application and the existing method on the MIRFlickr dataset is shown. As shown in Table 2, the performance of the method of the present application and the existing method on the NUS-WIDE dataset is shown. From the results of Tables 1 and 2, it can be seen that compared with the existing method, the effect of the method of the present application is significantly improved.

[0101] In addition, taking the MIRFlickr dataset as an example, the training time of this application and other methods is shown in Table 3: Table 3 Training time on the MIRFlickr dataset (in seconds)

[0102] It can be seen from the result data in Table 3 that, compared with other existing methods (specifically including IMH and FSH), the method of the present application has a shorter training time than other algorithms and can better meet the requirements of large-scale data retrieval.

[0103] Through the above multimodal hash retrieval method for rapid association of massive remote sensing targets, on the one hand, a scalable large-scale unsupervised similarity graph generation strategy based on anchor points is introduced, and the Laplace regularization term of the similarity graph is used to restrict the encoding to contain more semantic information. Without sacrificing discrete constraints, high-quality encoding is directly learned, solving the problem of encoding accuracy loss caused by relaxation processing. On the other hand, by learning fine-grained multimodal feature information and mining the association between cross-modal data, the accuracy of multimodal remote sensing target retrieval is effectively improved in theory and practice.

[0104] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present disclosure, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0105] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification.

[0106] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any modification, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.

Claims

1. A multimodal hash retrieval method for rapid association of massive remote sensing targets, characterized in that: The method includes: Acquire multimodal remote sensing sample data; wherein the multimodal remote sensing sample data includes remote sensing image data and corresponding text information; Constructing a multimodal hash retrieval model; wherein the multimodal hash retrieval model at least includes a unimodal encoder, and the unimodal encoder includes an image encoder and a text encoder; Preprocessing the remote sensing image data and the text information, and extracting features from the preprocessed remote sensing image data and the text information using the image encoder and the text encoder, respectively, to obtain image features and text features; Based on the image features and the text features, a large-scale unsupervised similarity graph generation strategy based on anchor points is used to obtain a joint cross-modal graph of the remote sensing image data and the text information; Based on the image features, the text features and the joint cross-modal graph, a discrete hash code is generated using a factor decomposition method; Optimizing the discrete hash code by using the ADMM optimization algorithm to obtain optimal model parameters, and obtaining a trained multimodal hash retrieval model according to the optimal model parameters; The data to be retrieved is input into the trained multimodal hash retrieval model to obtain the retrieval result.

2. The multimodal hash retrieval method for rapid association of massive remote sensing targets according to claim 1 is characterized in that: The steps of preprocessing the remote sensing image data and the text information, and extracting features from the preprocessed remote sensing image data and the text information using the image encoder and the text encoder respectively to obtain image features and text features include: Performing image enhancement, denoising and alignment on the remote sensing image data to ensure the consistency and quality of the remote sensing image data; Using the image encoder to perform feature extraction on the preprocessed remote sensing image data to obtain image features; The text encoder is used to perform feature extraction on the preprocessed text information to obtain the text features.

3. The multimodal hash retrieval method for rapid association of massive remote sensing targets according to claim 1 is characterized in that: Based on the image features and the text features, a large-scale unsupervised similarity graph generation strategy based on anchor points is used to obtain a joint cross-modal graph of the remote sensing image data and the text information, comprising: From the image features and the text features Select Anchor point As an anchor point set, based on n sample pairs and Anchor point pairs , to obtain the relationship diagram between samples and anchor points in image and text modalities , and the mapping matrix that maps images and text to a common space ;in, v Indicates the mode, v =1 means image, v =2 means text; According to the mapping matrix within the image and text modalities , and obtain the relationship diagram between cross-modal anchor points , for the dimension of public space; According to the relationship diagram between samples and anchor points in the image and text modalities , we get the relationship graph between samples in image and text modalities ; According to the relationship diagram between the image and text cross-modal anchor points Q and the relationship diagram between samples and anchor points in the image and text modalities , and obtain the relationship diagram between cross-modal samples ; According to the relationship diagram between samples in the image and text modalities and the relationship diagram between the cross-modal samples , and obtain the joint cross-modal graph; wherein, using the t-step transition matrix To depict the relationship between samples that can be reached in multiple steps.

4. The multimodal hash retrieval method for rapid association of massive remote sensing targets according to claim 3 is characterized in that: The expression of the sample-anchor relationship graph in the modality is: The expression of the sample-anchor relationship graph in the modality can be obtained by sorting: in, is an image feature or a text feature, is the anchor point corresponding to the modal feature, is the first hyperparameter, is the second hyperparameter, For the The image feature or text feature is The similarity relationship between text features or image features, represents the number of anchor points, It is a picture The row degree matrix of It is a picture The column degree matrix of yes The row vector of The optimal solution is the intermediate term The matrix composed of the eigenvectors corresponding to the smallest eigenvalues; The expression of the relationship graph between samples and anchor points in the modality is: in, is the relationship diagram between samples and anchor points in image and text modalities, For the i image features or text features, is the first j Anchor features, is the third hyperparameter of the corresponding mode; The expression of the relationship graph between samples in the image and text modalities is: in, is the relationship diagram between samples in the image modality, It is a relationship diagram between samples in the text modality; The expression of the relationship graph between the cross-modal anchor points is: in, is the anchor point of the image modality, is the anchor point of the text modal, For the image anchors and The similarity relationship between text anchors, is the fourth hyperparameter; The expression of the relationship graph between the cross-modal samples is: The expression of the joint cross-modal graph is: Using the t-step transition matrix The expression to describe the relationship between samples that can be reached in multiple steps is: in, for of Power, is the relationship between image samples after multi-step reachability, is the relationship between text samples after multi-step reachability, It is the cross-modal similarity information obtained through the unsupervised similarity generation strategy. for The transpose of .

5. The multimodal hash retrieval method for rapid association of massive remote sensing targets according to claim 4 is characterized in that: The step of generating a discrete hash code using a factor decomposition method based on the image features, the text features and the joint cross-modal graph includes: Construct the hash code loss function of the hash code factorization method; The image features and the text features are uniformly mapped to a common Hamming space to generate a uniform initial hash code; Based on the hash coding loss function, a Laplacian matrix of the joint cross-modal graph is introduced into the initial hash coding to obtain the discrete hash coding.

6. The multimodal hash retrieval method for rapid association of massive remote sensing targets according to claim 5 is characterized in that: The expression of the hash coding loss function is: in, is the fifth hyperparameter, is the sixth hyperparameter, is the seventh hyperparameter, is the eighth hyperparameter, and Respectively represent the features of the original image and text after passing through the encoder, that is and The transpose of d 1 is the dimension of image features, d 2 is the dimension of text features, is a hash code, and are the decomposition matrices of image features and text features respectively, r is the length of the hash code, represents the image mapping matrix from feature space to common space, represents the text mapping matrix from feature space to common space, represents the Laplacian matrix of the joint cross-modal graph, for A diagonal matrix consisting of the sum of the elements in each row, It means to calculate the square of the corresponding Frobenius norm for each matrix in the brackets.

7. The multimodal hash retrieval method for rapid association of massive remote sensing targets according to claim 6 is characterized in that: The step of optimizing the discrete hash code by using the ADMM optimization algorithm to obtain optimal model parameters and obtaining a trained multimodal hash retrieval model according to the optimal model parameters includes: The matrices to be solved by alternating optimization include , , , and ; The discrete constraints in the discrete hash code are equivalently changed into the intersection of two continuous spaces, and the ADMM optimization algorithm is used to convert it into the intersection of two continuous spaces; Define an augmented Lagrangian function, decompose the problem into multiple sub-problems, and iteratively solve the sub-problems until convergence to obtain the optimal model parameters; The multimodal hash retrieval model is updated using the optimal model parameters to obtain the trained multimodal hash retrieval model.

8. The multimodal hash retrieval method for rapid association of massive remote sensing targets according to claim 7 is characterized in that: The problem is: in, For the image features, For the text features, For the hash codes, is a cube-like r-dimensional space, is an r-dimensional sphere; The expression of the augmented Lagrangian function is: in, is the first Lagrange multiplier, is the second Lagrange multiplier, is the ninth hyperparameter, is the tenth hyperparameter.

Images