Remote sensing image-text retrieval method, system and equipment based on deep learning and medium

Abstract

The invention relates to the field of image processing, and discloses a remote sensing image-text retrieval method, system and device based on deep learning and a medium, and the method comprises the steps: inputting a remote sensing image or remote sensing text data to be retrieved, and extracting the multi-scale features of the image and the multi-scale features of the text at the same time through a cross-modal multi-scale extraction network, through a cross-modal multi-scale alignment network, image features of different scales are utilized to guide the joint text features, text features of different modals after scale alignment are obtained, and a plurality of subspaces after scale alignment are constructed; the image features and the text features are updated in the global space and the constructed multiple subspaces through the subspace updating network and the global space updating network, and a retrieval result is generated through the similarity matching network according to the similarity matching degree. According to the method, multi-scale modeling of the remote sensing data and scale alignment among different modals are realized, irrelevant modeling among different modals is ensured, and the relationship of data of different modals is enriched.

Description

Technical Field

[0001] The present application relates to the field of remote sensing data processing, and specifically to a remote sensing image and text retrieval method, system, equipment and medium based on deep learning. Background Art

[0002] With the rapid development of aerospace technology and computer technology, remote sensing data and its applications have exploded. Reasonable use of remote sensing data can better help and improve people's lives and production methods. Remote sensing image and text retrieval refers to obtaining more interesting and valuable data (text or images) from massive remote sensing data (images or texts). It plays an important role in resource surveys, disaster detection, agricultural production and other fields. Existing remote sensing image and text retrieval methods inevitably have problems such as low matching accuracy and high matching error rate. Summary of the invention

[0003] The technical problem to be solved by this application is to overcome the shortcomings of the existing technology and provide a remote sensing image and text retrieval method, system, equipment and medium based on deep learning, which can improve the matching accuracy in remote sensing image and text retrieval and reduce the matching error rate of remote sensing image and text retrieval.

[0004] To achieve the above objectives, the first aspect of the present application provides a remote sensing image and text retrieval method based on deep learning, the method comprising the following steps: Step S1, constructing a cross-modal multi-scale extraction network; The cross-modal multi-scale extraction network at least includes a large kernel convolution multi-scale image feature extraction module and a multi-scale text feature extraction module, and the two modules simultaneously extract the multi-scale image features of the image. Multi-scale text features of text Perform extraction; Step S2, constructing a cross-modal multi-scale alignment network; The cross-modal multi-scale alignment network at least includes a subspace construction module, and the subspace construction module uses multi-scale image features to Multi-scale text features Cascaded joint text features Provide guidance to obtain text features after scale alignment between different modalities , and use multi-scale image features and text features Constructing subspaces aligned at multiple scales ; Step S3, construct subspace and global space update networks; The subspace and global space update network at least includes a subspace mapping module based on irrelevance and a global space mapping module, wherein the two mapping modules are respectively in the global space and in the constructed multiple subspaces. In the above example, image features and text features are updated; Step S4, constructing a similarity matching network; The similarity matching network at least includes a multi-granularity similarity matching module, and the multi-granularity similarity matching module performs similarity matching on updated image features and text features globally and at multiple scale levels; Step S5, combining the networks established in the above steps, inputting the remote sensing image or remote sensing text data to be retrieved, and generating retrieval results based on the similarity matching degree.

[0005] Furthermore, in step S1, the large kernel convolution multi-scale image feature extraction module extracts image features through a convolutional neural network. , multiple convolution operations of different sizes are used to further extract multi-scale image features ; Multi-scale image features and image features Aggregate into global image features ; Obtaining global image features It is expressed as: ; in, represents a 3×3 convolution operation, represents the matrix dot multiplication operation, Represents image features extracted by convolutional neural network.

[0006] Furthermore, in step S1, the multi-scale text feature extraction module extracts global text features through a bidirectional gated recurrent unit GRU , after obtaining the global text features Afterwards, multiple one-dimensional convolutions of different lengths are used to extract text features of different scales to obtain multi-scale text features. .

[0007] Furthermore, in step S2, the subspace construction module converts the multi-scale text features Concatenation into joint text features , the subspace construction module uses multi-scale image features Combined text features Provide guidance to obtain text features after scale alignment between the two modalities , using the scale-aligned text features With multi-scale image features , construct multiple subspaces ; Get text features after scale alignment It is expressed as: ; Construct the subspace It is expressed as: ; Among them, Cat represents the cascade operation, σ represents the sigmoid function, express function, Represents matrix dot product.

[0008] Further, in step S3, the subspace mapping module uses the subspace Multi-scale image features and multi-scale text features Update to obtain the final output multi-scale image features and multi-scale text features ; Multi-scale image features The update is expressed as: ; Multi-scale image features The update is expressed as: ; in, and Represents the sigmoid operation after different linear layers, and represents the tanh operation after different linear layers, Represents a matrix dot product operation.

[0009] Furthermore, in step S3, the global space mapping module uses global image features and global text features Constructing joint features The global spatial mapping module uses the joint features of the global features of the image and text Global image features and global text features Update them separately to get the final output global image features and global text features ; Global image features The update is expressed as: ; Global text features The update is expressed as: ; in, and Represents the sigmoid operation after different linear layers, and represents the tanh operation after different linear layers, Represents a matrix dot product operation.

[0010] Furthermore, in step S3, the global space mapping module uses global image features and global text features Constructing joint features ; Constructing joint features It is expressed as: ; Among them, linear represents linear layer operation, and Cat represents cascade operation.

[0011] To achieve the above objectives, the second aspect of the present application provides a cross-modal remote sensing image and text retrieval system based on deep learning, the system comprising: The cross-modal multi-scale extraction module includes a large kernel convolution multi-scale image feature extraction module and a multi-scale text feature extraction module. The two feature extraction modules respectively extract the multi-scale image features of the image. Multi-scale images and text Extract features; The cross-modal multi-scale alignment module comprises at least a subspace construction module, wherein the subspace construction module uses multi-scale image features to Multi-scale text features Cascaded joint text features Provide guidance to obtain text features after scale alignment between different modalities , and use multi-scale image features and text features Constructing subspaces aligned at multiple scales ; Subspace and global space update modules, the update module at least includes a subspace mapping module based on correlation and irrelevance and a global space mapping module, the two mapping modules are respectively in the global space and the constructed multiple subspaces In the above example, image features and text features are updated; The similarity matching module at least includes a multi-granularity similarity matching module, and the multi-granularity similarity matching module performs similarity matching on updated image features and text features globally and at multiple scale levels.

[0012] To achieve the above-mentioned purpose, the third aspect of the present application provides a remote sensing image and text retrieval device based on deep learning, including a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the remote sensing image and text retrieval method based on deep learning as described above is implemented.

[0013] To achieve the above-mentioned purpose, the fourth aspect of the present application provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the remote sensing image and text retrieval method based on deep learning as described above.

[0014] After adopting the above technical solution, the present application has the following beneficial effects compared with the prior art: In this application, a cross-modal multi-scale extraction network is used to further extract the multi-scale features of the image, and the multi-scale features of the remote sensing image and remote sensing text are extracted at the same time, thereby ensuring the comprehensiveness of the feature extraction of remote sensing data; a cross-modal multi-scale alignment network is used to guide the text features by using multi-scale image features to perform scale alignment between different modalities, taking into account the multi-scale features of remote sensing data, and achieving scale alignment between different modalities; feature updates are performed on multiple feature spaces through subspace and global space update networks, and the updated features take into account the irrelevance between different modalities, adding an "irrelevance" perspective to image-text matching, and effectively improving the retrieval accuracy; a similarity matching network is used to simultaneously measure similarity at the global level and multiple scale levels, thereby ensuring the comprehensiveness of similarity matching and improving the accuracy of cross-modal retrieval.

[0015] The specific implementation methods of the present application are further described in detail below in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings are part of this application and are used to provide a further understanding of this application. The illustrative embodiments of this application and their descriptions are used to explain this application, but do not constitute an improper limitation on this application. Obviously, the drawings described below are only some embodiments. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0017] In the drawings of the specification: Figure 1 It is an overall logic schematic diagram of this specific implementation method; Figure 2 It is a logical schematic diagram of the global space mapping module in this specific implementation method. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. The following embodiments are used to illustrate the present application but are not used to limit the scope of the present application.

[0019] What we know is that cross-modal retrieval of remote sensing images and texts refers to a retrieval method in which the input query data (such as remote sensing images) and the output query data (such as remote sensing texts) belong to different modalities. It accurately retrieves the text describing the image based on the remote sensing image or retrieves the corresponding image based on the given text.

[0020] Cross-modal retrieval of remote sensing images and texts is mostly implemented using methods based on deep learning, which is mainly divided into two steps: feature engineering and similarity matching. First, the features of the image and text are mined, and after the saliency modeling of the features of the image and text is performed, similarity matching is performed. In the prior art, the multi-scale features of the image are mined and saliency modeling is performed by constructing a saliency feature extraction module, and the features with high correlation between the image and text are modeled by visually guiding the text. The advantage of this model is that it takes into account the multi-scale features of remote sensing images and the features with high correlation between different modalities. It performs multi-scale modeling of remote sensing images through convolutional neural networks, and then performs saliency modeling on the areas with high correlation between the image and text. However, the existing methods have the following problems: First, the problem of image and text scale alignment is not considered. In the process of remote sensing image scale classification modeling, the large-scale features of the remote sensing image obtained usually contain more information, which often requires multiple words to describe. On the contrary, the small-scale features of the remote sensing image obtained only need to be described with fewer words. The existing methods do not consider the problem of image and text scale alignment, that is, image features of different scales match text features of different scales, resulting in large-scale features of images containing more information cannot be matched with multiple words, and small-scale features of images containing less information cannot be matched with fewer words, which greatly reduces the matching accuracy.

[0021] Second, the irrelevance of different modal data is not considered. In cross-modal retrieval of remote sensing images and texts, data of different modalities have both relevance and irrelevance. Existing cross-modal retrieval methods for remote sensing images and texts only consider the relevance of different modal data and ignore the irrelevance of different modal data. In fact, the relevance and irrelevance between different modal data play an important role in achieving accurate retrieval. Only considering the relevance of different modal data is not robust. For example, there are pictures of "a ship docked at the port" and texts of "a ship sailing near the port" in the data set, and they are not matching pairs. However, the current methods can only consider the relevance between "ships and ships" and "ports and ports", but cannot consider the irrelevance of "docking and sailing", resulting in a high degree of matching between pictures and texts, resulting in incorrect matching. Therefore, the existing methods cannot explore the irrelevance between different modal data, which reduces the accuracy of remote sensing image and text retrieval and greatly limits the performance of remote sensing image and text retrieval models.

[0022] Based on this, in order to achieve scale alignment between images and texts and consider the correlation and irrelevance between different modalities, see Figure 1 , the present application provides a remote sensing image and text retrieval method based on deep learning, the method comprising the following steps: Step S1, constructing a cross-modal multi-scale extraction network; The cross-modal multi-scale extraction network includes at least a large kernel convolution multi-scale image feature extraction module and a multi-scale text feature extraction module. The two feature extraction modules respectively extract the multi-scale image features of the image. Multi-scale text features of text Perform extraction; Step S2, constructing a cross-modal multi-scale alignment network; The cross-modal multi-scale alignment network includes at least a subspace construction module, which uses multi-scale image features Multi-scale text features Cascaded joint text features Provide guidance to obtain text features after scale alignment between different modalities , and use multi-scale image features and text features Constructing subspaces aligned at multiple scales ; Step S3, construct subspace and global space update networks; The subspace and global space update network includes at least a global space mapping module and a subspace mapping module based on correlation and irrelevance. The two mapping modules are respectively in the global space and the constructed multiple subspaces. In the above example, image features and text features are updated; Step S4, constructing a similarity matching network; The similarity matching network includes at least a multi-granularity similarity matching module, which performs similarity matching on updated image features and text features globally and at multiple scale levels; Step S5, combining the networks established in the above steps, inputting the remote sensing image or remote sensing text data to be retrieved, and generating retrieval results based on the similarity matching degree.

[0023] It should be noted that the execution subject of the retrieval method in this embodiment is a remote sensing image and text retrieval device based on deep learning, which can be an electronic device, a component in an electronic device, an integrated circuit, or a chip. The electronic device can be a mobile electronic device or a non-mobile electronic device. Exemplarily, the mobile electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a wearable device, etc., and the non-mobile electronic device can be a server and a personal computer, etc., which are not specifically limited in this application. The following takes the execution subject as an example of a server to describe the remote sensing image and text retrieval method based on deep learning in this embodiment.

[0024] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0025] In this embodiment, in step S1, the large kernel convolution multi-scale image feature extraction module extracts image features through a convolutional neural network. , multiple convolution operations of different sizes are used to further extract multi-scale image features ; Extracting image features It is expressed as: ; Extracting multi-scale image features It is expressed as: ; Where I represents the input image, Represents the parameters of the convolutional neural network CNN, Represents image features of different scales, represents multiple branches used to extract features of images of different scales. represents a 5×5 convolution operation, Indicates different scales, such as large, medium, and small.

[0026] Specifically, for the input image I, the large kernel convolution multi-scale image feature extraction module first uses the convolutional neural network CNN to preliminarily extract the features of the image , three convolution operations of different sizes are used to further extract the multi-scale features of the image .

[0027] Image features Extraction process: The large kernel convolution multi-scale image feature extraction module extracts the features of the input image I through the convolutional neural network CNN .

[0028] Multi-scale image features Extraction process: The large kernel convolution multi-scale image feature extraction module first uses a 5×5 convolution layer to perform preliminary aggregation of image features, and then uses three branches to use convolution kernels of different sizes to extract multi-scale features of images to obtain image features of different scales. , , .

[0029] In this embodiment, the large kernel convolution multi-scale image feature extraction module extracts the image multi-scale features Aggregate into global image features ; Obtaining global image features It is expressed as: ; in, represents a 3×3 convolution operation, Represents a matrix dot product operation.

[0030] Global image features Aggregation process: The multi-scale features of the image are extracted through the large kernel convolution multi-scale image feature extraction module. , , Perform a 3×3 convolution operation and then combine it with the image features initially extracted by the convolutional neural network CNN Perform matrix dot multiplication to obtain global image features that integrate multiple scales .

[0031] In this embodiment, in step S1, the multi-scale text feature extraction module obtains the global text feature extracted by the bidirectional gated recurrent unit GRU. Afterwards, multiple one-dimensional convolutions of different lengths are used to extract text features of different scales; The text features extracted at different scales are expressed as: ; ; ; in, , Indicates one-dimensional convolutions of different sizes to extract multi-scale features of text, , , They represent one-dimensional convolution operations with convolution kernel lengths of 1, 3, and 5 respectively, and L is the convolution kernel length.

[0032] Extracting global text features Process: For the input text T, the multi-scale text feature extraction module first uses the word embedding operation to obtain the word embedding vector E; The word embedding operation is expressed as: ; in, is the word embedding operation, T is the input text, E is the word embedding vector, are the parameters of the bidirectional gated recurrent unit GRU.

[0033] Parameters of the bidirectional gated recurrent unit GRU obtained from the word embedding operation And word embedding vector E, and then extract the global features of the text through the bidirectional gated recurrent unit GRU .

[0034] The bidirectional gated recurrent unit GRU extraction is expressed as: ; Where E is the word embedding vector, are the parameters of the bidirectional gated recurrent unit GRU.

[0035] The process of extracting text features at different scales: extracting multi-scale features of text through three one-dimensional convolutions of different sizes , , global text features extracted based on the multi-scale text feature extraction module Through a one-dimensional convolution operation with a convolution kernel length of 1, the multi-scale features of the text are obtained. , global text features extracted based on the multi-scale text feature extraction module One-dimensional convolution operation with a convolution kernel length of 3 to obtain multi-scale features of the text , global text features extracted based on the multi-scale text feature extraction module One-dimensional convolution operation with a convolution kernel length of 5 to obtain multi-scale features of the text .

[0036] In this embodiment, in step S2, the subspace construction module uses image features of different scales Combined text features Provide guidance to obtain text features after scale alignment between the two modalities , using the scale-aligned text features With multi-scale image features , construct multiple subspaces ; Get scale-aligned text features It is expressed as: ; Construct the subspace It is expressed as: ; Among them, Cat represents the cascade operation, σ represents the sigmoid function, express function, Represents matrix dot product.

[0037] Specifically, the subspace construction module uses multi-scale image features Combined text features Provide guidance, achieve scale alignment between different modalities in this process, and obtain text features after scale alignment between two modalities , and finally use the scale-aligned text features With multi-scale image features Constructing multiple scale aligned subspaces .

[0038] Get text features :By taking the first part and the second part, and then summing them up, we get the first part. The first part uses the sigmoid function to combine the text features Calculate, by Function to combine text features Calculate and perform matrix dot multiplication on the two calculation results; the second part is to use the sigmoid function to combine the text features Calculate and compare with image features Matrix dot multiplication.

[0039] Constructing subspace :Based on the text features obtained above Image features obtained by large kernel convolution multi-scale image feature extraction module , perform cascade operations to obtain multiple subspaces .

[0040] In this embodiment, in step S2, the subspace construction module converts the multi-scale text features Concatenation into joint text features ; Multi-scale text features The cascade is represented as: ; Among them, Cat represents cascade operation, linear represents linear layer operation, , Indicates that one-dimensional convolutions of different sizes extract multi-scale features of text.

[0041] It should be noted that through multi-scale image features The method of guiding text features can achieve scale alignment between the two modalities and construct multiple subspaces. First, the multi-scale text features Concatenation into joint text features , the obtained joint text features All scales of text features are considered.

[0042] See also Figure 1 and Figure 2 In this embodiment, in step S3, the global space mapping module uses the joint feature Global image features and global text features Update them separately to get the final output global image features and global text features ;

[0043] Global image features The update is expressed as: ; Global text features The update is expressed as: ; in, and Represents the sigmoid operation after different linear layers, and represents the tanh operation after different linear layers, Represents a matrix dot product operation.

[0044] Global image features Update process: First, the joint features The sigmoid activation function is used after different linear layers to calculate and combine the features The matrix dot product is calculated by the tanh activation function after different linear layers; second, the joint features pass Function calculation and global image features Perform matrix dot multiplication and sum the first and second calculation results to obtain the updated global image features .

[0045] Global text features Update process: joint features The sigmoid activation function is used after different linear layers to calculate and combine the features The matrix dot product is calculated by the tanh activation function after different linear layers; second, the joint features pass Function calculation and global text features Perform matrix dot multiplication and sum the first and second calculation results to obtain the updated global text feature .

[0046] In this embodiment, the combined feature Global image features and global text features Update them separately to get the final output global image features and global text features ,This update takes into account the correlation and irrelevance among different modalities to achieve a more fine-grained alignment.

[0047] In this embodiment, for the correlation modeling between different modalities, the joint feature After being activated by the sigmoid function as a gate, the features with higher correlation between the two modalities are selected to model the correlation between different modalities. In addition, unlike using the attention mechanism to find the correlation between the two modalities, in order to model the irrelevance between different modalities, the present invention uses Modeling the irrelevance between different modalities: Modeling irrelevance adds a new perspective to image-text matching and helps improve retrieval accuracy.

[0048] See also Figure 2 In this embodiment, in step S3, the global space mapping module uses the global image features and global text features Constructing joint features , expressed as: ; Among them, linear represents linear layer operation, and Cat represents cascade operation.

[0049] In this embodiment, in step S3, the subspace mapping module uses multi-scale joint features Multi-scale image features and multi-scale text features Update to obtain the final output multi-scale image features and multi-scale text features , Multi-scale image features The update is expressed as: ; Multi-scale image features The update is expressed as: ; in, and Represents the sigmoid operation after different linear layers, and represents the tanh operation after different linear layers, Represents a matrix dot product operation.

[0050] Specifically, the subspace mapping module includes subspace mapping 1, subspace mapping 2 and subspace mapping 3. The subspace mapping module performs mapping considering correlation and irrelevance in multiple subspaces constructed by the subspace construction module. It can be understood that the subspace mapping module uses multi-scale joint features Multi-scale image features and multi-scale text features An update that takes into account the correlation and irrelevance between different modalities is performed to achieve a more fine-grained alignment.

[0051] Multi-scale image features Update process: First, multi-scale joint features The sigmoid activation function is used after different linear layers to calculate and combine the multi-scale joint features The matrix dot product is calculated by the tanh activation function after different linear layers; second, the multi-scale joint features pass Function computation and multi-scale image features Perform matrix dot multiplication and sum the first and second calculation results to obtain the updated multi-scale image features .

[0052] Multi-scale image features Update process: First, multi-scale joint features The sigmoid activation function is used after different linear layers to calculate and combine the multi-scale joint features The matrix dot product is calculated by the tanh activation function after different linear layers; second, the multi-scale joint features pass Function computation and multi-scale image features Perform matrix dot multiplication and sum the first and second calculation results to obtain the updated multi-scale text features .

[0053] In this embodiment, in step S4, the similarity matching module not only uses the global image features and global level text features In addition to similarity matching, multi-scale similarity matching is also added, including small-scale image features and text features , mesoscale image features and text features And large-scale image features and text features .

[0054] Specifically, based on the global image features updated in step S3 and global text features Perform global similarity calculation based on the multi-scale image features updated in step S3 and multi-scale text features Perform multi-scale similarity calculation and output retrieval results based on the similarity scores.

[0055] In this embodiment, the loss function is a triple loss: ; The first part of the loss function is based on finding text in an image. Represents the similarity between the image and the corresponding positive sample, Indicates the similarity between the image and the negative sample. The second part of the loss function is based on the situation of finding images based on text. Represents the similarity between the text and its corresponding positive sample, Represents the similarity between text and negative samples.

[0056] Based on the same inventive concept, the present application also provides a remote sensing image and text retrieval system based on deep learning, which includes: The cross-modal multi-scale extraction module includes a large kernel convolution multi-scale image feature extraction module and a multi-scale text feature extraction module. The two feature extraction modules extract the multi-scale features of the image and the multi-scale features of the text at the same time. A cross-modal multi-scale alignment module includes at least a subspace construction module, which guides the joint text features through image features of different scales to obtain text features after scale alignment between different modalities, and constructs subspaces after multiple scale alignment using image features and text features; Subspace and global space update modules, the subspace and global space update modules at least include a global space mapping module and a subspace mapping module based on correlation and irrelevance, the two mapping modules are respectively in the global space and the constructed multiple subspaces In the above example, image features and text features are updated; The similarity matching module includes at least a multi-granularity similarity matching module, and the multi-granularity similarity matching module performs similarity matching on updated image features and text features globally and at multiple scale levels.

[0057] Based on the same inventive concept, the present application also provides a remote sensing image and text retrieval device based on deep learning, including a processor and a memory, wherein a computer program is stored in the memory. When the computer program is executed by the processor, the remote sensing image and text retrieval method based on deep learning as described above is implemented.

[0058] Based on the same inventive concept, the present application also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the remote sensing image and text retrieval method based on deep learning as described above.

[0059] The program product of the present application for implementing the above method may adopt a portable compact disk read-only memory and include program code, and may be run on a terminal device, such as a personal computer. However, the program product of the present application is not limited thereto. In the present application, a readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, apparatus, or device.

[0060] It should be noted that a computer-readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program code contained on the readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.

[0061] The above are only preferred embodiments of the present application, and are not intended to limit the present application in any form. Although the present application has been disclosed as above with preferred embodiments, it is not intended to limit the present application. Any technician familiar with this patent can make some changes or modifications to equivalent embodiments of equivalent changes using the above-mentioned technical contents without departing from the scope of the technical solution of the present application. The implementation schemes in the above embodiments can also be further combined or replaced. However, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present application without departing from the content of the technical solution of the present application are still within the scope of the solution of the present application.

Claims

1. A remote sensing image and text retrieval method based on deep learning, characterized in that: The method comprises the following steps: Step S1, constructing a cross-modal multi-scale extraction network; The cross-modal multi-scale extraction network at least includes a large kernel convolution multi-scale image feature extraction module and a multi-scale text feature extraction module. The two feature extraction modules respectively extract the multi-scale image features of the image. Multi-scale text features of text Perform extraction; Step S2, constructing a cross-modal multi-scale alignment network; The cross-modal multi-scale alignment network at least includes a subspace construction module, and the subspace construction module uses multi-scale image features to Multi-scale text features Cascaded joint text features Provide guidance to obtain text features after scale alignment between different modalities , and use multi-scale image features and text features Constructing subspaces aligned at multiple scales ; Step S3, construct subspace and global space update networks; The subspace and global space update network at least includes a global space mapping module and a subspace mapping module based on correlation and irrelevance, and the two mapping modules are respectively in the global space and the constructed multiple subspaces. In the above example, image features and text features are updated; Step S4, constructing a similarity matching network; The similarity matching network at least includes a multi-granularity similarity matching module, and the multi-granularity similarity matching module performs similarity matching on updated image features and text features at a global level and multiple scale levels; Step S5, combining the networks established in the above steps, inputting the remote sensing image or remote sensing text data to be retrieved, and generating retrieval results based on the similarity matching degree.

2. The remote sensing image and text retrieval method based on deep learning according to claim 1, characterized in that: In step S1, the large kernel convolution multi-scale image feature extraction module extracts image features through a convolutional neural network , multiple convolution operations of different sizes are used to further extract multi-scale image features ; Multi-scale image features and image features Aggregate into global image features ; Obtaining global image features It is expressed as: ； in, represents a 3×3 convolution operation, represents the matrix dot multiplication operation, Represents image features extracted by convolutional neural network.

3. The remote sensing image and text retrieval method based on deep learning according to claim 2 is characterized in that: In step S1, the multi-scale text feature extraction module extracts global text features through a bidirectional gated recurrent unit GRU , after obtaining the global text features Afterwards, multiple one-dimensional convolutions of different lengths are used to extract text features of different scales to obtain multi-scale text features. .

4. The remote sensing image and text retrieval method based on deep learning according to claim 3 is characterized in that: In step S2, the subspace construction module converts the multi-scale text features Concatenation into joint text features , the subspace construction module uses multi-scale image features Combined text features Provide guidance to obtain text features after scale alignment between the two modalities , using the scale-aligned text features With multi-scale image features , construct multiple subspaces ; Get scale-aligned text features It is expressed as: ; Construct the subspace It is expressed as: ； Among them, Cat represents the cascade operation, σ represents the sigmoid function, express function, Represents matrix dot product.

5. The remote sensing image and text retrieval method based on deep learning according to claim 4 is characterized in that: In step S3, the subspace mapping module uses the subspace Multi-scale image features and multi-scale text features Update to obtain the final output multi-scale image features and multi-scale text features ; Multi-scale image features The update is expressed as: ； Multi-scale image features The update is expressed as: ； in, and Represents the sigmoid operation after different linear layers, and represents the tanh operation after different linear layers, Represents a matrix dot product operation.

6. The remote sensing image and text retrieval method based on deep learning according to claim 5, characterized in that: In step S3, the global space mapping module uses global image features and global text features Constructing joint features The global spatial mapping module uses the joint features of the global features of the image and text Global image features and global text features Update them separately to get the final output global image features and global text features ; Global image features The update is expressed as: ； Global text features The update is expressed as: ； in, and Represents the sigmoid operation after different linear layers, and represents the tanh operation after different linear layers, Represents a matrix dot product operation.

7. The remote sensing image and text retrieval method based on deep learning according to claim 6, characterized in that: In step S3, the global space mapping module uses global image features and global text features Constructing joint features It is expressed as: ； Among them, linear represents linear layer operation, and Cat represents cascade operation.

8. A remote sensing image and text retrieval system based on deep learning, characterized by: The system comprises: The cross-modal multi-scale extraction module includes a large kernel convolution multi-scale image feature extraction module and a multi-scale text feature extraction module. The two feature extraction modules respectively extract the multi-scale image features of the image. Multi-scale images and text Extract features; The cross-modal multi-scale alignment module comprises at least a subspace construction module, wherein the subspace construction module uses multi-scale image features to Multi-scale text features Cascaded joint text features Provide guidance to obtain text features after scale alignment between different modalities , and use multi-scale image features and text features Constructing subspaces aligned at multiple scales ; Subspace and global space update modules, the update module at least includes a subspace mapping module based on correlation and irrelevance and a global space mapping module, the two mapping modules are respectively in the global space and the constructed multiple subspaces In the above example, image features and text features are updated; The similarity matching module at least includes a multi-granularity similarity matching module, and the multi-granularity similarity matching module performs similarity matching on updated image features and text features globally and at multiple scale levels.

9. A remote sensing image and text retrieval device based on deep learning, characterized in that: It includes a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the remote sensing image and text retrieval method based on deep learning as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it implements the remote sensing image and text retrieval method based on deep learning as described in any one of claims 1 to 7.

Images