Remote sensing image retrieval method and device based on hierarchical semantic feature fusion

Abstract

The invention discloses a remote sensing image retrieval method and device based on hierarchical semantic feature fusion, and relates to the technical field of remote sensing. And constructing an off-line feature library based on the satellite image. Real-time aerial images of the unmanned aerial vehicle are received, global feature vectors are obtained through the feature extraction network, dimensionality reduction is carried out through principal component analysis, and the proportion of first three-dimensional feature active areas is calculated to measure an effective semantic information area. And acquiring an attention score by using an attention mechanism, and calculating a normalized attention mean value. And synthesizing the proportion of the front three-dimensional feature active region and the normalized attention mean value, constructing a comprehensive searchable evaluation index, and screening a searchable image. And for the retrievable image, returning a preset number of candidate results according to a sequence by using the cosine similarity between the global feature vector of the retrievable image and the global feature vector of the offline feature library, associating geographic position information, and returning a retrieval result list. The problems of insufficient accuracy and low efficiency in the existing remote sensing image retrieval technology are solved.

Description

Technical Field

[0001] The present application relates to the field of remote sensing technology, and in particular to a remote sensing image retrieval method based on hierarchical semantic feature fusion. Background Art

[0002] Remote sensing image retrieval technology plays a crucial role in drone navigation missions. By comparing remote sensing images captured by drones with pre-stored image databases, this technology enables rapid and accurate target identification and positioning, providing a reliable position reference for autonomous navigation. However, in practice, the complex and variable environments in which remote sensing images are collected can lead to poor image quality in some areas, severely impacting retrieval accuracy and efficiency.

[0003] Low-quality remote sensing imagery is a key factor limiting retrieval performance. Due to adverse acquisition environments or geographical limitations, these images exhibit insufficient texture information, severe occlusions, and dramatic lighting variations. Furthermore, shadows from trees, buildings, and obstructions from drone structures encountered during flight can cause information loss in localized areas, impacting the overall representational capabilities of the image. Furthermore, variations in lighting conditions at different acquisition times can compromise feature consistency, increasing matching errors during retrieval.

[0004] Therefore, how to solve the problems of insufficient retrieval accuracy and low efficiency in existing remote sensing image retrieval technology and provide a more reliable location reference for autonomous navigation of UAVs has become an urgent problem to be solved in current remote sensing image retrieval technology. Summary of the Invention

[0005] In the embodiments of the present application, a remote sensing image retrieval method based on hierarchical semantic feature fusion is provided to solve the problems of insufficient retrieval accuracy and low efficiency in existing remote sensing image retrieval technologies.

[0006] In the first aspect, an embodiment of the present application provides a remote sensing image retrieval method based on hierarchical semantic feature fusion, the method comprising: constructing an offline feature library based on satellite images, and realizing online real-time retrieval of aerial images based on the offline feature library; the online real-time retrieval comprises: receiving aerial images acquired by a drone in real time, performing feature extraction on the aerial images using a feature extraction network, and obtaining a global feature vector; performing principal component analysis and dimensionality reduction on the extracted global feature vector, and obtaining the proportion of active areas of the first 3-dimensional features after dimensionality reduction; wherein the proportion of active areas of the first 3-dimensional features is used to measure the proportion of areas in the aerial image that contain valid semantic information, and the active area refers to the need for the image block to enter a preset threshold in at least one feature direction; in the feature extraction process, the attention mechanism of the feature extraction network is utilized to obtain the attention scores of the aerial image on different attention heads; the attention scores on different attention heads are obtained based on the attention scores on the attention heads. The attention intensity mean is calculated and normalized to obtain the normalized attention mean to represent the attention distribution characteristics of the aerial image in the feature extraction network; the proportion of the active area of the first three-dimensional features and the normalized attention mean are combined to construct a comprehensive retrievability evaluation index; if the comprehensive retrievability evaluation index is greater than the attention intensity threshold, the aerial image is marked as retrievable, otherwise, it is marked as non-retrievable; for the aerial image marked as retrievable, the cosine similarity between the global feature vector of the aerial image marked as retrievable and the global feature vector of the offline feature library is used to return a preset number of candidate results in descending order; each candidate result is associated with the geographic location information in the offline feature library to locate the satellite image most similar to the aerial image; a retrieval result list containing a set of preset number of candidate results is returned, and each retrieval result includes matching geographic location information and cosine similarity.

[0007] In one possible implementation, constructing an offline feature library based on satellite images includes: establishing a geographic coordinate system for the satellite image and dividing the satellite image into multiple satellite image blocks; recording geographic location information of each satellite image block in the satellite image in which it is located, including a grid index, pixel coordinates, and geographic coordinates; preprocessing the satellite image blocks; extracting features from the preprocessed satellite image blocks using a feature extraction network, where the extracted features include a global feature vector and a multi-head attention distribution score of the offline feature library; organizing the extracted global feature vector, multi-head attention distribution score, and geographic location information of the offline feature library into structured feature data, and storing the data in the offline feature library; and associating the stored global feature vector with the geographic location information using an index structure based on the global feature vector of the offline feature library.

[0008] In a possible implementation, the active area ratio of the first three-dimensional features and the normalized attention mean are integrated to construct a comprehensive retrievability evaluation index, including: the expression of the comprehensive retrievability evaluation index is: CRS = α·NZR + β·ASborm ; Among them, CRS is the comprehensive retrievability evaluation index, NZR is the ratio of the active area of the first three-dimensional features, α and β are weighting coefficients, and α+β=1. AS is the mean attention value, AS borm is the normalized mean attention value.

[0009] In one possible implementation, the expression for determining the attention intensity threshold is: T = μ - k·σ; wherein T is the attention intensity threshold, k is a hyperparameter, μ is the mean of the global attention intensity of all aerial image blocks in the aerial image, and σ is the standard deviation of the global attention intensity of all aerial image blocks in the aerial image.

[0010] In one possible implementation, the global attention strength of the aerial image block is obtained by utilizing the attention mechanism in the feature extraction network to obtain the attention weight of the aerial image block on the multi-head attention head on the last block of the Transformer, and obtaining the global attention strength of the aerial image block; the expression for obtaining the attention weight of the aerial image block on the multi-head attention head on the last block of the Transformer is: Among them, N is the number of image blocks into which the aerial image is divided, A h is the attention weight of the hth attention head on the last block of the aerial image block, h is the index of the attention head, i is the index of the aerial image block, is the attention score of the attention weight on the i-th image block in the h-th attention head of the last layer; the expression for obtaining the global attention strength of the aerial image block is: Among them, A global is the global attention strength of the aerial image block, L is the last block of Transformer, and H is the number of multi-head attention heads.

[0011] In a possible implementation, the expression of the global feature vector of the aerial image marked as retrievable is: f I =Φ(I;θ); where Φ(·) represents the feature extraction network, θ is the network parameter, and f I ∈R d is the extracted D-dimensional searchable global feature vector of the aerial image, R D is a D-dimensional real vector space.

[0012] In one possible implementation, the cosine similarity between the global feature vector of the aerial image marked as searchable and the global feature vector of the offline feature library is used to return a preset number of candidate results in descending order, including: the calculation formula of the cosine similarity is: Among them, f j represents the jth feature vector in the offline feature library, S(f I ,f j ) is the cosine similarity between the jth feature vector in the offline feature library and the global feature vector of the searchable aerial image; the expression for returning a preset number of candidate results in descending order is: TopN′={j1,j2,…,j N}, Among them, TopN′ is a set of N′ candidate results, j1, j2, …, j N is the index of the candidate result in the offline feature library, and N is the total number of candidate results.

[0013] In a possible implementation, associating each candidate result with the geographic location information in the offline feature library to locate the satellite image most similar to the aerial image includes: associating each candidate result with the geographic location information in the offline feature library is expressed as follows: Location(j i )=(longitude i ,latitude i ); where j i is the index of the i-th candidate result after sorting, Location(j i ) is the candidate result j i The corresponding geographic location information, (longitude i ,latitude i ) is the candidate result j i Longitude and latitude coordinates, longitude i is the candidate result j i longitude, latitude i is the candidate result j i The satellite image corresponding to the first candidate result after descending sorting is the satellite image most similar to the aerial image.

[0014] In the second aspect, an embodiment of the present application provides a remote sensing image retrieval device based on hierarchical semantic feature fusion, including: a construction module for constructing an offline feature library based on satellite images, and realizing online real-time retrieval of aerial images based on the offline feature library; a global feature vector acquisition module for receiving aerial images acquired by a drone in real time, and using a feature extraction network to extract features from the aerial images to obtain global feature vectors; a first 3-dimensional feature active area ratio acquisition module for performing principal component analysis dimensionality reduction on the extracted global feature vectors, and obtaining the first 3-dimensional feature active area ratio after dimensionality reduction; wherein, the first 3-dimensional feature active area ratio is used to measure the area ratio containing valid semantic information in the aerial image, and the active area refers to the image block that needs to enter a preset threshold in at least one feature direction; an attention score acquisition module for obtaining the attention score of the aerial image on different attention heads using the attention mechanism of the feature extraction network during the feature extraction process; an attention mean acquisition module for obtaining the attention score based on the attention scores on different attention heads. The mean of attention strength is calculated and normalized to obtain the normalized mean of attention to represent the attention distribution characteristics of the aerial image in the feature extraction network; a comprehensive retrievability evaluation index module is constructed to integrate the active area ratio of the first three-dimensional features and the normalized mean of attention to construct a comprehensive retrievability evaluation index; a marking module is used to mark the aerial image as retrievable if the comprehensive retrievability evaluation index is greater than the attention strength threshold, otherwise, it is marked as non-retrievable; a cosine similarity acquisition module is used to use the cosine similarity between the global feature vector of the aerial image marked as retrievable and the global feature vector of the offline feature library for the aerial image marked as retrievable, and return a preset number of candidate results in descending order; a positioning module is used to associate each candidate result with the geographic location information in the offline feature library to locate the satellite image most similar to the aerial image; a return module is used to return a retrieval result list containing a set of preset number of candidate results, each retrieval result includes matching geographic location information and cosine similarity.

[0015] One or more technical solutions provided in the embodiments of this application have at least the following technical effects:

[0016] The embodiment of the present application provides a remote sensing image retrieval method based on hierarchical semantic feature fusion. The method uses a deep feature extraction network to extract features from aerial images acquired by a drone in real time, effectively captures the semantic information in the aerial images, and generates a high-dimensional global feature vector. The feature vector is reduced in dimension by principal component analysis, which not only reduces the amount of calculation, but also measures the proportion of areas containing effective semantic information in the aerial image by obtaining the proportion of active areas of the first three-dimensional features in the global feature vector after dimensionality reduction, providing an important basis for subsequent retrievability evaluation. In the feature extraction process, an attention mechanism is introduced to obtain the attention scores of the aerial image on different attention heads, reflecting the degree of attention of different areas in the aerial image. The attention mean is calculated based on the attention score and normalized to obtain the normalized attention mean, which accurately represents the attention distribution characteristics of the aerial image in the feature extraction network. The proportion of active areas of the first three-dimensional features and the normalized attention mean are combined to construct a comprehensive retrievability evaluation index, which effectively evaluates the retrievability of the aerial image and improves the accuracy and efficiency of the retrieval. By setting an attention intensity threshold, aerial images are filtered and only searchable images are marked as searchable, avoiding interference from low-quality or invalid images and improving retrieval accuracy. For aerial images marked as searchable, the cosine similarity between the global feature vector of the marked aerial image and the global feature vector in the offline feature library is used to sort and return a preset number of candidate results in descending order. Each candidate result is associated with the geolocation information in the offline feature library to accurately locate the satellite image most similar to the aerial image, providing a reliable geolocation reference for the drone's autonomous navigation. A search result list containing a set of the preset number of candidate results is returned, each including the matching geolocation information and cosine similarity, allowing users to clearly understand the accuracy and relevance of the search results. The entire online real-time retrieval process is efficient and fast, meeting the requirements of real-time drone navigation. This improves the retrieval system's robustness to low-quality or complex imagery, ensuring the stability and reliability of retrieval results. This addresses the shortcomings of existing remote sensing image retrieval technologies, which often suffer from insufficient accuracy and low efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments of the present application or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 A flowchart of a remote sensing image retrieval method based on hierarchical semantic feature fusion provided in an embodiment of the present application;

[0019] Figure 2 A schematic diagram of the process of a remote sensing image retrieval method based on hierarchical semantic feature fusion provided in an embodiment of the present application;

[0020] Figure 3 A schematic diagram of a partially filtered aerial image provided in an embodiment of the present application;

[0021] Figure 4 A schematic diagram of the top 3 search results obtained using the search method of the present application provided in an embodiment of the present application;

[0022] Figure 5 A schematic diagram of the top 5 search results obtained using the search method of the present application provided in an embodiment of the present application;

[0023] Figure 6 A schematic diagram of the top 10 search results obtained using the search method of the present application provided in an embodiment of the present application;

[0024] Figure 7 A schematic diagram of the retrieval effect of the Top 3 search results provided in an embodiment of the present application under multiple light sources;

[0025] Figure 8 A schematic diagram of the retrieval effect of the Top 5 search results provided in an embodiment of the present application under multiple light sources;

[0026] Figure 9 A schematic diagram of the retrieval effect of the Top 10 search results provided in an embodiment of the present application under multiple light sources;

[0027] Figure 10 A schematic diagram of a remote sensing image retrieval device based on layered semantic feature fusion provided in an embodiment of the present application;

[0028] Figure 11 A schematic diagram of a remote sensing image retrieval server based on hierarchical semantic feature fusion provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0030] The following description of some of the technologies involved in the embodiments of this application is provided to facilitate understanding and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of this application. Similarly, for the sake of clarity and conciseness, some descriptions of well-known functions and structures are omitted from the following description.

[0031] The present application embodiment provides a remote sensing image retrieval method based on layered semantic feature fusion, such as Figure 1 As shown, the method includes steps S101 to S111. Figure 1 This is only an execution order shown in the embodiment of the present application, and does not represent the only execution order of a remote sensing image retrieval method based on layered semantic feature fusion. Figure 1 The steps shown may be performed in parallel or reversed.

[0032] S101: Build an offline feature library based on satellite images, and implement online real-time retrieval of aerial images based on the offline feature library.

[0033] An offline feature library is constructed based on satellite imagery, including: establishing a geographic coordinate system for the satellite imagery and dividing the imagery into multiple satellite image blocks. The geographic location information of each satellite image block within its satellite imagery is recorded, including its grid index, pixel coordinates, and geographic coordinates. The satellite image blocks are preprocessed. A feature extraction network is used to extract features from the preprocessed satellite image blocks. The extracted features include the global feature vector and multi-head attention distribution score of the offline feature library. The global feature vector, multi-head attention distribution score, and geographic location information of the extracted offline feature library are organized into structured feature data and stored in the offline feature library.

[0034] An index structure of a global feature vector based on an offline feature library is used to associate the stored global feature vector with geographic location information.

[0035] Specifically, the aerial image in this application is a type of remote sensing image.

[0036] Furthermore, the preprocessing of satellite image blocks includes the following steps. 1. Establishment of geographic coordinate system. A geographic coordinate reference system is constructed for large satellite images to ensure an accurate mapping relationship between image pixels and actual geographic locations. In actual operation, this is usually achieved by reading the metadata file or geographic tag information of the satellite image, and then establishing a conversion relationship between longitude and latitude and pixel coordinates. 2. Grid division. Based on actual application requirements, such as retrieval accuracy requirements and computing resource limitations, large satellite images are divided into regular grids. The grid is generally divided at equal intervals, for example, each grid block corresponds to an area of 500m×500m or 1km×1km on the ground. 3. Image cropping. Based on the grid division results, the large image is cropped into satellite image blocks of the same size, and each satellite image block is used as the basic unit for feature extraction and retrieval. The original resolution of the image is maintained during the cropping process to ensure that surface details are not lost. 3. Coordinate recording. For each cropped satellite image block, record its location information in the original satellite image, including the grid index (row and column number), pixel coordinates (upper left and lower right corner coordinates), and corresponding geographic coordinates (latitude and longitude range). These coordinate information will be stored together with the feature vector extracted subsequently so that the retrieval results can be mapped back to the actual geographic location. 4. Image normalization. Perform basic color normalization and size adjustment on the cropped satellite image blocks to make them suitable for the input requirements of the feature extraction network. Specific operations may include adjusting contrast, brightness, and scaling the image to the network's expected input size, such as 512×512 pixels.

[0037] The preprocessed satellite imagery is processed through a feature extraction network. This application adopts a lightweight ViT model, which is optimized through a two-stage training method (self-supervised large model distillation and feature orthogonality training), which not only maintains a strong feature representation capability, but also adapts to computing resource constraints. As a model that migrates the successful Transformer architecture in the field of natural language processing to the field of computer vision, Vision Transformer (ViT) has shown unique advantages in extracting semantic features from remote sensing images. The core idea is to regard the image as a series of discrete visual token sequences, and capture the global dependencies between these tokens through a self-attention mechanism. Unlike traditional CNNs that extract features through local receptive fields and hierarchical features, ViT can directly model the relationship between any two image regions, providing a new perspective for extracting complex surface semantic information.

[0038] Specifically, for each satellite image patch, the following operations are performed: A global feature vector is extracted to capture overall semantic information. A multi-head attention distribution score is extracted to record the importance weights of key regions. The actual coordinates corresponding to the top-left and bottom-right corners of the satellite image patch are retained as geographic location information.

[0039] The online real-time search includes steps S102 to S111.

[0040] S102: Receive aerial images acquired by the drone in real time, use a feature extraction network to extract features from the aerial images, and obtain a global feature vector.

[0041] It should be noted that the feature extraction network used here can also be Vision Transformer (ViT).

[0042] S103: Perform principal component analysis on the extracted global feature vector to reduce its dimensionality and obtain the ratio of the active regions of the first three-dimensional features after dimensionality reduction. The ratio of the active regions of the first three-dimensional features is used to measure the proportion of regions in the aerial image that contain valid semantic information. The active region refers to the image block that needs to enter the preset threshold in at least one feature direction.

[0043] The extracted global feature vector is patchtoken, with a dimension of (196,384). The preset threshold before entry is to enter the top 150, that is, the preset threshold is 150.

[0044] Specifically, assuming that the size of the aerial image is H×W×C, the aerial image is first divided into N aerial image blocks, each of which has a size of P×P. The number of aerial image blocks is calculated according to the formula: Where N is the number of aerial image blocks into which the aerial image is divided. The matrix composed of each aerial image block is input into the feature extraction network. Assume that the feature vector of each image block is x i ∈R D , all eigenvectors form a characteristic matrix X: X=[x1,x2,…,x N ]∈R N ×D . x1,x2,…,x N is the index of the eigenvector of each image block, N×D is the dimension of the feature matrix X, D is the dimension of each eigenvector, that is, each eigenvector is a D-dimensional real vector. This feature matrix X is used as the input for the subsequent PCA (principal component analysis) dimensionality reduction, which represents the local information of different areas in the aerial image. Given the extracted feature matrix X, the principal component analysis method is used for dimensionality reduction. Assuming that the first n principal components are selected for dimensionality reduction, the projection matrix V∈R N×D Then project the feature matrix X onto these principal components to obtain the reduced dimension feature matrix X pca .X pca = XV. The features of each aerial image block are reduced from D dimensions to n dimensions, and the reduced dimension feature matrix X with the shape of (N, n) is obtained. pca .X pca ∈R N×nIn order to further filter out the foreground area of the aerial image, the background and foreground are separated based on the features after PCA dimensionality reduction. Specifically, the feature matrix X after dimensionality reduction can be calculated. pca The first 150 largest eigenvalues on each principal component are used to identify the corresponding aerial image blocks as foreground. If an aerial image block belongs to the foreground on at least one principal component, it is classified as foreground; otherwise, it is classified as background. Specifically, the proportion of the foreground in the entire aerial image is called the proportion of the active area of the first three-dimensional features. It is considered that the corresponding image contains valid semantic information. Only the foreground aerial image blocks are subjected to secondary PCA dimensionality reduction to further optimize the feature representation. Ultimately, a more refined foreground feature representation is obtained, avoiding interference from background areas and making the retrievability evaluation more accurate.

[0045] The feature matrix X after dimensionality reduction based on principal component analysis pca , this application proposes a retrievability evaluation method based on the ratio of active areas of the first 3-dimensional features. In the feature space after dimensionality reduction, the active areas of the first 3-dimensional features usually correspond to the parts of the image that contain valid semantic information, which are crucial for image retrieval; after visual analysis, the semantic information of the background area is messy, while the semantic information of the foreground area is clear, usually coherent texture information, so the ratio of active areas of the first 3-dimensional features can be used to measure the retrievability of the image. Define the foreground feature matrix of the foreground aerial image block in the feature matrix after dimensionality reduction based on principal component analysis, and calculate its ratio of active areas of the first 3-dimensional features NZR. The foreground area refers to the number of image blocks that are active on at least one principal component, and the total image area refers to the total number of image blocks. Calculating the NZR quantifies the proportion of areas in aerial images that contain valid semantic information, providing an important basis for measuring the searchability of aerial images. The foreground and background are artificially defined. Visual analysis shows that foreground areas typically contain significant texture information, while background areas are more diffuse.

[0046] S104: During the feature extraction process, the attention mechanism of the feature extraction network is used to obtain the attention scores of the aerial image on different attention heads.

[0047] Specifically, for a multi-head attention mechanism with h heads, three different sets of weight matrices Q are defined for each head: i , K i and V i : Where i=1,…h, h is the index of the attention head, and is the linear transformation weight matrix used by the i-th attention head to generate query, key, and value.

[0048] d model is the dimension of the input feature, d k and d v The dimensions of the query / key and value of each header, respectively, usually set to d k =d v =d model / h, for the input feature matrix Each attention head first computes its corresponding query, key, and value matrices. Q i , K i and V i are the query matrix, key matrix, and value matrix of the i-th attention head respectively. Next, each attention head independently calculates its attention output head i .

[0049] in, The calculated result is the attention weight matrix, which is used to calculate the value matrix V i Perform weighted summation. Contains the feature representation after attention weighting. Finally, the outputs of all heads are concatenated and passed through an additional linear transformation W O Get the final multi-head attention output Multihead(X). Multihead(X)=Concat(head1,head2,…,head h )W O Among them, Concat is the concatenation operation head1, head2,…, head h The index of the attention output of each attention head. The multi-head attention mechanism demonstrates its powerful capabilities when processing aerial imagery with the Vision Transformer (ViT). It enables the model to simultaneously focus on different types of surface features and spatial relationships. For example, some heads may focus on texture features to capture the texture details of the surface in the image; while other heads may focus on color distribution or geometric shapes to understand the color patterns and geometric structures of different regions in the image. This multi-dimensional feature extraction capability is crucial for the accurate understanding and matching of aerial imagery, especially when dealing with complex situations such as changes in perspective and lighting conditions, which can help the model understand the image content more comprehensively.

[0050] The attention score matrix is the key to understanding the working principle of the self-attention mechanism. It not only reflects the calculation process within the model, but also contains rich semantic information. In mathematical representation, each element A of the attention score matrix A is ijRepresents the degree of attention paid by the i-th token to the j-th token. Tokens are the basic units of data processing and model input, used to convert raw data into a format that the model can understand and process. This value quantifies the strength of the model's belief in the correlation between two locations, with larger values indicating a stronger association. Specifically, in the context of ViT processing aerial imagery, these tokens correspond to the individual aerial image patches and the additional cls-token tag (a type of label). Therefore, the attention score actually reflects the strength of the semantic association between different regions of the aerial image. This semantic association is intuitively interpretable. When visualizing the attention score matrix, high-value areas often reveal semantically related parts of the image. For example, in an aerial image of an airport, patches representing the runway and the terminal often have high attention scores, even though these patches may be spatially far apart. This indicates that the model has learned to recognize that these regions collectively constitute the semantic concept of "airport," rather than relying solely on simple spatial proximity. The attention distribution of the cls-token tag is particularly noteworthy. In ViT, the cls-token tag is designed to aggregate global information across the entire image. Its attention distribution directly reveals which areas the model considers most important for overall image understanding. By analyzing the attention scores between the cls-token and each patch, key semantic regions in the image can be identified. For example, in urban remote sensing imagery, the cls-token tag typically assigns higher attention scores to highly discriminative areas such as landmark buildings, major road intersections, and unique terrain. These attention scores provide a deep understanding of how the model understands remote sensing imagery, providing strong support for subsequent aerial image retrieval and analysis.

[0051] S105: Obtain an attention mean based on the attention scores of different attention heads, and perform normalization processing to obtain a normalized attention mean to represent the attention distribution characteristics of the aerial image in the feature extraction network.

[0052] S106: Construct a comprehensive retrievability evaluation index by combining the active area ratio of the first three-dimensional features and the normalized attention mean.

[0053] The comprehensive retrievability evaluation index is constructed by integrating the active area ratio of the first three-dimensional features and the normalized attention mean, including: The expression of the comprehensive retrievability evaluation index is: CRS = α·NZR + β·AS norm Among them, CRS is the comprehensive retrievability evaluation index, NZR is the ratio of the active area of the first three-dimensional features, α and β are weighting coefficients, and α+β=1. AS is the mean attention value, ASnorm is the normalized mean attention value.

[0054] S107: If the comprehensive retrievability evaluation index is greater than the attention intensity threshold, the aerial image is marked as retrievable; otherwise, it is marked as non-retrievable.

[0055] The expression for determining the attention strength threshold is: T = μ - k · σ. Where T is the attention strength threshold, k is a hyperparameter, μ is the mean of the global attention strength of all aerial image patches in the aerial image, and σ is the standard deviation of the global attention strength of all aerial image patches in the aerial image.

[0056] Specifically, the above can be understood as a retrievability evaluation mechanism. When the comprehensive retrievability evaluation index value is greater than the attention intensity threshold, the aerial image is marked as retrievable and allowed to proceed to the subsequent feature extraction stage. Conversely, if the comprehensive retrievability evaluation index value is less than or equal to the attention intensity threshold, the aerial image is marked as non-retrievable and the retrieval process is terminated at the current stage, avoiding unnecessary consumption of computing resources.

[0057] After extensive comparative experiments, we found that a setting of T = 0.5 achieves the optimal balance between retrieval accuracy and computational efficiency in various application scenarios. When T is lower than 0.5, there is a tendency to retain too many low-quality images. While these images are numerous, their contribution to improving retrieval accuracy is limited, and they actually increase the computational burden. When T is higher than 0.5, the screening criteria become overly stringent, potentially leading to the inadvertent deletion of aerial images that actually contain valid information, compromising the comprehensiveness of the retrieval. Therefore, a threshold of 0.5 is recommended for attention intensity.

[0058] Further analysis shows that aerial images with low attention intensity thresholds often feature large, homogeneous areas (such as water surfaces and deserts) or highly repetitive texture features (such as regular farmland and homogeneous residential areas). These types of images are prone to ambiguity during feature matching, leading to erroneous retrieval results. Therefore, implementing the aforementioned retrievability assessment mechanism effectively identifies and eliminates these unsuitable images early in the processing process, avoiding wasting computational resources on low-value targets and significantly improving the overall system's retrieval accuracy and computational efficiency.

[0059] The global attention strength of the aerial image block is obtained by using the attention mechanism in the feature extraction network to obtain the attention weight of the aerial image block on the multi-head attention head on the last block of the Transformer, and then obtaining the global attention strength of the aerial image block.

[0060] The expression for obtaining the attention weight of the aerial image block on the multi-head attention head of the last block of Transformer is: Among them, N is the number of image blocks into which the aerial image is divided, A h is the attention weight of the hth attention head on the last block of the aerial image block, h is the index of the attention head, i is the index of the aerial image block, is the attention score of the h-th attention head in the last layer on the i-th image block.

[0061] Specifically, the attention weight is the cls-token attention weight. The multi-head attention is the output of the last block of the Transformer. For multi-head attention, the feature dimension extracted is (6,384), representing 6 heads, and the feature dimension of each head is 384.

[0062] The expression for obtaining the global attention strength of the aerial image block is: Among them, A global is the global attention strength of the aerial image block, L is the last block of Transformer, and H is the number of multi-head attention heads.

[0063] S108: For the aerial image marked as retrievable, use the cosine similarity between the global feature vector of the aerial image marked as retrievable and the global feature vector of the offline feature library, and return a preset number of candidate results in descending order.

[0064] The expression of the global feature vector of the aerial image marked as retrievable is: f I =Φ(I;θ); where Φ(·) represents the feature extraction network, θ is the network parameter, and f I ∈R D is the extracted D-dimensional searchable global feature vector of the aerial image, R D is a real vector space of dimension D. The global feature vector of the aerial image marked as retrievable captures the high-level semantic information of the aerial image and has the same dimension and representation space as the feature vector stored in the offline feature library.

[0065] Specifically, the global feature vector of aerial images marked as retrievable is cls token, with a dimension of (1,384).

[0066] The cosine similarity between the global feature vector of the aerial image marked as retrievable and the global feature vector of the offline feature library is used to return a preset number of candidate results in descending order, including: the calculation formula of the cosine similarity is: Among them, f j represents the jth feature vector in the offline feature library, S(f I ,f j) is the cosine similarity of the global feature vector of the aerial image that can be retrieved by the jth feature vector in the offline feature library. The expression for returning a preset number of candidate results in descending order is: TopN′={j1,j2,…,j N}, Among them, TopN′ is a set of N′ candidate results, j1, j2, …, j N is the index of the candidate result in the offline feature library, and N is the total number of candidate results.

[0067] Specifically, S(f I ,f j )∈[-1,1], which represents the similarity between the jth feature vector in the offline feature library and the target feature vector. The closer the value is to 1, the higher the matching degree.

[0068] S109: Associating each candidate result with the geographic location information in the offline feature library to locate the satellite image that is most similar to the aerial image.

[0069] Associating each candidate result with the geographic location information in the offline feature library to locate the satellite image that is most similar to the aerial image, including: Associating each candidate result with the geographic location information in the offline feature library is expressed as: Location(j i )=(longitude i ,latitude i ). Among them, j i is the index of the i-th candidate result after sorting, Location(j i ) is the candidate result j i The corresponding geographic location information, (longitude i ,latitude i ) is the candidate result j i Longitude and latitude coordinates, longitude i is the candidate result j i longitude, latitude i is the candidate result j i The satellite image corresponding to the first candidate result after descending sorting is the satellite image most similar to the aerial image.

[0070] S110: Returning a search result list containing a set of a preset number of candidate results, each search result including matching geographic location information and cosine similarity.

[0071] Figure 2 A schematic diagram of the process of a remote sensing image retrieval method based on hierarchical semantic feature fusion provided in an embodiment of the present application.

[0072] Specifically, the index structure characteristics of the present application are as follows. 1. Global feature association. The feature vector of each satellite image block is closely associated with its corresponding geographic location information and stored. This association method can not only quickly locate the most similar satellite image, but also accurately return its geographic location information, providing strong support for subsequent navigation or analysis. 2. Efficient retrieval performance. This application uses the FAISS (Facebook AI Similarity Search) library to construct an index structure for high-dimensional feature vectors. The FAISS library is known for its efficient retrieval performance and can support millisecond-level retrieval responses for image libraries of hundreds of millions, greatly improving the processing speed and efficiency of the system. 3. Memory optimization technology. For large-scale offline feature libraries, memory optimization technologies such as hierarchical quantization and product quantization are adopted. These technologies can effectively control memory usage while maintaining high retrieval accuracy, ensuring stability and scalability in large-scale data processing scenarios.

[0073] This application ultimately returns a search result list containing a set of a preset number of candidate results, where each search result contains the following key information. 1. Matching geographic location coordinates. Expressed in the form of longitude and latitude, it accurately indicates the specific location of the matching satellite image on the earth. 2. Similarity score. The cosine similarity reflects the degree of similarity between the query image, i.e., the aerial image, and the matching image and satellite image, helping users to quickly evaluate the reliability of the search results. 3. Optional matching area visualization information. Providing an intuitive display of the matching area helps users to more intuitively understand the search results and conduct subsequent analysis. The remote sensing image retrieval method based on hierarchical semantic feature fusion of this application is based on global feature retrieval and combined with an efficient index structure to achieve fast and accurate positioning of aerial images. It provides a reliable location reference for UAV visual navigation, which helps to improve the autonomous navigation capability and task execution efficiency of UAVs.

[0074] This application selects 1,000 pairs of satellite images and aerial images from the dataset as a test set. Among these image pairs, aerial images are used as query images, and all corresponding satellite images together constitute the retrieval database images for verifying the retrieval performance. The dataset selected in this application can be the SpaceNet Dataset dataset. The experimental environment of this application is built on the Windows operating system, with the following hardware and software configurations: Windows is used as the operating system, PyTorch is used as the deep learning framework, and the graphics card is equipped with NVIDIA GeForce GTX 1080Ti, which has 11GB of video memory and provides powerful computing support for the experiment. During the experiment, data processing and testing work were carried out based on the PyTorch framework. By utilizing the parallel computing power of the GTX1080Ti graphics card, the experimental process was significantly accelerated, ensuring the stability and efficiency of the testing process. This configuration not only meets the computing resources required for the experiment, but also lays a solid foundation for subsequent analysis and verification work.

[0075] To comprehensively evaluate the method introduced in this application and verify whether excluding low-retrieval-ability images can effectively improve retrieval accuracy, this application designed the following experiments and evaluation metrics. The baseline method uses direct retrieval of all database images as a benchmark for performance comparison. The proposed method first filters the database images, removing low-quality images before performing retrieval. Retrieval accuracy is calculated as the proportion of correct matches among the top k retrieval results, where k is 3, 5, and 10, respectively, to comprehensively evaluate the performance of the retrieval system. The number of database images before and after filtering is recorded to analyze the effectiveness of the retrievability evaluation module in reducing database size. The experimental process is as follows. Feature extraction is performed on 1,000 satellite images from the original database in preparation for subsequent retrieval experiments. The proposed retrievability evaluation mechanism is used to score query (aerial) images to quantify their retrievability. A threshold T = 0.5 is set, and images with retrievability scores below this threshold are removed to form a filtered database. Retrieval experiments are conducted using both the original query images and the filtered query images, and the retrieval performance of the two methods is recorded and compared. The experimental results are statistically analyzed to verify the effectiveness of the retrievability evaluation module.

[0076] To gain a deeper understanding of the independent contributions of each component of the retrievability evaluation mechanism, this application further designed an ablation experiment. Specifically, the performance of the active region ratio (NZR) based on the first three dimensions of PCA dimensionality reduction and the attention score-based metric AS were evaluated independently, and compared with the combined metric CRS. The evaluation metrics and experimental parameters of the ablation experiment were designed to be consistent with the above experiments to ensure the comparability and accuracy of the experimental results.

[0077] The experimental results are shown in Table 1. By introducing the retrievability evaluation mechanism, the database size was significantly reduced from the original 1,000 images to 792, a 20.8% reduction. This data indicates that approximately one-fifth of low-retrieval images were successfully identified and filtered. These filtered images typically feature large, homogeneous areas or lack discriminative features, such as vast expanses of water, homogeneous farmland, and extensive cloud cover. These images are difficult to represent effectively in the visual feature space and are therefore classified as low-retrieval and eliminated. The retrieval method using retrievability evaluation achieved significant improvements in retrieval performance across all evaluation metrics. Success Rate@k represents the proportion of correct matches in the top k retrieval results, with k selected as 3, 5, or 10. Success Rate@3 increased from 68.00% to 72.12%, and Success Rate@5 increased from 73.86% to 79.73%. This improvement demonstrates that by eliminating low-retrieval images, the system's retrieval accuracy has been effectively enhanced. The improvement in retrieval performance gradually decreases as the k value increases. That is, the improvement in Success Rate@3 is greater than that in Success Rate@5. This phenomenon suggests that retrievability evaluation is particularly effective in improving the quality of top-ranked search results. In practical applications, the quality of top-ranked search results has a more direct impact on the user experience and practicality of the system. The top-ranked search results refer to the first k search results in the search result list, after being sorted according to the sorting rules.

[0078] Table 1

[0079] Method Data quantity SuccessRate@3 SuccessRate@5 Before adding searchability assessment 1000 68.00 73.86 After adding the searchability assessment 792 72.12 79.73

[0080] Figure 3 Schematic diagram of some of the aerial images that were screened out in the embodiments of this application. These images intuitively reflect the retrievability evaluation mechanism, that is, identifying and eliminating low-retrievability images that are difficult to form effective discriminant representations.

[0081] In summary, the experimental results fully demonstrate the effectiveness of the retrievability evaluation mechanism in improving the performance of aerial image retrieval systems. By identifying and eliminating low-retrievability images in the early stages of retrieval, this not only improves retrieval accuracy but also reduces the computational burden, achieving a dual improvement in both precision and efficiency. This achievement has important practical value for resource-constrained real-time visual navigation systems.

[0082] As shown in Table 2, by comparing and analyzing the NZR indicator based on PCA dimensionality reduction and the AS indicator based on attention score, it is found that the two indicators each have unique advantages in improving retrieval results and are significantly complementary.

[0083] Table 2

[0084] Findability Module Data quantity SR@3 SR@5 SR@10 / 1000 66.89 73.1 88.17 PCA 715 68.06 74.52 89.06 Atten_Score 863 71.41 77.38 90.35 PCA+Atten_Score 792 72.12 79.73 92.69

[0085] The PCA metric uses a stricter filtering criteria, successfully removing 28.5% of samples. These samples were primarily concentrated in large water areas, homogeneous vegetation areas, or areas with monotonous surface textures. By removing these low-information samples, retrieval accuracy has steadily improved. Notably, the PCA metric is particularly effective in retaining areas with strong structure but lacking distinctive landmarks, such as regularly gridded farmland and repetitive residential areas. While these areas exhibit high visual similarity, they still retain rich dimensional information in the principal component space, which helps improve retrieval accuracy. In contrast, the attention score metric demonstrates sensitivity to salient landmarks. While filtering out only 13.7% of samples, it yields a significantly greater improvement in accuracy. This demonstrates that the Transformer-based attention mechanism is able to effectively identify elements with unique visual features (such as distinctive buildings, road intersections, and land-water boundaries), providing strong matching points for retrieval. The filtered samples are mostly scenes that lack visual focus or are distracting, such as cloud cover and large areas of homogeneous landforms. The combined use of the two metrics yields the best performance, achieving an optimal balance between retrieval accuracy and computational efficiency. This result verifies the hypothesis of this application: retrievability is a comprehensive reflection of multi-dimensional characteristics, and evaluation from a single perspective is often not comprehensive enough. The fusion method can not only identify complex scenes with rich structures, but also retain simple scenes with a small but significant number of landmarks, while effectively filtering out low-value samples that lack both characteristics, thereby achieving comprehensive performance improvement with a 20.8% reduction in data volume. Table 2 lists the results of the ablation experiment in detail, including the amount of data under different retrievability modules, SR@3, SR@5, and SR@10 indicators. It can be seen from the table that when the PCA and attention score indicators are integrated, all retrieval indicators reach the optimal value.

[0086] Figure 4 This is a schematic diagram of the top 3 search results obtained using the search method of the present application provided in an embodiment of the present application. The top 3 is a set of 3 candidate results.

[0087] Figure 5 This is a schematic diagram of the top 5 search results obtained using the search method of the present application provided in an embodiment of the present application. The top 3 is a set of 5 candidate results.

[0088] Figure 6 This is a schematic diagram of the Top 10 search results obtained using the search method of the present application provided in an embodiment of the present application. The Top 10 is a set of 10 candidate results.

[0089] from Figures 4 to 6It can be seen that under different TopN′ parameter settings, this application shows accurate and stable effects.

[0090] Figure 7 A schematic diagram of the retrieval effect of the Top 3 search results provided in an embodiment of the present application under multiple light sources.

[0091] Figure 8 A schematic diagram of the retrieval effect of the Top 5 search results provided in an embodiment of the present application under multiple light sources.

[0092] Figure 9 A schematic diagram of the retrieval effect of the Top 10 search results provided in an embodiment of the present application under multiple light sources.

[0093] from Figures 7 to 9 It can be seen that even under the interference of different light sources, this application can still accurately identify targets that are highly similar to the query image, demonstrating its powerful feature extraction and matching capabilities.

[0094] The present application also provides a remote sensing image retrieval device 1000 based on layered semantic feature fusion. Figure 10 As shown, the device includes: a construction module 1001, a global feature vector acquisition module 1002, a front 3-dimensional feature active area ratio acquisition module 1003, an attention score acquisition module 1004, an attention mean acquisition module 1005, a comprehensive retrievability evaluation index construction module 1006, a marking module 1007, a cosine similarity acquisition module 1008, a positioning module 1009 and a return module 1010.

[0095] The construction module 1001 is used to construct an offline feature library based on satellite images, and realize online real-time retrieval of aerial images based on the offline feature library.

[0096] The module 1002 for obtaining a global feature vector is used to receive an aerial image acquired by a drone in real time, and to perform feature extraction on the aerial image using a feature extraction network to obtain a global feature vector.

[0097] The module 1003 for obtaining the active area ratio of the first three-dimensional features is used to perform principal component analysis dimensionality reduction on the extracted global feature vector to obtain the active area ratio of the first three-dimensional features after dimensionality reduction; wherein, the active area ratio of the first three-dimensional features is used to measure the proportion of the area containing valid semantic information in the aerial image, and the active area refers to the image block that needs to enter the preset threshold in at least one feature direction.

[0098] The attention score acquisition module 1004 is used to obtain the attention scores of the aerial image on different attention heads by using the attention mechanism of the feature extraction network during the feature extraction process.

[0099] The attention mean acquisition module 1005 is used to obtain the attention mean based on the attention scores of different attention heads, and perform normalization processing to obtain the normalized attention mean to represent the attention distribution characteristics of the aerial image in the feature extraction network.

[0100] The module 1006 for constructing a comprehensive retrievability evaluation index is used to construct a comprehensive retrievability evaluation index by integrating the active area ratio of the first three-dimensional features and the normalized attention mean.

[0101] The marking module 1007 is used to mark the aerial image as retrievable if the comprehensive retrievability evaluation index is greater than the attention intensity threshold, otherwise, mark it as non-retrievable.

[0102] The cosine similarity obtaining module 1008 is used to return a preset number of candidate results in descending order using the cosine similarity between the global feature vector of the aerial image marked as searchable and the global feature vector of the offline feature library.

[0103] The positioning module 1009 is used to associate each candidate result with the geographical location information in the offline feature library to locate the satellite image that is most similar to the aerial image.

[0104] The return module 1010 is used to return a search result list containing a set of a preset number of candidate results, each search result including matching geographic location information and cosine similarity.

[0105] Some modules in the apparatus described herein may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, classes, etc. that perform specific tasks or implement specific abstract data types. The present application may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communications network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0106] The devices or modules described in the above application embodiments can be implemented by computer chips or physical devices, or by products with certain functions. For ease of description, the above devices are described separately by function in various modules. When implementing the embodiments of this application, the functions of each module can be implemented in the same or multiple software and / or hardware. Of course, a module that implements a certain function can also be implemented by combining multiple sub-modules or sub-units.

[0107] The methods, devices, or modules described in this application can be implemented in the form of computer-readable program code. The controller can be implemented in any appropriate manner. For example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also know that in addition to implementing the controller in the form of pure computer-readable program code, it is entirely possible to implement the same function of the controller in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, such a controller can be considered a hardware component, and the devices included therein for implementing various functions can also be considered as structures within the hardware component. Or even, the means for implementing various functions may be considered to be both a software module for implementing the method and a structure within a hardware component.

[0108] like Figure 11 As shown, an embodiment of the present application also provides a remote sensing image retrieval server based on hierarchical semantic feature fusion, including a memory 1101 and a processor 1102; the memory 1101 is used to store computer-executable instructions; the processor 1102 is used to execute computer-executable instructions to implement a remote sensing image retrieval method based on hierarchical semantic feature fusion described above in an embodiment of the present application.

[0109] An embodiment of the present application also provides a computer-readable storage medium, which stores executable instructions. When a computer executes the executable instructions, it can implement the remote sensing image retrieval method based on hierarchical semantic feature fusion described above in the embodiment of the present application.

[0110] Through the description of the above implementation methods, it can be known that those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary hardware. Based on this understanding, the technical solution of the present application can be essentially or the part that contributes to the prior art can be embodied in the form of a software product, or it can be embodied in the implementation process of data migration. The computer software product can be stored in a storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a mobile terminal, a server, or a network device, etc.) to execute the method described in the embodiment of the present application.

[0111] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to in detail. Each embodiment focuses on the differences from other embodiments. All or part of this application can be used in many general or special computer system environments or configurations.

[0112] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit the present application. Although the present application has been described in detail with reference to the aforementioned embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the aforementioned embodiments can still be modified, or some or all of the technical features therein can be replaced by equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the present application.

Claims

1. A remote sensing image retrieval method based on hierarchical semantic feature fusion, characterized in that: include: Build an offline feature library based on satellite images, and realize online real-time retrieval of aerial images based on the offline feature library; Online real-time search includes: Receive aerial images acquired by the drone in real time, use the feature extraction network to extract features from the aerial images, and obtain the global feature vector; Perform principal component analysis on the extracted global feature vector to reduce its dimensionality and obtain the proportion of active regions of the first three-dimensional features after dimensionality reduction. The proportion of active regions of the first three-dimensional features is used to measure the proportion of regions in the aerial image that contain valid semantic information. An active region refers to an image block that needs to enter a preset threshold in at least one feature direction. During the feature extraction process, the attention mechanism of the feature extraction network is used to obtain the attention scores of the aerial image on different attention heads; The attention mean is obtained based on the attention scores of different attention heads, and normalized to obtain the normalized attention mean to represent the attention distribution characteristics of the aerial image in the feature extraction network; The comprehensive retrievability evaluation index is constructed by combining the active area ratio of the first three-dimensional features and the normalized attention mean; If the comprehensive retrievability evaluation index is greater than the attention intensity threshold, the aerial image is marked as retrievable, otherwise, it is marked as non-retrievable; For aerial images marked as retrievable, use the cosine similarity between the global feature vector of the aerial image marked as retrievable and the global feature vector of the offline feature library, and return a preset number of candidate results in descending order; Associate each candidate result with the geographic location information in the offline feature library to locate the satellite image that is most similar to the aerial image; Returns a search result list containing a set of preset number of candidate results, each of which includes matching geographic location information and cosine similarity.

2. The remote sensing image retrieval method based on hierarchical semantic feature fusion according to claim 1, characterized in that: The offline feature library is constructed based on satellite images, including: Establishing a geographic coordinate system for the satellite image and dividing the satellite image into multiple satellite image blocks; Record the geographic location information of each satellite image block in the satellite image it is in, including grid index, pixel coordinates and geographic coordinates; Preprocess satellite image blocks; Use the feature extraction network to extract features from the preprocessed satellite image blocks. The extracted features include the global feature vector and multi-head attention distribution score of the offline feature library. The global feature vector, multi-head attention distribution score and geographic location information of the extracted offline feature library are organized into structured feature data and stored in the offline feature library; An index structure of a global feature vector based on an offline feature library is used to associate the stored global feature vector with geographic location information.

3. The remote sensing image retrieval method based on hierarchical semantic feature fusion according to claim 2, characterized in that: The above-mentioned comprehensive three-dimensional feature active area ratio and normalized attention mean are used to construct a comprehensive retrievability evaluation index, including: The expression of comprehensive searchability evaluation index is: CRS = α·NZR+β·AS norm ; Among them, CRS is the comprehensive retrievability evaluation index, NZR is the ratio of the active area of the first three-dimensional features, α and β are weighting coefficients, and α+β=1. AS is the mean attention value, AS norm is the normalized mean attention value.

4. The remote sensing image retrieval method based on hierarchical semantic feature fusion according to claim 3, characterized in that: The expression for determining the attention intensity threshold is: T = μ-k·σ; wherein T is the attention intensity threshold, k is a hyperparameter, μ is the mean of the global attention intensity of all aerial image blocks in the aerial image, and σ is the standard deviation of the global attention intensity of all aerial image blocks in the aerial image.

5. The remote sensing image retrieval method based on hierarchical semantic feature fusion according to claim 4, characterized in that: The method for obtaining the global attention strength of the aerial image block includes: using the attention mechanism in the feature extraction network to obtain the attention weight of the aerial image block on the multi-head attention head on the last block of the Transformer, and obtaining the global attention strength of the aerial image block; The expression for obtaining the attention weight of the aerial image block on the multi-head attention head of the last block of Transformer is: Among them, N is the number of image blocks into which the aerial image is divided, A h is the attention weight of the hth attention head on the last block of the aerial image block, h is the index of the attention head, i is the index of the aerial image block, is the attention score of the h-th attention head in the last layer on the i-th image block; The expression for obtaining the global attention strength of the aerial image block is: Among them, A global is the global attention strength of the aerial image block, L is the last block of Transformer, and H is the number of multi-head attention heads.

6. The remote sensing image retrieval method based on hierarchical semantic feature fusion according to claim 5, characterized in that: The expression of the global feature vector of the aerial image marked as retrievable is: f I =Φ(I;θ); where Φ(·) represents the feature extraction network, θ is the network parameter, and f I ∈R D is the extracted D-dimensional searchable global feature vector of the aerial image, R D is a D-dimensional real vector space.

7. The remote sensing image retrieval method based on hierarchical semantic feature fusion according to claim 6, characterized in that: The method uses the cosine similarity between the global feature vector of the aerial image marked as retrievable and the global feature vector of the offline feature library, and returns a preset number of candidate results in descending order, including: The calculation formula for cosine similarity is: Among them, f j represents the jth feature vector in the offline feature library, S(f I ,f j ) is the cosine similarity between the jth feature vector in the offline feature library and the global feature vector of the retrievable aerial image; The expression for returning a preset number of candidate results in descending order is: Among them, TopN′ is a set of N′ candidate results, j1, j2, …, j N is the index of the candidate result in the offline feature library, and N is the total number of candidate results.

8. The remote sensing image retrieval method based on hierarchical semantic feature fusion according to claim 7, characterized in that: The step of associating each candidate result with the geographic location information in the offline feature library to locate the satellite image most similar to the aerial image includes: The expression for associating each candidate result with the geographic location information in the offline feature library is: Location(j i )=(longitude i ,latitude i ); where j i is the index of the i-th candidate result after sorting, Location(j i ) is the candidate result j i The corresponding geographic location information, (longitude i ,latitude i ) is the candidate result j i Longitude and latitude coordinates, longitude i is the candidate result j i longitude, latitude i is the candidate result j i Latitude; The satellite image corresponding to the first candidate result after descending sorting is the satellite image most similar to the aerial image.

9. A remote sensing image retrieval device based on hierarchical semantic feature fusion, characterized in that: include: A construction module is used to build an offline feature library based on satellite images and realize online real-time retrieval of aerial images based on the offline feature library; A global feature vector acquisition module is used to receive aerial images acquired by a UAV in real time, and use a feature extraction network to extract features from the aerial images to obtain a global feature vector; The module for obtaining the active area ratio of the first 3-dimensional features is used to perform principal component analysis and dimensionality reduction on the extracted global feature vectors to obtain the active area ratio of the first 3-dimensional features after dimensionality reduction. The active area ratio of the first 3-dimensional features is used to measure the proportion of areas in the aerial image that contain valid semantic information. The active area refers to the image block that needs to enter the preset threshold in at least one feature direction. The attention score acquisition module is used to obtain the attention scores of aerial images on different attention heads during the feature extraction process by using the attention mechanism of the feature extraction network; The attention mean acquisition module is used to obtain the attention mean based on the attention scores of different attention heads and perform normalization to obtain the normalized attention mean to represent the attention distribution characteristics of the aerial image in the feature extraction network; Construct a comprehensive retrievability evaluation index module to integrate the active area ratio of the first three-dimensional features and the normalized attention mean to construct a comprehensive retrievability evaluation index; a marking module, for marking the aerial image as retrievable if the comprehensive retrievability evaluation index is greater than the attention intensity threshold, and marking it as non-retrievable otherwise; A cosine similarity module is obtained, which is used to return a preset number of candidate results in descending order using the cosine similarity between the global feature vector of the aerial image marked as retrievable and the global feature vector of the offline feature library; The positioning module is used to associate each candidate result with the geographic location information in the offline feature library to locate the satellite image that is most similar to the aerial image; The return module is used to return a search result list containing a set of a preset number of candidate results, each search result including matching geographic location information and cosine similarity.

Images