Unmanned aerial vehicle-satellite scene matching method based on two stages

Abstract

The invention discloses an unmanned aerial vehicle-satellite scene matching method based on two stages. Precise scene matching and positioning are completed from coarse granularity to fine granularity. An existing satellite base map retrieval task is used as a first-stage task, and a local feature point matching method is introduced in a second stage. Compared with an existing method, the method has the advantages that the matching robustness in a complex scene is remarkably improved, meter-scale accurate positioning of the unmanned aerial vehicle can be realized on an existing satellite map, and a new universal solution is provided for the field of scene matching.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision, and particularly relates to a two-stage unmanned aerial vehicle (UAV)-satellite scene matching method. Background Art

[0002] The definition of the scene matching task is a process of aligning two images captured at the same scene location from two different sensors in space to determine their relative displacement. Applying the scene matching task technology to the visual positioning of UAVs can get rid of the dependence of the UAV positioning system on positioning signals, so as to still perform effective positioning under adverse conditions such as the absence of GPS signals. In recent years, the research on scene matching in UAV positioning has also received extensive attention. Traditional UAV scene matching technologies are realized by designing local invariant features (handcrafted features) of pictures. First, the local information of UAV images and satellite images is extracted, mainly including the intensity and gradient of the images. Subsequently, a normalization strategy is used to generate feature descriptors to descriptively distinguish different image data. Finally, a similarity estimate of the descriptors between the UAV image and the satellite image is given to complete the matching. However, this method based on local invariant features has limitations in that the features that can be explicitly expressed are insufficient when facing the rich and diverse appearances and shapes of complex scenes.

[0003] In order to overcome the limitations of the scene matching method based on local invariant features, in recent years, UAV scene matching methods based on deep learning have been proposed and widely studied. The deep neural network improves the robustness of the matching algorithm in complex scenes through its ability to capture high-dimensional semantic features in images. Currently, most of the existing deep learning-based scene matching methods use the Vision Transformer network to extract the global features of UAV and satellite images, and then calculate the similarity of the global feature vectors of UAV and satellite images, and finally complete the matching through the sorting of similarities. However, the robustness of this method is still limited when facing multiple scenes, multiple altitudes, and natural environmental interferences (such as occlusion, night, rain, and snow). Moreover, the current research on UAV-satellite scene matching methods stays at a relatively coarse-grained stage, aiming to retrieve the satellite image most similar to the UAV query image from the satellite base map database, and cannot achieve centimeter-level precise positioning. Summary of the Invention

[0004] To overcome the deficiencies of the prior art, the present invention provides a two-stage UAV-satellite scene matching method, which accomplishes precise scene matching and positioning from coarse-grained to fine-grained. The existing satellite base map retrieval task is used as the first-stage task, and the method of local feature point matching is introduced in the second stage. Compared with the existing methods, the robustness of this method in complex scenes is significantly improved, and it can achieve meter-level precise positioning of the UAV on the existing satellite map, providing a new general solution for the field of scene matching.

[0005] The technical solutions adopted by the present invention to solve its technical problems are as follows:

[0006] Step 1: Data preprocessing and dataset loading;

[0007] Load the satellite base map in the dataset, and after data preprocessing, use it as the input for the feature extraction part;

[0008] Step 2: Global feature extraction;

[0009] Step 3: Indexing and retrieval of feature vectors to obtain the matching "UAV-satellite" image pairs;

[0010] Step 4: Feature extraction based on the "UAV-satellite" image pairs obtained in Step 3;

[0011] Step 5: Graph construction;

[0012] Step 6: Information propagation;

[0013] Step 7: Key point matching.

[0014] Preferably, the specific content of Step 1 is as follows:

[0015] For each satellite base map, use the sliding window method to set the size of the patch and the size of the sliding step, and cut the satellite base map into several patches; for each patch, use the resize method to adjust its resolution. Subsequently, use the RandomPerspective method in the torchvision.transforms library to perform perspective enhancement on each patch, and perform normalization processing, which is used as the input for the feature extraction part.

[0016] Preferably, the specific content of Step 2 is as follows:

[0017] Step 2-1: Input each satellite base map slice patch in step 1 into the backbone network DINOV2 in the form of (1, C, H, W), where 1 represents that the batch_size of the image to be extracted is 1; C represents the number of image channels; H and W represent the width and height of the image respectively; each satellite base map slice patch is further divided into image blocks of a fixed size after entering the backbone network DINOV2; assuming there are N image blocks after division, and the size of each image block is P×P, we get:

[0018]

[0019] Step 2-2: The backbone network DINOV2 performs a linear projection on each image block, projects it to a fixed dimension, and converts each image block into a high-dimensional feature vector:

[0020]

[0021] where W p is the weight matrix of the linear projection, b p is the bias vector, z i 0 is the embedded representation of the i-th image block, x i is the embedded representation of the i-th image block;

[0022] Step 2-3: The backbone network DINOV2 adds the position encoding of each image block to its embedding vector to ensure that the model learns the spatial relationship; uses the CLS Token to participate in the Transformer calculation together with the image blocks to aggregate the global features from the entire image:

[0023]

[0024] z 0 ←z 0 +E pos

[0025] where, E pos represents the position encoding, Z cls 0 represents the CLS Token for aggregating global features;

[0026] Step 2-4: After adding the position encoding and the CLS Token, it is processed by the Transformer Encoder, and then the input features are processed through the multi-head attention mechanism and the feed-forward network;

[0027] Through the self-attention mechanism, the global context information of each picture is captured, thus obtaining the initial global feature vector:

[0028] Q = z lW q , K = z l W k , V = z l W v

[0029]

[0030] Among them, W q , W k , W v are the linear weight change matrices of the attention layer, D represents the scaling factor, Q, K, and V respectively represent the matrices of queries, keys, and values in the attention layer, and z l represents the input embedding vector of the l-th layer of attention;

[0031] The backbone network DINOV2 adopts the multi-head self-attention mechanism. There are multiple parallel attention layers in the network, and the results of multiple layers are concatenated together and then linearly transformed to obtain the output;

[0032] The initial global vector obtained through the Transformer Encoder layer enters a feed-forward network layer for dimensional change. The CLS Token part in the output contains the global features of the entire image:

[0033] FFN(z) = GELU(zW1 + b1)W2 + b2

[0034]

[0035] Among them, FFN represents the feed-forward network layer, Z cls L represents the CLS Token part, GELU represents the activation function, z represents the initial global vector obtained through the Transformer Encoder layer, W1 represents the weight matrix of the first linear layer of FFN, W2 represents the weight matrix of the second linear layer of FFN, b1 represents the bias vector of the first linear layer, b2 represents the bias vector of the second linear layer, and f global represents the finally extracted global image features;

[0036] Step 2-5: Introduce the local feature aggregation algorithm SALAD based on the optimal transport sinkhorn algorithm to perform feature aggregation on the features extracted by DINOV2; after each satellite slice patch obtains global features through the backbone network DINOV2, the global features enter the SALAD module for aggregation; the SALAD module constructs the cost matrix between the global features and the clusters; through continuous iteration of the sinkhorn algorithm until a correspondence between the features and the clusters is found to minimize the total cost; finally, the global features of the image will be aggregated into 64 clusters, and the dimension of each cluster is 128.

[0037] After the preliminary extraction of the global features by the backbone network DINOV2 and the feature aggregation of the SALAD module, the final global features of each satellite base map slice patch are obtained.

[0038] Preferably, C takes the value of 3.

[0039] Preferably, Step 3 is specifically as follows:

[0040] Use the faiss library to construct an index for the global feature vectors of each satellite base map slice patch, and store the name of the satellite base map corresponding to each patch and its position on the satellite base map in a csv file; for each drone image, perform a resize operation to modify the resolution, and extract its global features in the same way through Step 2, calculate the similarity of the global feature vectors between the drone image and each satellite base map slice patch, and after sorting according to the similarity, the matching "drone - satellite" image pairs can be found;

[0041] Preferably, Step 4 is specifically as follows:

[0042] Based on the "drone - satellite" image pair (A, B), extract key points from the two images through the SuperPoint algorithm, denoted as: A = {A1, A2,..., A N}, B = {B1, B2,..., B M}, where N and M respectively represent the number of key points recognized in the two images; and calculate the local descriptors of each key point: At the same time, perform position embedding and multi-layer perceptron processing on the position features of the key points to further refine their spatial positions, and obtain the position embedding: In addition to the SuperPoint descriptor, extract the DINOV2 feature map and obtain the DINOV2 descriptor of each key point through interpolation operation: Finally, each key point has three types of features: local descriptor, position feature and DINOV2 feature, denoted as: for the drone image A: {d A , p A, g A}, for satellite perspective image B: {d B , p B , g B}.

[0043] Preferably, step 5 is specifically as follows:

[0044] For the input image pair, construct four key-point association graphs, including two inter-image graphs and two intra-image graphs;

[0045] The inter-image graph calculates the similarity between key points through DINOV2 features and connects the most similar key points, thus avoiding redundant matching; specifically, the inter-image graph respectively represents the connection relationship from image I A to I B , denoted as G A→B , and the connection relationship from image I B to I A , denoted as G B→A ; for each pair of key points A i and B j , calculate the similarity between their DINOV2 features, expressed as:

[0046]

[0047] The intra-image graph forms an undirected graph by connecting all key points within the same image; the connection between each key point and other key points ensures that information can be fully propagated between the same image and different images; specifically, within the same image, the connection relationship between all key points is represented by an undirected graph, and the connection between each key point A i and A j is represented by the following similarity metric:

[0048]

[0049] Preferably, step 6 is specifically as follows:

[0050] Update the key points through the self-attention layer and the cross-image attention layer; the self-attention layer processes the intra-image graph and updates the features of the key points within the same image; the cross-image attention layer processes the inter-image graph and updates the key point features according to the key point matching information guided by DINOV2; in addition, introduce a position-guided attention mechanism to decouple the position information and local descriptors of the key points, so that the model can rely only on the local descriptors when updating the features, and use the position information as the spatial context; specifically, the attention layer is defined in the following form:

[0051]

[0052] Among them, S represents the key set related to set A; K is a scalar representing the number of key points in set S; C is a scalar representing the number of feature channels; represents the local feature descriptor extracted by the superpoint of the i-th key point in set A; represents the position encoding of the i-th key point in set A in image A; d S represents the set of local feature descriptors of set S, containing K feature points, and the dimension of each descriptor is C; p s represents the position encoding matrix of all key points in set S, with a dimension of K×C; W q , W k , W v represents the projection matrix used to transform the input features into the Query / Key / Value space of the transformer; b q , b k , b v are all bias terms used to adjust the output of Query / Key / Value; represents the query vector generated by the transformer layer according to and ; k s and v s represent the key vector and value vector respectively.

[0053] Preferably, step 7 is specifically as follows:

[0054] In the matching stage, the model uses the updated key point features to calculate the similarity matrix between key point pairs Among them, Optimize the matching similarity through the Sinkhorn algorithm to obtain a matching matrix M∈[0, 1] N×M , where M i，j represents the matching probability between key points A i and B j ; represents the matching probability of each pair of key points; finally, use the binary classification task to train the model by minimizing the negative log-likelihood loss function of the matching matrix, so that the model can accurately match key points between different images.

[0055] The beneficial effects of the present invention are as follows:

[0056] In view of the challenges of the UAV scene matching task, the present invention proposes an improved method to enhance the matching accuracy and robustness, especially the adaptability in complex environments and multi-scene conditions. By introducing deep learning technology and using advanced network structures such as Vision Transformer to extract global features, compared with traditional local invariant feature matching methods, it has stronger matching stability in complex scenarios such as different altitudes and natural environment interference. At the same time, this method breaks through the coarse-grained limitation of the existing UAV-satellite scene matching and realizes precise positioning with meter-level accuracy, enabling the UAV to still be reliably positioned in the case of GPS signal loss or interference. In addition, by optimizing the matching algorithm, the accuracy and efficiency of feature extraction and similarity calculation are improved, making it applicable to a wider range of practical application scenarios, thus providing an efficient and accurate solution for UAV autonomous navigation and visual positioning. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 It is a schematic diagram of the method of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0058] The present invention will be further described below with reference to the drawings and embodiments.

[0059] The present invention mainly consists of two stages. The first stage is to use the DINOV2+SALAD model for global feature extraction, vector indexing, and retrieval of drone and satellite images. DINOV2 is a global feature extraction backbone network trained using self-supervised learning and large-scale data, and SALAD is a method based on optimal transport that aggregates global features into clusters of specific dimensions. The combination of the two can enhance the ability to extract global features. After extracting the global features, the global features of the satellite base map are indexed through the faiss library to construct a global feature database of the satellite, while the drone image is used as the query image. After extracting the global features of the drone, the similarity between the drone's global features and each satellite's global features is calculated, and they are sorted according to the similarity level to obtain the "drone-satellite" image pair with the highest similarity. The second stage can be specifically divided into the following four sub-modules. First, in the first part, feature extraction is performed based on the "drone-satellite" image pair matched in the first stage. The features of the key points are extracted from the input image pair, including detecting and describing the key points using SuperPoint, extracting the global image features using DINOV2, and refining the spatial position information of the key points through position embedding. These features provide the necessary local descriptors, global context information, and accurate spatial position descriptions for the subsequent matching process. Second, the image construction module. By constructing the intra-image graph and the inter-image graph, the associations between the key points in the image are represented. The inter-image graph calculates the similarity between the key points through DINOV2 features to construct a bidirectional association graph, and the intra-image graph represents the relationship between the key points in the same image. These graphs provide a structured basis for information propagation, enabling the model to capture the connections between the key points. Third, the information code propagation module. In the information propagation module, the information update between the key points is achieved through the self-attention layer and the cross-image attention layer. The self-attention layer updates the representation of the key points in the same image, while the cross-image attention layer updates the key points between images. This module combines the DINOV2 features to guide the cross-image matching process and decouples the spatial information and descriptors of the key points through the position guidance mechanism, avoiding over-reliance on position priors and improving the accuracy and generalization ability of the matching. Fourth, the matching and optimization module. The matching and optimization module is responsible for calculating the final matching relationship between the key points. By calculating the similarity between the descriptors, a matching matrix is generated, and the Sinkhorn algorithm is used to optimize the matching result. In addition, the model trains and optimizes the matching ability of the model by minimizing the negative log-likelihood loss of the matching matrix to ensure accurate key point pairing and an efficient matching process. Finally, the feature point matching between the "drone-satellite" image pairs is achieved.

[0060] The first stage of this method mainly includes the following steps:

[0061] Step 1: Data preprocessing and dataset loading;

[0062] First, load the satellite base maps in the dataset. For each satellite base map, using the sliding window method, set the size of the patch and the sliding step size by yourself, and cut the satellite base map into several patches. For each patch, use the resize method to adjust its resolution. Subsequently, use the RandomPerspective method in the torchvision.transforms library to perform perspective enhancement on each patch and perform normalization processing, which serves as the input for the feature extraction part.

[0063] Step 2: Global feature extraction;

[0064] Input each satellite base map slice patch in Step 1 into the backbone network DINOV2 in the form of (1, C, H, W), where 1 represents that the batch_size of the image to be extracted is 1; C represents the number of image channels, usually 3 (RGB image); H and W represent the width and height of the image. Since the backbone of the DINOV2 network is the Vision Transformer model, each satellite base map slice patch will be further divided into image blocks of a fixed size after entering the DINOV2 network. Suppose there are N image blocks after division, and the size of each image block is P×P, then we can get:

[0065]

[0066] These image blocks will be flattened into one-dimensional vectors.

[0067] Subsequently, DINOV2 performs a linear projection on each image block, projects it to a fixed dimension, and converts each image block into a high-dimensional feature vector:

[0068]

[0069] Then, the DINOV2 network will add the position encoding of each image block to its embedding vector to ensure that the model learns the spatial relationship. In addition, DINOV2 based on the Vision Transformer will also use a special CLS Token to participate in the Transformer calculation together with the image blocks, and it will aggregate the global features from the entire image:

[0070]

[0071] z 0 ← 0 +E pos

[0072] Among them, E pos represents the position encoding, Zcls 0 The CLS Token representing the aggregated global features.

[0073] After adding the positional encoding and the CLS Token, the above input will be processed by the Transformer Encoder, and the input features will be processed through the multi-head attention mechanism and the feed-forward network. Through the self-attention mechanism, the network can capture the global context information of each image, thus obtaining the initial global feature vector:

[0074] Q = z l W q , K = z l W k , V = z l W v

[0075]

[0076] where D represents the scaling factor to ensure the stability of the inner product value, and the results of multiple heads will be concatenated together and then passed through a linear transformation to obtain the output.

[0077] The initial global vector obtained through the Trasformer Encoder layer will then enter a feed-forward network layer for dimensional transformation. The CLS Token part in the output contains the global features of the entire image:

[0078] FFN(z) = GELU(zW1 + b1)W2 + b2

[0079]

[0080] After each satellite slice patch obtains the global features through the DINOV2 network, the global features will also enter the SALAD module for aggregation. The SALAD module uses the optimal transport method for feature-to-cluster aggregation, and the dimension and number of clusters can be set by itself when using. After the setting is completed, the SALAD module will construct the cost matrix between the global features and the clusters. Through continuous iteration of the sinkhorn algorithm until a corresponding relationship between features and clusters is found to minimize the total cost. Finally, the global features of the image will be aggregated into 64 clusters, and the dimension of each cluster is 128, which further improves the quality and robustness of the global features of the image.

[0081] After the initial extraction of the global features by DINOV2 and the feature aggregation by SALAD, the final global features of each satellite base map slice patch can be obtained.

[0082] Step 3: Indexing and retrieval of feature vectors;

[0083] Using the Faiss library, construct an index for the global feature vectors of each satellite base map slice patch, and store the name of the corresponding satellite base map for each patch and its position on the satellite base map in a CSV file. For each drone image, perform a resize operation to modify the resolution, and extract its global features in the same way through Step 2. Then, the similarity between the global feature vectors of the drone image and each satellite base map slice patch can be calculated. After sorting by similarity, the matching "drone - satellite" image pairs can be found.

[0084] The second stage of this method mainly includes the following steps:

[0085] Step 1: Feature extraction;

[0086] Based on the "drone - satellite" image pairs (A, B) in the first stage, extract key points from the two images through the SuperPoint algorithm, denoted as: A = {A1, A2,..., A N}, B = {B1, B2,..., B M}. Among them, N and M respectively represent the number of key points recognized in the two images. And calculate the local descriptor of each key point: At the same time, perform position embedding and multi - layer perceptron (MLP) processing on the position features of the key points to further refine their spatial positions, and obtain the position embedding: In addition to the SuperPoint descriptor, extract the DINOV2 feature map, and obtain the DINOV2 descriptor of each key point through interpolation operation: Finally, each key point has three features: local descriptor, position feature, and DINOV2 feature, denoted as: for the drone image A: {d A , p A , g A}, for the satellite - view image B: {d B , p B , g B}

[0087] Step 2: Graph construction;

[0088] In this stage, for the input image pairs, construct four key - point association graphs, including two inter - image graphs and two intra - image graphs.

[0089] The inter - image graph calculates the similarity between key points through DINOV2 features and connects the most similar key points, thus avoiding redundant matches. Specifically, the inter - image graph represents the connection relationship from image I A to I B , denoted as G A→B , and the connection relationship from image I B to IA The connection relationship is denoted as G B→A For each pair of key points A i and B j , we calculate the similarity between their DINOV2 features, expressed as:

[0090]

[0091] The intra-image graph is formed by connecting all key points within the same image, creating an undirected graph. The connections between each key point and other key points ensure that information can be fully propagated within the same image and between different images. Specifically, within the same image, the connection relationship between all key points is represented by an undirected graph. The connection between each key point A i and A j is represented by the following similarity metric:

[0092]

[0093] Step 3: Information propagation;

[0094] At this stage, the key points are updated through self-attention layers and cross-image attention layers. The self-attention layer processes the intra-image graph and updates the features of the key points within the same image. The cross-image attention layer processes the inter-image graph and updates the key point features based on the key point matching information guided by DINOV2. Additionally, a position-guided attention mechanism is introduced to decouple the position information and local descriptors of the key points, enabling the model to rely only on the local descriptors when updating features while using the position information as spatial context. Specifically, the attention layer is defined in the following form:

[0095]

[0096] As described in Equation (1), the attention has a residual connection that integrates the attention update value ← is the update operation, ·|· is channel-wise concatenation. To calculate the attention update value, as shown in Equation (2), we calculate the feature similarity between key point A i and the key points connected to the source in the graph, where the graph contains K key points. The query, key, and value of the attention are k S and v S respectively. Specifically, as shown in Equations (3)-(5), the query and key are calculated by fusing the local descriptor and position information. While the value is only transformed from the local descriptor. We note that the weights W and biases b that map features to the query, key, and value are not shared between different attention layers. In the self-attention G A and G B , S contains all key points; in the cross-attention G A→B and GB→A where S contains the key points determined by DINO.

[0097] Step 4: Key point matching;

[0098] In the matching stage, the model uses the updated key point features to calculate the similarity matrix between key point pairs where the matching similarity is optimized by the Sinkhorn algorithm to obtain a matching matrix M ∈ [0, 1] N×M where M i，j represents the matching probability between key points A i and B j representing the matching probability of each pair of key points. Finally, the model is trained using a binary classification task by minimizing the negative log-likelihood loss function of the matching matrix, enabling the model to accurately match key points between different images.

[0099] Example:

[0100] The present invention provides a scene matching algorithm based on optimal transport, and the whole method is as Figure 1 shown. The specific process is as follows.

[0101] The first stage:

[0102] 1. Data preprocessing;

[0103] Input 11 high-resolution satellite base maps corresponding to the UAV-VisLoc dataset constructed using real drone shots and real satellite images. Set the width and height dimensions of each base map slice to (1500, 1000), and the moving step size of the sliding window to (600, 400). Then slice the 11 satellite base maps. Resize each satellite base map slice patch and perform perspective enhancement. Then input the drone image samples corresponding to 11 scenes in UAV-VisLoc and resize each drone image. The resized dimensions of the above base map slices patch and drone images are both 322×322, serving as the input for the feature extraction part.

[0104] 2. Global feature extraction;

[0105] Input each satellite base map slice patch into the DINOV2 network to obtain a preliminary global feature vector f sate1 . The input dimension is (1, 3, 322, 322), representing the number of pictures processed each time, the number of channels of the image, the length and width of the image respectively. The dimension of the feature vector output after passing through the feed-forward network is 768. Subsequently, the output feature vector is passed through the SALAD layer for feature aggregation to obtain the final global feature vector f sate2The dimension of the feature cluster during convergence is 64.

[0106] 3. UAV satellite image retrieval;

[0107] Using the faiss library, index the global feature vectors of each satellite slice patch. And write the information of all satellite slice patches (the name of the base map, the position in the base map) into a csv file. For each UAV image, calculate the similarity inner product between its global feature vector and the global feature vectors of each satellite slice in the index library. Select the index of the satellite slice with the highest similarity, find its corresponding specific information in the csv file, and return the matching "UAV - satellite slice" image pair.

[0108] The second stage:

[0109] 1. Feature extraction;

[0110] Input the UAV image and the satellite slice image (both with a size of 322×322) into the Superpoint network to obtain sp_features0 and sp_features1. Among them, sp_features0[0] and sp_features1[0] are arrays of key point coordinates, with a shape of (N,2), where N is the number of key points in the image. sp_features0[1] and sp_features1[2] are arrays of descriptors, with a shape of (N,D), where D is the dimension of the descriptor. sp_features0[2] and sp_features1[2] are the confidence scores for generating key points, with a shape of (N,), representing the confidence of each key point. Then input the image pair into the DINOV2 network, with an input dimension of (1, 3, 322, 322), representing the number of pictures processed each time, the number of channels of the image, the length and width of the image respectively. After passing through the feed-forward network, the output feature vectors dino_features0 and dino_features1 have a dimension of (N,768), where N is the number of key points.

[0111] 2. Graph construction;

[0112] Using the extracted SuperPoint features sp_features0, sp_features1 and DINO features dino_features0, dino_features1, construct the intra-image and inter-image association graphs according to the formula in step 2 of the second stage in the invention content to initialize the key point features and obtain the intra-image association graph: G A ,G B , with dimensions of (N,768), (M,768) and the inter-image association graph: G A→B, GB→A , with dimensions (M, N). During this process, DINOV2 features are used to filter out potentially matching key points, thereby reducing the matching space and ensuring that subsequent matching only focuses on the potentially matching key points.

[0113] 3. Information dissemination;

[0114] Based on G A , G B , G A→B , G B→A , an attention mechanism is used for feature update. Input G A , G B into the self-attention module. Through the self-attention mechanism, the key point features within the image are updated. In self-attention, each input key point will interact with all other key points in the same image, thereby updating their features. The attention weighted value is calculated by computing the similarity between the query, key, and value. The output dimension of the self-attention layer is the same as the input dimension, still (N, 768). Input G A→B , G B→A into the cross-attention module. In cross-attention, each key point in image A only interacts with the potentially matching key points in image B, thereby updating the features of the key points in image A. Similarly, the attention value is calculated through the query, key, and value mechanism, but the query comes from the key points in image A, and the key and value come from the potentially matching key points in image B. The output dimension of cross-attention also remains consistent with the input dimension.

[0115] 4. Key point matching;

[0116] Based on the above feature acquisition and the construction of the association graph, construct the input tensor dictionary (width0, height0, width1, height1, sp_features0, sp_features1, dino_descriptors0, dino_descriptors1, GA, GB, GA_B, GB_A), calculate the similarity matrix S with dimensions (M, N), use the Sinkhorn algorithm to optimize the matching similarity, and obtain the probability matching matrix D ∈ [0, 1] M×N, this matrix represents the matching probability between each pair of key points. According to the set threshold MATCH_THRESHOLD, it is converted into a matching matrix with dimension (N, M) using the soft_assignment_to_match_matrix function. Subsequently, combined with the confidence scores extracted by SuperPoint, the matching points with confidence greater than zero are filtered out. The final output includes three arrays: match_kp0s, match_kp1s, and match_confidences. match_kp0s and match_kp1s respectively represent the matching key points in the drone image and the satellite slice image, both with dimension (K, 2), where K is the number of successfully matched key point pairs. match_confidences is the confidence for each pair of matches, with dimension (K,).

Claims

1. A two-stage unmanned aerial vehicle-satellite scene matching method, characterized in that It includes the following steps: Step 1: Data preprocessing and dataset loading; Load the satellite base map in the dataset, and after data preprocessing, use it as the input for the feature extraction part; Step 2: Global feature extraction; Step 3 : Indexing and retrieval of feature vectors to obtain matching "drone - satellite" image pairs; Step 4: Feature extraction based on the "drone - satellite" image pairs obtained in Step 3; Step 5: Graph construction; Step 6: Information propagation; Step 7: Key point matching.

2. The two-stage UAV-satellite scene matching method according to claim 1, wherein The specific content of Step 1 is as follows: For each satellite base map, use the sliding window method to set the size of the patch and the size of the sliding step, and cut the satellite base map into several patches; for each patch, use the resize method to adjust its resolution; Subsequently, use the RandomPerspective method in the torchvision.transforms library to perform perspective enhancement on each patch and perform normalization processing, which is used as the input for the feature extraction part.

3. The method for drone-satellite scene matching based on two stages according to claim 2, wherein, The specific content of Step 2 is as follows: Step 2 - 1: Input each satellite base map slice patch in Step 1 into the backbone network DINOV2 in the form of (1, C, H, W), where 1 represents that the batch_size of the image to be extracted is 1; C represents the number of image channels; H and W respectively represent the width and height of the image; each satellite base map slice patch is further divided into fixed - size image blocks after entering the backbone network DINOV2; assume that there are N image blocks after division, and the size of each image block is P×P, and we get: Step 2 - 2: The backbone network DINOV2 performs linear projection on each image block, projects it to a fixed dimension, and converts each image block into a high - dimensional feature vector: Among which W p is the weight matrix of the linear projection, b p is the bias vector, z i 0 is the embedded representation of the i-th image patch, x i is the embedded representation of the i-th image patch; Step 2 - 3: The backbone network DINOV2 adds the position encoding of each image block to its embedding vector to ensure that the model learns spatial relationships; use the CLS Token to participate in the Transformer calculation together with the image blocks to aggregate the global features from the entire image: z 0 ←z 0 +E pos Among them, E pos represents the position encoding, and Z cls 0 represents the CLS Token that aggregates the global features; Step 2 - 4: After adding the position encoding and the CLS Token, process the input features through the Transformer Encoder, and then use the multi - head attention mechanism and the feed - forward network; Through the self - attention mechanism, capture the global context information of each picture, thereby obtaining the initial global feature vector: Q = z l W q ,K = z l W k ,V = z l W v Among them, W q , W k , W v are the linear weight change matrices of the attention layer, D represents the scaling factor, Q, K, and V respectively represent the matrices of query, key, and value in the attention layer, and z l represents the input embedding vector of the l-th layer of attention; The backbone network DINOV2 adopts the multi - head self - attention mechanism. There are multiple parallel attention layers in the network, and the results of multiple layers are concatenated together and then obtained through linear transformation; The initial global vector obtained through the Trasformer Encoder layer enters a feed - forward network layer for dimension change, and the CLS Token part in the output contains the global features of the entire image: FFN(z) = GELU(zW1 + b1)W2 + b2 Among them, FFN represents the feed-forward network layer, Z cls L represents the CLS Token part, GELU represents the activation function, z represents the initial global vector obtained through the Transformer Encoder layer, W1 represents the weight matrix of the first linear layer of the FFN, W2 represents the weight matrix of the second linear layer of the FFN, b1 represents the bias vector of the first linear layer, b2 represents the bias vector of the second linear layer, f global represents the finally extracted global image feature; Step 2-5: Introduce the local feature aggregation algorithm SALAD based on the optimal transport sinkhorn algorithm to perform feature aggregation on the features extracted by DINOV2; after each satellite slice patch obtains global features through the backbone network DINOV2, the global features enter the SALAD module for aggregation; the SALAD module constructs the cost matrix between the global features and the clusters; through continuous iteration of the sinkhorn algorithm until a correspondence between features and clusters is found to minimize the total cost; finally, the global features of the image will be aggregated into 64 clusters, and the dimension of each cluster is 128; After the initial extraction of the global features by the backbone network DINOV2 and the feature aggregation of the SALAD module, the final global features of each satellite base map slice patch are obtained.

4. A two-stage UAV-satellite scene matching method according to claim 3, characterized in that The value of C is 3.

5. A two-stage UAV-satellite scene matching method according to claim 4, characterized in that, The specific content of Step 3 is as follows: Using the faiss library, construct an index for the global feature vectors of each satellite base map slice patch, and store the name of the corresponding satellite base map of each patch and its position on the satellite base map in a csv file; for each drone image, perform a resize operation to modify the resolution, and extract its global features in the same way through Step 2. Calculate the similarity of the global feature vectors between the initial drone image and each satellite base map slice patch, and after sorting according to the similarity, the matching "drone - satellite" image pairs can be found.

6. The method for matching images of an unmanned aerial vehicle and a satellite based on two stages according to claim 5, wherein The specific content of Step 4 is as follows: Based on the "drone - satellite" image pair (A, B), extract key points from the two images through the SuperPoint algorithm, denoted as: A = {A1, A2,..., A N}, B = {B1, B2,..., B M}, where N and M respectively represent the number of key points identified in the two images; and calculate the local descriptors of each key point: Meanwhile, perform position embedding and multi - layer perceptron processing on the position features of the key points to further refine their spatial positions, obtaining the position embedding: In addition to the SuperPoint descriptor, extract the DINOV2 feature map and obtain the DINOV2 descriptor of each key point through interpolation operations: Finally, each key point has three types of features: local descriptor, position feature, and DINOV2 feature, denoted as: for the drone image A: {d A , p A , g A}, for the satellite - perspective image B: {d B , p B , g B}.

7. A two-stage drone-satellite scene matching method according to claim 6, characterized in that, The specific content of Step 5 is as follows: For the input image pairs, construct four key point association graphs, including two inter-image graphs and two intra-image graphs; The inter-image graph calculates the similarity between key points through DINOV2 features and connects the most similar key points to avoid redundant matching. Specifically, the inter-image graph represents the connection relationship from image I A to I B , denoted as G A→B , and the connection relationship from image I B to I A , denoted as G B→A . For each pair of key points A i and B j , calculate the similarity between their DINOV2 features, expressed as: The in-image graph is formed by connecting all key points within the same image to form an undirected graph; the connections between each key point and other key points ensure that information can be fully propagated within the same image and between different images; specifically, within the same image, the connection relationships between all key points are represented by an undirected graph, and each key point A i and A j The connection between them is represented by the following similarity metric:

8. A two-stage UAV-satellite scene matching method according to claim 7, characterized in that, The specific content of Step 6 is as follows: Update the key points through the self-attention layer and the cross-image attention layer; the self-attention layer processes the intra-image graph and updates the features of the key points within the same image; The cross-image attention layer processes the inter-image graph and updates the key point features according to the key point matching information guided by DINOV2; In addition, introduce a position-guided attention mechanism to decouple the position information and local descriptors of the key points, enabling the model to rely only on the local descriptors when updating features, while using the position information as spatial context; specifically, the attention layer is defined in the following form: Among them, S represents the key set related to set A; K is a scalar representing the number of key points in set S; C is a scalar representing the number of feature channels; represents the local feature descriptor extracted by the superpoint of the i-th key point in set A; represents the position encoding of the i-th key point in set A in image A; d S represents the set of local feature descriptors of set S, containing K feature points, and the dimension of each descriptor is C; p S represents the position encoding matrix of all key points in set S, with a dimension of K×C; W q , W k , W v represents the projection matrix used to transform the input features into the Query / Key / Value space of the transformer; b q , b k , b v are all bias terms used to adjust the output of Query / Key / Value; represents the query vector generated by the transformer layer according to and ; k s and v s represent the key vector and the value vector respectively.

9. A two-stage UAV-satellite scene matching method according to claim 8, characterized in that, The specific content of Step 7 is as follows: In the matching stage, the model calculates the similarity matrix between key-point pairs using the updated key-point features Among them, The matching similarity is optimized by the Sinkhorn algorithm to obtain a matching matrix M ∈ [0, 1] N×M , where M i，j represents the matching probability between key-point A i and B j ; that is, it represents the matching probability of each pair of key-points. Finally, the binary classification task is used to train the model by minimizing the negative log-likelihood loss function of the matching matrix, so that the model can accurately match key-points between different images.

Images