A method for scene matching of unmanned aerial vehicle images and satellite images

Abstract

The present application relates to a kind of unmanned aerial vehicle image and satellite image scene matching method, the method comprises the following steps: firstly, the input scene matching model of image and corresponding satellite image captured by unmanned aerial vehicle is matched, and preprocessed by data enhancement module;Second, the feature extraction module is extracted to the image, and feature integration is carried out by Head module;Then, by representation learning, metric learning and mutual learning three kinds of supervised learning method training model;Finally, by calculating the cosine similarity between unmanned aerial vehicle and satellite image is matched.Compared with prior art, by improving the accuracy and robustness of scene matching model, the problem that unmanned aerial vehicle image and satellite image are prone to no matching even matching failure in matching process is effectively reduced.

Description

Technical Field

[0001] This invention relates to the field of scene matching technology, and in particular to a method for scene matching of UAV images and satellite images. Background Technology

[0002] To maintain high-precision positioning for UAVs in GNSS-denied scenarios, scene matching technology is currently the most commonly used method. Scene matching is a computer vision technique that identifies or finds the correspondence between feature images from corresponding scene regions captured by different sensors. UAVs determine their own position information by matching real-time captured images with a pre-processed and stored database of remote sensing reference images, calculating the position of the matching image within the remote sensing image.

[0003] However, scene matching technology heavily relies on offline remote sensing baseline maps. Therefore, high-precision maps are one of the key factors affecting the accuracy of scene matching. Since the baseline map is created before the real-time UAV imagery, the baseline map and the real-time UAV imagery may have been obtained on different dates, using different sensors, at different observation points or angles, and under different weather conditions. Therefore, scene matching algorithms need to have strong invariance to illumination, seasons, and angles. Current traditional algorithms are largely unable to cope with changes caused by different sensors, time, seasons, and angles, while deep learning-based matching algorithms, due to their powerful ability to automatically extract features, have better robustness.

[0004] Although deep learning-based matching algorithms have significant advantages over traditional algorithms, some problems still need to be solved: the imaging conditions of the benchmark satellite images and UAV images with geographic information are different, and the domain differences in this heterogeneous image matching can lead to matching failures or a large number of mismatches. Summary of the Invention

[0005] To overcome the aforementioned deficiencies of the prior art, the present invention provides a scene matching method for UAV images and satellite images.

[0006] The objective of this invention can be achieved through the following technical solutions:

[0007] A method for scene matching between drone images and satellite images, the method comprising the following steps:

[0008] Step 1: Match the images captured by the UAV with the corresponding satellite images in the scene matching model, and preprocess them through the data augmentation module;

[0009] Step 2: Feature extraction is performed on the image using a feature extraction module, which includes a backbone network and an SPKA module. First, image features are extracted using the backbone network. Then, the extracted features are further refined using the SPKA module to obtain better features, and finally, feature integration is performed using the Head module.

[0010] Step 3: Train the model using three supervised learning methods: representation learning, metric learning, and mutual learning;

[0011] Step 4: Matching is performed by calculating the cosine similarity between the drone and the satellite image.

[0012] Furthermore, in step 2, the specific method is as follows:

[0013] Step 2-1: Input the image into the backbone network. The backbone network adopts the VisionTransformer network model proposed by Alexey et al. to extract image features.

[0014] Step 2-2: Extract salient location features from the features extracted from the backbone network using the SPKA module;

[0015] Steps 2-3: Integrate the extracted features using the Head module.

[0016] Furthermore, in step 3, the specific method is as follows: when in the training phase, the vector output by the feature extraction module maps the feature dimension to the number of categories through a fully connected layer, and normalizes the output value through a softmax layer. The weight parameters obtained by the scene matching model during training are optimized based on a multi-supervised approach using cross-entropy loss function, soft-weighted triplet loss, and KL divergence loss.

[0017] Furthermore, in step 4, the specific method is as follows: the feature vectors of the obtained satellite image and the feature vectors of the UAV image are measured by cosine similarity, and the feature vector pair with the closest distance is used as the match;

[0018] Compared with the prior art, the present invention has the following beneficial effects:

[0019] 1. This invention improves the accuracy and robustness of scene matching models by designing a scene matching model based on salient location features, and effectively reduces the problem of no matching or even matching failure in the matching process of UAV images and satellite images;

[0020] 2. This invention improves model performance through a multi-supervision approach by integrating cross-entropy loss, soft-weighted triplet loss, and KL divergence loss. Attached Figure Description

[0021] Figure 1 This is an architecture diagram of the scene matching model of the present invention;

[0022] Figure 2 This is a retrieval result diagram of the scene matching model of the present invention;

[0023] Figure 3 This is a comparison chart of the retrieval results between the scene matching model and the baseline model of the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and beneficial effects of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0025] Example 1: As Figure 1 This invention provides a scene matching method for UAV images and satellite images. By extracting salient location features, it improves the accuracy and robustness of the model and reduces the problem of frequent matching failures or numerous mismatches between UAV images and satellite images. The specific steps are as follows:

[0026] Step 1: Match the images captured by the UAV with the corresponding satellite images in the scene matching model, and preprocess them through the data augmentation module;

[0027] Step 2: Feature extraction is performed on the image using a feature extraction module, which includes a backbone network and an SPKA module. First, image features are extracted using the backbone network. Then, the extracted features are further refined using the SPKA module to obtain better features, and finally, feature integration is performed using the Head module.

[0028] Step 3: Train the model using three supervised learning methods: representation learning, metric learning, and mutual learning;

[0029] Step 4: Matching is performed by calculating the cosine similarity between the drone and the satellite image.

[0030] Furthermore, in step 1, the specific method is as follows: first, both the drone and satellite images are reset to 224*224 images, and then image enhancement is performed using random rotation, random brightness, and random erasure;

[0031] Furthermore, in step 2, the specific method is as follows:

[0032] Step 2-1: Input the image into the backbone network. The backbone network adopts the VisionTransformer network model proposed by Alexey et al., and after extracting the image features, it is sent to the SPKA module.

[0033] Step 2-2: The features extracted from the backbone network are used to extract salient location features using the SPKA module proposed by Fang et al. The SPKA module works as follows: For the input feature X, a corresponding query matrix Q and value matrix V are generated through linear projection; the SPS algorithm is applied to Q to select salient locations, obtaining a salient location matrix K; and the corresponding query matrix Q is generated by linear projection onto K. k Bond matrix K k Then, the KNN attention mechanism is used, which calculates the similarity between each query vector and all key vectors, selects the k most similar keys for each query, and then uses these selected keys to construct a sparse attention matrix A. knn Finally, the output features weighted by the KNN attention mechanism are reconstructed back to the original feature map dimensions. SPS stands for Salience Position Selection Algorithm. The specific method is as follows: First, calculate the square of the transpose of Q; second, sum it along the channel dimension, denoted as Qpow; third, identify the k most salient positions with the highest values ​​from Qpow, and denote the indices of these k positions as indexk; finally, construct a new matrix, denoted as K, by extracting the columns corresponding to the selected indices (indexk) from the original matrix Q. The resulting matrix K has dimensions [c,k], where c represents the number of channels and k represents the number of selected salient positions. The SPS algorithm reduces data dimensionality while preserving the most salient features.

[0034] Steps 2-3: Integrate the extracted features through the Head module and compress them to a specific dimension. Then, use the global pooling method to convert the 768-dimensional features output by the feature extraction module into a 512-dimensional feature vector for classification.

[0035] Further, in step 3, the specific method is as follows: During the training phase, the selected dataset is the recently released DenseUAV dataset, which includes drone images and satellite images. The dataset is divided into a training set and a test set. The training set includes 6768 drone view images and 13536 satellite view images captured from 2256 sampling points at ten universities; the test set includes 2331 drone view images and 4662 satellite view images from 777 sampling points at four universities. During training, the vector output by the feature extraction module is mapped to the number of categories (2256) through a fully connected layer, and the output values ​​are normalized through a softmax layer. The performance of the scene matching model is improved based on a joint loss function of cross-entropy loss, soft-weighted triplet loss, and KL divergence loss.

[0036] The formula for calculating the cross-entropy loss function is as follows:

[0037]

[0038]

[0039] Where q i This represents the result of the model output logits after softmax, p i This indicates the corresponding tag.

[0040] The formula for calculating the soft-weighted triplet loss is as follows:

[0041] SWTriLoss(a,p,n)=log(1+e α×(D(a,p)-D(a,p) ))

[0042] Where a is the feature vector of the anchor sample, p is the positive sample feature vector of the anchor sample, n is the negative sample feature vector, m is the interval that controls the expected difference in distance between positive and negative samples, and D(a,b) represents the cosine similarity between samples a and b.

[0043] The formula for calculating the KL divergence loss is as follows:

[0044]

[0045] KLLoss=KLDiv(O d ||O s )+KLDiv(O s ||O d )

[0046] Among them O p and O q Let O represent the probability distributions of the teacher and student class vectors respectively through softmax. d This represents the class vector output of the drone image, while O s The output represents the class vector of the satellite image.

[0047] Furthermore, in step 4, the specific method is as follows: the feature vectors of the obtained satellite image and the feature vectors of the UAV image are measured by cosine similarity, and the feature vector pair with the closest distance is used as the match.

[0048] The formula for calculating the cosine similarity is as follows:

[0049]

[0050] The feature vector of the UAV image is represented as follows: The feature vector of a satellite image is defined as i represents f d or f s The i-th element.

[0051] This invention addresses the issue of mismatches or no matching often occurring in the matching of UAV and satellite images due to their different imaging conditions and domain differences. It addresses this problem by designing a scene matching model based on a Siamese network with salient location features. The Vision Transformer network model proposed by Alexey et al. is used as the backbone network to extract image features, and then the SPKA module is used to extract more effective salient location features. This improves the model's accuracy and robustness, effectively reducing the problems of no matching or even matching failures that easily occur during the matching process between UAV and satellite images. Figure 2 and Figure 3 The image to the left of the line is from a drone, and the image to the right is from a retrieved satellite image. Images in red are mismatched. Figure 3 The above are the search results of the present invention, and the below are the search results of the baseline model. The first-ranked results of the scene matching method of the present invention are all correct matches. Compared with the baseline model, which has the problem of no match or even failure to match, the scene matching method of the present invention can match successfully and has fewer false matches.

[0052] It should be noted that the above content merely illustrates the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. For those skilled in the art, various improvements and modifications can be made without departing from the principle of the present invention, and all such improvements and modifications fall within the scope of protection of the claims of the present invention.

Claims

1. A method for scene matching between UAV images and satellite images, characterized in that, Includes the following steps: Step 1: Match the images captured by the UAV with the corresponding satellite images in the input scene model, and preprocess them using the data augmentation module; Step 2: Extract features from the image using the feature extraction module, and integrate the features using the Head module; Step 3: Train the model using three supervised learning methods: representation learning, metric learning, and mutual learning; Step 4: Matching is performed by calculating the cosine similarity between the drone and the satellite image; In step 2, the specific method is as follows: Step 2-1: The feature extraction module includes a backbone network and SPKA, a Salient Positions based KNNAttention module. First, the image features are extracted through the backbone network Vision Transformer, and then the extracted features are extracted through the SPKA module to obtain salient position features. Step 2-2: Extract salient location features from the backbone network using the SPKA module. The SPKA module works as follows: For the input feature X, a corresponding query matrix Q and value matrix V are generated through linear projection. The SPS algorithm is applied to Q to select salient locations, resulting in a salient location matrix K. A corresponding query matrix is ​​then generated by linear projection onto K. Bond matrix Then, the KNN attention mechanism is used, which calculates the similarity between each query vector and all key vectors, selects the k most similar keys for each query, and then uses these selected keys to construct a sparse attention matrix. Finally, the output features weighted by the KNN attention mechanism are reshaped back to the original feature map dimension. SPS stands for Salience Position Selection Algorithm. The specific method is as follows: First, calculate the square of the transpose of Q; second, sum it along the channel dimension, denoted as Qpow; third, identify the k most salient positions with the highest values ​​from Qpow, and denote the indices of these k positions as indexk; finally, construct a new matrix, denoted as K, by extracting the column corresponding to the selected indexk from the original matrix Q. The dimension of the resulting matrix K is [c, k], where c represents the number of channels and k represents the number of selected salient positions. The SPS algorithm retains the most salient features while reducing the data dimension.

2. The scene matching method for UAV images and satellite images according to claim 1, characterized in that, In step 1, the scene matching model includes a data augmentation module, a feature extraction module, a head module, and a matching module. The specific method of the data augmentation module is as follows: first, both the UAV and satellite images are reset to 224*224 images, and then random rotation, random brightness, and random erasure are used to enhance the images.

3. The scene matching method for UAV images and satellite images according to claim 1, characterized in that, In step 3, the specific method is as follows: During training, the vector output by the feature extraction module is mapped to the number of categories (2256) through a fully connected layer, and the output value is normalized through a softmax layer. The weight parameters obtained during training of the scene matching model are optimized based on a joint loss function of cross-entropy loss, soft-weighted triplet loss, and KL divergence loss. The formula for calculating the cross-entropy loss function is as follows: in This represents the result of the model output logits after applying softmax. This indicates the corresponding tag. The formula for calculating the soft-weighted triplet loss is as follows: Where a is the feature vector of the anchor sample, p is the positive sample feature vector of the anchor sample, n is the negative sample feature vector, m is the margin controlling the expected difference in distance between positive and negative samples, and D(a,b) represents the cosine similarity between samples a and b. The formula for calculating the KL divergence loss is as follows: in and Let represent the probability distributions of the teacher and student category vectors respectively through softmax. Furthermore, The output represents the class vector of the drone image, while The output represents the class vector of the satellite image.

4. The scene matching method for UAV images and satellite images according to claim 1, characterized in that, In step 4, the specific method is as follows: the feature vectors of the obtained satellite image and the feature vectors of the UAV image are measured using cosine similarity, and the feature vector pair with the closest distance is selected as the match. The formula for calculating the cosine similarity is as follows: The feature vector of the UAV image is represented as follows: The feature vector of a satellite image is defined as , express or The Each element.

Images