A cross-view scene matching method for uav images and satellite images

Abstract

The application discloses a kind of for unmanned aerial vehicle image and satellite image cross view scene matching method, including obtaining at least thousands of unmanned aerial vehicle graph and corresponding scene satellite graph as data set respectively, unmanned aerial vehicle graph in data set is converted into feature vector, satellite graph is also converted into feature vector, scene matching model based on twin neural network is trained according to these feature vectors, and the scene matching model trained is obtained etc.Step, the present application is designed based on the hybrid architecture neural network model of local feature and global feature, the technical effect of improving the precision of model inference stage and greatly improving the robustness under different environment and attitude is realized by HBP strengthening feature representation, based on the joint measurement loss function of Identityloss and Circleloss, the technical effect of improving the convergence speed of scene matching model is realized, and better weight parameter is learned to the technical effect of helping scene matching model.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of unmanned aerial vehicle navigation and positioning, and particularly discloses a cross-view scene matching method for unmanned aerial vehicle images and satellite images. BACKGROUND

[0002] The purpose of cross-view image matching is to match images from the same target scene but different views, so as to infer the location of the place. In recent years, with the rapid development of unmanned aerial vehicle technology, how to help unmanned aerial vehicles navigate through cross-view matching technology has become a research hotspot. The common cross-view matching process is as follows: a picture to be searched is given in a certain view of the search data set, and the matching system can find the picture under the same place in the candidate data set of another view. For cross-view matching under satellite and unmanned aerial vehicle platforms, two tasks need to be mainly solved: 1. Unmanned aerial vehicle positioning: given an image in the view of an unmanned aerial vehicle, find the most matched satellite view image. 2. Unmanned aerial vehicle navigation: given an image in the view of a satellite, find the most matched unmanned aerial vehicle image. Therefore, the core of the cross-view matching technology is to train a deep learning model to learn the invariant features of images in different views, so as to improve the accuracy and robustness of the matching system.

[0003] The cross-view matching algorithm has experienced a development process from the initial extraction of point feature descriptors of images to the extraction of global features of images through deep convolutional networks. VLAD (vector of locally aggregated descriptors) is a classic image retrieval algorithm and can also be applied to the field of cross-view scene matching. VLAD constructs a vector by aggregating point local descriptors, uses the vector as a global descriptor of the image, and then performs optimal feature matching on the image according to the feature vector. Arandjelovic proposed NetVLAD in 2016, which improved VLAD by using a neural network. The local features described by the hand-craft descriptor in VLAD are replaced by global features extracted by a convolutional neural network, and the non-differentiable hard-assignment part in VLAD is replaced by a differentiable softmax function, so that NetVLAD can optimize the parameters through training.

[0004] With the foundation laid by deep convolutional neural networks in recent years, deep learning-based models are widely used to extract image features. Workman's 2015 study found that features extracted by deep convolutional networks have high distinguishability and are easy to calculate. Experimental results show better performance than point features on the Place dataset, indicating the direction for future cross-view matching research. With the release of the University1652 dataset, cross-view matching algorithms based on unmanned aerial vehicle and satellite perspectives have made some progress in the past year. Ding proposed LCM (a cross-view matching method based on location classification) in 2021. During the training phase, LCM treats image retrieval as a classification problem and considers the impact of different feature sizes on the final results. The results show that LCM improves the baseline model of University1652 by 5-10%.

[0005] Wang considered the context information of the adjacent area in 2021 and proposed the LPN (Local Pattern Network) deep neural network, which uses a square ring feature segmentation strategy to improve attention based on the distance from the image center. This strategy has good robustness to rotation changes and greatly improves the matching efficiency. Tian proposed a method that considers the spatial correspondence between satellite views and surrounding areas in 2021. This method consists of two parts: 1. Convert the unmanned aerial vehicle's oblique view to a vertical view through perspective transformation. 2. Use a conditional generative adversarial network to make the unmanned aerial vehicle's view closer to the satellite image. Experimental results show that this method improves the LPN by 5%.

[0006] Defects and deficiencies of the prior art:

[0007] 1. Traditional feature extraction algorithms that match by extracting image corner features have poor robustness and are easily affected by factors such as light and occlusion. Especially when the unmanned aerial vehicle flies at a low altitude, the difference between the collected image and the satellite view is too large, often causing the matching system to fail or a large number of false matches.

[0008] 2. The matching accuracy of existing deep learning methods is still low, and most networks still use hard attention strategies, which can easily lead to the loss of image semantic information and negatively affect subsequent feature matching results. SUMMARY

[0009] The present application aims to overcome the defects of the prior art and provides a cross-view scene matching method for unmanned aerial vehicle images and satellite images.

[0010] The purpose of the present application can be achieved by the following technical solutions:

[0011] A cross-view scene matching method for unmanned aerial vehicle images and satellite images, the method comprising the following steps:

[0012] Step 1: Obtain at least a thousand unmanned aerial vehicle images and corresponding satellite images of the scene as a data set, respectively;

[0013] Step 2: Convert the unmanned aerial vehicle images in the data set into feature vectors, and also convert the satellite images into feature vectors, train a scene matching model based on a twin neural network according to these feature vectors, and obtain a trained scene matching model;

[0014] Step 3: Preprocess the unmanned aerial vehicle images according to the preprocessing layer of the trained scene matching model to obtain preprocessed unmanned aerial vehicle images, and also preprocess the satellite images to obtain preprocessed satellite images;

[0015] Step 4: Input the preprocessed unmanned aerial vehicle images into the backbone network layer of the scene matching model to perform scene feature extraction on the unmanned aerial vehicle images and obtain primary unmanned aerial vehicle feature maps, and input the preprocessed satellite images into the backbone network layer of the scene matching model to also perform scene feature extraction on the satellite images and obtain primary satellite feature maps;

[0016] Step 5: Strengthen the unmanned aerial vehicle feature maps according to the HBP module and input them into the MLP layer to unify the feature dimension, to obtain strengthened unmanned aerial vehicle feature maps, and also strengthen the satellite feature maps according to the HBP module and input them into the MLP layer to unify the feature dimension, to obtain strengthened satellite feature maps;

[0017] Step 6: Calculate the feature distance between the strengthened unmanned aerial vehicle feature maps and the strengthened satellite feature maps, and perform feature matching to realize unmanned aerial vehicle positioning and navigation tasks.

[0018] As a further improvement, in step 2 of the present application, the unmanned aerial vehicle images in the data set are converted into feature vectors, and the satellite images are also converted into feature vectors, a scene matching model based on a twin neural network is trained according to these feature vectors, and a trained scene matching model is obtained, the scene matching model comprises a preprocessing layer, a feature extractor, and a matching layer, and the feature extractor adopts a hybrid architecture of CNN and Transformer layers.

[0019] As a further improvement, in step 3 of the application, the process of preprocessing the UAV image according to the preprocessing layer of the trained scene matching model to obtain the preprocessed UAV image and preprocessing the satellite image to obtain the preprocessed satellite image is specifically as follows:

[0020] The size of the image is unified, and random patching and random horizontal flipping image enhancement operations are adopted, random patching is performed at a random position of the image and cutting is performed according to a set size, and random horizontal flipping is performed by a certain probability to horizontally flip the image.

[0021] As a further improvement, in step 4 of the application, the process of inputting the preprocessed UAV image into the backbone network layer of the scene matching model to perform scene feature extraction on the UAV image and obtaining a primary UAV feature map, and inputting the preprocessed satellite image into the backbone network layer of the scene matching model to also perform scene feature extraction on the satellite image and obtaining a primary satellite feature map specifically includes the following steps:

[0022] Step 401: input the image into ResNetV2 to output a feature map with a dimension of (1024, 24, 24), square the input feature map, and linearly map the square;

[0023] Step 402: input the vector obtained by linear mapping into 12 consecutive Transformer layers, when in the training stage, the Transformer layers send the output vector to the fully connected layer and the softmax layer of the classification layer for classification and training to obtain the probability of each class, thereby improving the feature extraction capability of the scene matching model in the form of training the classification network, and when in the test stage, the Transformer network block directly exports the features and maps them to 512 dimensions as the feature map of the image.

[0024] As a further improvement, in step 401 of the application, ResNetV2 has undergone 4 Blocks, each Block has 3, 4, 6 and 3 Bottlenecks respectively, and a pre-activation residual unit is used in the Bottleneck to improve the training efficiency of the network, the feature map output by each Block increases the depth of the feature map in the form of a feature pyramid, thereby improving the effective information density when the network structure is calculated, and focusing the network attention on the target scene, i.e., extracting the local features of the image.

[0025] Then, a convolution operation is performed on the feature map using a convolution kernel with a size of 1 and a step of 1, i.e., cutting the feature map into 1x1 squares to obtain a feature vector, so as to convert the feature map into a feature vector that can be processed by the Transformer layer

[0026] As a further improvement, in step 402, the input of the Transformer layer is the grid feature vector, a class token similar to a flag is introduced, and a position embedding is used to number each grid, and then it is sent into the Transformer block. The structure of the Transformer block includes Layer Norm, multi-head attention mechanism, dropout layer and MLP layer. The Transformer layer transfers attention to the global through the multi-head attention mechanism in the grid vector.

[0027] As a further improvement, in step 402, when in the training phase, the vector output by the Transformer layer is mapped to the target scene number to be classified through the neurons of the full connection layer in the classification layer, and the output value is normalized through the softmax layer. Based on the loss function and the joint measurement loss function, the weight parameters of the scene matching model obtained in the training are optimized.

[0028] As a further improvement, the calculation formula of the Cross-entropy loss function is:

[0029]

[0030]

[0031] wherein, is the predicted value of the scene matching model, y is the label value, is the predicted value the predicted probability of being recognized as y class, is the column vector of the label, is the column vector of the sample, i is the index of the total number of scenes, j is the jth vector output by the scene matching model, c is the cth sample in this round of training, and C is the number of training samples.

[0032] The calculation formula of the Circle Loss loss function is:

[0033]

[0034] wherein, s ij indicates whether the sample x i belongs to the category j, d ij indicates whether the sample x iThe Euclidean distance of the center point of the category j, m represents a predefined margin, which can control the closeness of the samples in the same category and the separation degree between different categories, by minimizing the Circle Loss, the samples in the same category can be as close as possible in a circular area, and the distance between different categories can be as far as possible, thereby improving the matching accuracy.

[0035] As a further improvement, in step 5 of the application, the HBP module is used to enhance the feature map of the UAV and input the MLP layer to unify the feature dimension, and the enhanced UAV feature map is obtained, and the HBP module is also used to enhance the satellite feature map and input the MLP layer to unify the feature dimension, and the enhanced satellite feature map is obtained.

[0036] First, the HBP (Hierarchical Bilinear Pooling) module is used to enhance the features output by the backbone network, and the specific process is to input the (1024, 24, 24) feature map output by ResNet and the (1024, 24, 24) feature map output by the backbone network into the HBP module for fusion and enhancement, and the structure of the HBP module is as shown in Figure 2 The implementation of HBP is divided into two steps: bilinear transformation and spatial global pooling. j is represented as the features extracted from the input image, F conv is represented as a CNN module, which includes convolution layers, batch normalization layers and Relu layers, and the calculation method of the specific formula is as follows:

[0037] f j =F conv (f j )

[0038] In the second step, is a global feature, is a local feature, and x represents the outer product of two vectors:

[0039]

[0040]

[0041] Then, the fusion feature is normalized to generate the final feature vector At this time, ∈=1×10 -12 :

[0042]

[0043]

[0044] The HBP combined feature map is input into an MLP (Multilayer Perceptron) module, the structure of which is shown in Figure 3 The Dense represents a fully connected layer, the Gelu represents an activation function, the BN represents batch normalization, and the Drop out represents a partial discard operation. The MLP performs a feature dimension unification operation to unify the dimension of the HBP output feature to 512 dimensions.

[0045] As a further improvement, in step 6, the feature distance between the enhanced UAV feature map and the enhanced satellite feature map is calculated, and the feature matching is performed to realize the UAV positioning and navigation task process, which is specifically:

[0046] The obtained feature vectors of the satellite map and the feature vectors of the UAV map are measured by the Euclidean distance, and the feature vector pair with the closest distance is matched. When performing the UAV positioning task, the distance between one UAV image and all satellite images is calculated, and the vector pair with the closest distance is selected as the matching pair. When performing the UAV navigation task, the distance between one satellite image and all UAV images is calculated, and the vector pair with the closest distance is selected as the matching pair.

[0047] The calculation formula of the Euclidean distance is:

[0048]

[0049] Wherein, p and q represent two n-dimensional vectors, p i and q i represent the i-th element in the vector.

[0050] The present application aims at the technical problem that the existing cross-view matching method is not applicable to the cross-view image matching processing between the UAV and the satellite view. Based on the similarity of the UAV image and the satellite image, the idea of combining local features and global features is used to realize the mutual matching of the satellite view image and the UAV view image. By constructing a deep learning model (scene matching model), the deep features of different images are extracted. The deep features have strong robustness to adverse factors such as rotation and illumination, which can help the UAV to perform positioning and navigation work at low altitude. The scene matching model adopts a hybrid architecture of CNN and Transformer layer as a feature extractor. The CNN can extract local features of the image, while the Transformer layer focuses on extracting global features of the image. The deep learning based on the hybrid architecture can effectively solve the problem of semantic information loss. In particular, the present application also proposes an HBP module and a joint measurement optimization function based on an Identity Loss function and a Circle Loss function to help the deep learning model obtain better parameters during training.

[0051] Compared with the prior art, the present application has the following advantages:

[0052] 1、 The present application adopts an image augmentation technology based on random horizontal flipping and random cropping to prevent overfitting of the scene matching model during training;

[0053] 2、 The present application designs a hybrid architecture neural network model based on local features and global features, and enhances feature representation through HBP to improve the accuracy of the model inference stage and significantly improve the robustness in different environments and poses;

[0054] 3、 The present application designs a joint measurement loss function based on Identity loss and Circle loss to improve the convergence speed of the scene matching model and help the scene matching model learn better weight parameters. BRIEF DESCRIPTION OF DRAWINGS

[0055] Figure 1 is a flowchart of cross-view scene matching

[0056] Figure 2 is a general structure diagram based on ResNet and Transformer backbone network;

[0057] Figure 3 is an HBP module structure diagram;

[0058] Figure 4 is an MLP module structure diagram. DETAILED DESCRIPTION

[0059] The technical solutions of the present application will be further described below by combining with the drawings of the specification and through specific embodiments:

[0060] The present application aims to solve the problems of poor robustness and low matching accuracy of existing scene matching algorithms, and proposes a cross-view scene matching method for unmanned aerial vehicle images and satellite images, Figure 1 is a cross-view scene matching flowchart of the present application:

[0061] Step 1: Obtain at least thousands of unmanned aerial vehicle images and corresponding satellite images of the scene as a data set;

[0062] Step 2: Convert the unmanned aerial vehicle images in the data set into feature vectors, and also convert the satellite images into feature vectors, train a scene matching model based on a twin neural network according to these feature vectors, and obtain a trained scene matching model;

[0063] Step 3: Preprocess the UAV image according to the preprocessing layer of the trained scene matching model to obtain a preprocessed UAV image, and preprocess the satellite image to obtain a preprocessed satellite image;

[0064] Step 4: input the preprocessed UAV image into the backbone network layer of the scene matching model to perform scene feature extraction on the UAV image and obtain a primary UAV feature map, and input the preprocessed satellite image into the backbone network layer of the scene matching model to perform scene feature extraction on the satellite image and obtain a primary satellite feature map;

[0065] Step 5: perform feature enhancement on the UAV feature map according to the HBP module and input the feature map into the MLP layer to unify the feature dimension, to obtain an enhanced UAV feature map, and perform feature enhancement on the satellite feature map according to the HBP module and input the feature map into the MLP layer to unify the feature dimension, to obtain an enhanced satellite feature map;

[0066] Step 6: calculate the feature distance between the enhanced UAV feature map and the enhanced satellite feature map, perform feature matching to realize the UAV positioning and navigation task.

[0067] In step 1, at least thousands of UAV images and corresponding satellite images of the scene are obtained as a data set.

[0068] In this embodiment, after the UAV collects aerial video at multiple target locations, image data is obtained by averaging and frame extraction, and then 17-level satellite tile maps of the same target location are collected as satellite images.

[0069] In step 2, the UAV images in the data set are converted into feature vectors, and the satellite images are also converted into feature vectors, and a scene matching model based on a twin neural network is trained according to the feature vectors to obtain a trained scene matching model.

[0070] In this embodiment, the model is trained by using a stochastic gradient descent, the learning rate is 0.01, the weight decay value is 0.0001, the number of training rounds is 80, and a set of parameters with the minimum loss function value in the training process is saved as the model. The overall structure of the scene matching model is as shown in Figure 2 .

[0071] In step 3, the UAV image is preprocessed according to the preprocessing layer of the trained scene matching model to obtain a preprocessed UAV image, and the satellite image is also preprocessed to obtain a preprocessed satellite image.

[0072] In the preprocessing stage, the twin neural network performs data augmentation on the image to prevent overfitting during model training. Random cropping involves cutting the image at random positions with a size of 128 pixels. This data augmentation method improves the feature extraction ability of the UAV when it only captures one side of the target scene.

[0073] Random horizontal flipping involves flipping the image at a given probability level (randomly). This helps improve the feature extraction ability of the UAV when the angle of the captured image does not align with that of the satellite image.

[0074] In step 4, the preprocessed UAV image is input into the backbone network layer of the scene matching model to extract scene features and obtain a primary UAV feature map. The preprocessed satellite image is also input into the backbone network layer of the scene matching model to extract scene features and obtain a primary satellite feature map. The process includes the following steps:

[0075] Step 401: The image enters the ResNetV2 of the CNN architecture and passes through multiple bottleneck modules, outputting a feature map with dimensions (1024, 24, 24). The input feature map is cut into 1x1 squares, and the squares are linearly mapped.

[0076] Step 402: The vector obtained by linear mapping is input into a 12-layer continuous Transformer network block. If in the training phase, the vector is sent to the fully connected layer and softmax layer for classification and training. If in the testing phase, the features are directly exported and mapped to 512 dimensions as the feature vector of the image.

[0077] In step 401, ResNetV2 passes through 3 bottleneck modules, each with 3, 4, and 9 residual blocks, respectively. In the residual blocks, pre-activation residual units are used to improve the training efficiency of the network. Each bottleneck module outputs a feature map with increasing depth in the form of a feature pyramid, improving the effective information density during network computation and focusing network attention on the target scene, i.e., extracting local features of the image.

[0078] The feature map output by ResNetV2 is then cut into 1x1 squares. The specific operation is to use a convolution kernel size of 1 and a step size of 1 to perform convolution on the feature map, obtaining a feature vector of (1024, 768). Squaring the feature map converts it into a feature vector that the Transformer layer can handle, and shifts the subsequent network attention to the global.

[0079] In step 402, the input of the Transformer layer is the square-ized feature vector, after adding the class token and position embedding, it is sent into a Transformer Block, the structure of the Transformer Block includes Layer Norm, multi-head attention mechanism, dropout layer and MLP layer, in the Transformer layer, through the multi-head attention mechanism, the network transfers attention to the global in the square-ized vector.

[0080] In the training stage, in the present embodiment, the unmanned aerial view image and the satellite view image in the data set are adopted, and are divided into training data and test data two parts, in the network part, the vector output by the Transformer layer is mapped to the target scene number to be classified through the full connection layer of the neurons in the classification layer, then the normalized output value is output through the softmax layer, finally the joint measurement loss function based on Identity Loss and Circle Loss is designed to optimize the weight parameters obtained by the model in the training.

[0081] The calculation formula of Identity Loss is:

[0082]

[0083]

[0084] wherein, is the predicted value of the scene matching model, y is the label value, is the predicted value is the predicted probability of being identified as y class, is the column vector of the label, is the column vector of the sample, i is the index of the total number of scenes, j is the jth vector output by the scene matching model, c is the cth sample in the current round of training, and C is the number of training samples;

[0085] The calculation formula of the Circle Loss loss function is:

[0086]

[0087] wherein, s ij represents whether the sample x i belongs to the category j, d ij represents the distance between the sample x iThe Euclidean distance of the center point of the category j, m represents a predefined margin = 0.1, which can control the closeness of the samples in the same category and the separation degree between different categories, by minimizing the Circle Loss, the samples in the same category can be gathered in a circular region as much as possible, and the distance between different categories is as far apart as possible, thereby improving the matching accuracy.

[0088] In step 5, the HBP module is used to enhance the features of the UAV feature map and input the MLP layer to unify the feature dimension, and the enhanced UAV feature map is obtained, and the HBP module is also used to enhance the features of the satellite feature map and input the MLP layer to unify the feature dimension, and the enhanced satellite feature map is obtained.

[0089] First, the features output by the backbone network are enhanced by the HBP (Hierarchical Bilinear Pooling) module, and the specific process is that the (1024, 24, 24) feature map output by ResNet is input into the HBP module together with the (1024, 24, 24) feature map output by the backbone network for fusion and enhancement. Figure 3 As shown in the structure of the HBP module, j is represented as the features extracted from the input image, F conv is represented as a CNN module, which includes a convolution layer, a batch normalization layer and a Relu layer, and the calculation method of the specific formula is as follows:

[0090] f j =F conv (f j )

[0091] In the second step, is a global feature, is a local feature, and x represents the outer product of two vectors:

[0092]

[0093]

[0094] Then, the fusion feature is normalized to generate the final feature vector At this time, ∈=1×10 -12 :

[0095]

[0096]

[0097] The HBP-merged feature map is then input into a MLP (Multilayer Perceptron) module, the structure of which is shown in Figure 4 The Dense represents a fully connected layer, the Gelu represents an activation function, the BN represents batch normalization, and the Drop out represents a partial dropout operation. The MLP performs a feature dimension unification operation to unify the dimension of the HBP output feature to 512.

[0098] In step 6, the feature distance between the enhanced UAV feature map and the enhanced satellite feature map is calculated, and the process of performing feature matching to achieve the UAV positioning and navigation task includes the following steps:

[0099] In this embodiment, the feature vectors of the UAV image and the satellite image are extracted respectively, and the Euclidean distance is used to measure the distance between them. In the UAV positioning task, the distance between a UAV image and all satellite images is calculated, and the set of vectors with the closest distance is selected as the matching pair. In the UAV navigation task, the distance between a satellite image and all UAV images is calculated, and the set of vectors with the closest distance is selected as the matching pair. The calculation formula of the Euclidean distance is:

[0100]

[0101] The above description is not a limitation of the present application. It should be noted that those skilled in the art can make several changes, modifications, additions or substitutions without departing from the essential scope of the present application. These improvements and refinements should also be considered within the scope of the present application.

Claims

1. A cross-view scene matching method for UAV images and satellite images, characterized in that, Includes the following steps: Step 1: Obtain at least a thousand drone images and corresponding satellite images of the scene as a dataset; Step 2: Convert the drone images in the dataset into feature vectors, and also convert the satellite images into feature vectors. Train a scene matching model based on Siamese neural network based on these feature vectors to obtain the trained scene matching model. Step 3: Based on the preprocessing layer of the trained scene matching model, preprocess the UAV image to obtain a preprocessed UAV image, and also preprocess the satellite image to obtain a preprocessed satellite image; Step 4: Input the preprocessed UAV image into the backbone network layer of the scene matching model to extract scene features from the UAV image and obtain a primary UAV feature map. Input the preprocessed satellite image into the backbone network layer of the scene matching model to extract scene features from the satellite image and obtain a primary satellite feature map. Step 5: Enhance the UAV feature map using the HBP module and input it into the MLP layer to unify the feature dimensions to obtain the enhanced UAV feature map. Also enhance the satellite feature map using the HBP module and input it into the MLP layer to unify the feature dimensions to obtain the enhanced satellite feature map. Step 6: Calculate the feature distance between the enhanced UAV feature map and the enhanced satellite feature map, and perform feature matching to achieve UAV positioning and navigation tasks; In step 4, the preprocessed UAV image is input into the backbone network layer of the scene matching model to extract scene features and obtain a primary UAV feature map. Similarly, the preprocessed satellite image is input into the backbone network layer of the scene matching model to extract scene features and obtain a primary satellite feature map. The specific steps include: Step 401: Input the image into ResNetV2 and output a feature map with dimensions (1024,24,24). Grid the input feature map and perform linear mapping on the grid. Step 402: Input the vector obtained by linear mapping into 12 consecutive Transformer layers. During the training phase, the Transformer layer feeds the output vector into the fully connected layer and softmax layer of the classification layer for classification and training, and obtains the probability of each class. This improves the feature extraction capability of the scene matching model by training the classification network. During the testing phase, the Transformer network block directly derives the features and maps them to 512 dimensions as the feature map of the image. In step 401, ResNetV2 passes through three bottleneck modules, each containing 3, 4, and 9 residual blocks respectively. Pre-activated residual units are used in the residual blocks to improve the training efficiency of the network. The feature maps output by each bottleneck module increase their depth in the form of a feature pyramid to improve the effective information density during network structure calculation and to focus the network attention on the target scene, i.e., extracting local features of the image. In step 402, when in the training phase, the vector output by the Transformer layer is mapped to the number of target scenes to be classified through the neurons of the fully connected layer in the classification layer, and the output value is normalized through the softmax layer. The weight parameters obtained by the scene matching model during training are optimized based on the joint metric loss function of the Cross-entropy loss function and the Circle loss function. Then, a convolution operation with a kernel size of 1 and a stride of 1 is used to convolve the feature map, that is, to cut it into 1x1 squares to obtain feature vectors, so as to convert the feature map into feature vectors that can be processed by the Transformer layer. The formula for calculating the Cross-entropy loss function is as follows: in, y represents the predicted value from the scene matching model, and y represents the label value. For predicted values The predicted probability of being identified as class y. For the column vector of labels, Let i be the column vector of the sample, i be the index of the total number of scenes, j be the j-th vector output by the scene matching model, c be the c-th sample in this round of training, and c be the number of training samples. The formula for calculating the Circle loss function is as follows: Among them, s ij Indicates sample x i Does it belong to category j, d? ij Indicates sample x i The Euclidean distance from the center point of category j, where m represents a predefined margin = 0.

1.

2. The cross-view scene matching method for UAV images and satellite images according to claim 1, characterized in that, In step 2, the drone images in the dataset are converted into feature vectors, and the satellite images are also converted into feature vectors. Based on these feature vectors, a scene matching model based on Siamese neural network is trained to obtain a trained scene matching model. The scene matching model includes a preprocessing layer, a feature extractor, and a matching layer. The feature extractor adopts a hybrid architecture of ResNetV2 and Transformer layers.

3. The cross-view scene matching method for UAV images and satellite images according to claim 1, characterized in that, In step 3, the process of preprocessing the UAV image to obtain a preprocessed UAV image based on the preprocessing layer of the trained scene matching model, and preprocessing the satellite image to obtain a preprocessed satellite image, is as follows: The image size is standardized to 384x384 pixels, and image enhancement operations such as random slicing and random horizontal flipping are used. Random slicing is performed at random positions in the image according to the set size, and random horizontal flipping is performed by horizontally flipping the image with a certain probability.

4. The cross-view scene matching method for UAV images and satellite images according to claim 1, characterized in that, In step 402, the input to the Transformer layer is a gridded feature vector. A class token similar to a flag and a position embedding used to number each square are introduced, and then fed into the Transformer block. The structure of the Transformer block includes Layer Norm, multi-head attention mechanism, dropout layer and MLP layer. The Transformer layer uses multi-head attention mechanism to enable the network to transfer attention to the global in the gridded vector.

5. The cross-view scene matching method for UAV images and satellite images according to claim 1, 2, or 3, characterized in that, In step 5, the process of enhancing the UAV feature map using the HBP module and inputting it into the MLP layer to unify the feature dimensions, resulting in an enhanced UAV feature map, and enhancing the satellite feature map using the HBP module and inputting it into the MLP layer to unify the feature dimensions, resulting in an enhanced satellite feature map, specifically includes the following steps: First, the features output by the backbone network are enhanced using the HBP (Hierarchical Bilinear Pooling) module. Specifically, the (1024, 24, 24) feature map output by ResNetV2 and the (1024, 24, 24) feature map output by the backbone network are input into the HBP module for fusion enhancement. The HBP implementation consists of two steps: bilinear transformation and spatial global pooling. In the first step, f... j F represents the features extracted from the input image. conv This is represented as a CNN module, which includes convolutional layers, batch normalization layers, and ReLU layers. The specific formula for calculation is as follows: f j =F conv (f j ) In the second step, It is a global feature. It represents a local feature, where × denotes the outer product of two vectors: Then, the fusion features Normalization is performed to generate the final feature vector. At this point, ∈ = 1 × 10 -12 : The feature maps merged by HBP are then input into the MLP (Multilayer Perceptron) module. The structure of MLP includes fully connected layers, activation functions, normalization, and partial dropout operations. MLP performs a feature dimension unification operation, unifying the dimensions of the HBP output features to 512 dimensions.

6. The cross-view scene matching method for UAV images and satellite images according to claim 1, characterized in that, In step 6, the process of calculating the feature distance between the enhanced UAV feature map and the enhanced satellite feature map, and performing feature matching to achieve UAV positioning and navigation tasks, is as follows: The feature vectors of the obtained satellite images and the feature vectors of the UAV images are measured using Euclidean distance. The feature vector pair with the closest distance is selected as the match. When performing UAV positioning tasks, the distance between a UAV image and all satellite images is calculated, and the set of vectors with the closest distance is selected as the match pair. When performing UAV navigation tasks, the distance between a satellite image and all UAV images is calculated, and the set of vectors with the closest distance is selected as the match pair. The formula for calculating the Euclidean distance is as follows: Where p and q represent two n-dimensional vectors, p i and q i This represents the i-th element in the vector.

Images