UAV visual positioning method and system based on multi-view

Abstract

This invention discloses a multi-perspective drone visual positioning method and system, relating to the field of drone vision. The method comprises: obtaining a satellite map of a target area and cropping it into overlapping tiles to construct a satellite library; obtaining images captured by a drone within the target geographic area from multiple perspectives, and constructing a multi-perspective drone image based on the captured images; processing these images using a multi-perspective drone visual positioning network, which includes an embedding coding module, a perspective consistency feature learning module, and a prototype construction and alignment module for extracting single-perspective and multi-perspective drone image features and satellite library features; and determining the drone's current position by calculating the cosine similarity between the multi-perspective drone image features and the satellite library features. By integrating DINOv2 feature extraction, adapter design, and a perspective-invariant attention network, the present invention ensures efficient information processing and feature extraction, thereby improving positioning accuracy and reliability.

Description

Technical Field

[0001] The present invention relates to the field of unmanned aerial vehicle (UAV) vision, and in particular to a multi-viewing-based UAV vision positioning method and system. Background Art

[0002] In recent years, drone visual positioning technology has made significant progress, and its application value in environments where Global Navigation Satellite System (GNSS) signals are limited or completely inoperative has become increasingly prominent. As an important supplement to GNSS positioning, this technology shows broad application prospects in emergency rescue, smart cities, agricultural monitoring, and military reconnaissance. In the field of emergency rescue, in particular, drone visual positioning technology can effectively overcome the limitations of GPS signal attenuation or interruption in complex environments, providing precise positioning support for life-saving operations and significantly improving rescue efficiency. However, because drone visual positioning involves complex visual scenes and diverse feature changes, existing technologies still face significant challenges in addressing scale, perspective, and cross-domain issues.

[0003] The scale problem refers to the fact that during the flight of a drone, the images captured by its onboard camera have large scale differences due to changes in flight altitude. This will exacerbate the scale inconsistency between drone and satellite images, posing a severe challenge to visual positioning. In addition, existing methods usually rely on fine-tuning the parameters of the pre-trained backbone network, which not only increases the computational cost but also impairs the general feature extraction ability of the pre-trained model. Therefore, a parameter-efficient transfer learning strategy based on dual adapters is introduced to achieve efficient multi-scale feature extraction. This strategy freezes the parameters of the pre-trained backbone network and only fine-tunes the adapter, thereby significantly reducing the training overhead. This method not only retains the global feature learning ability of the pre-trained model, but also introduces multi-scale local information for specific scenes, enabling the model to effectively deal with the scale problem in drone visual positioning, ultimately achieving fast and robust visual positioning effects.

[0004] The viewpoint problem refers to the significant difference in perspective between oblique UAV imagery and orthophoto satellite imagery, which poses a significant challenge to visual localization. Existing deep metric learning methods use contrastive learning to narrow the feature distances between pairs of UAV and satellite images of the same location in the embedding space, while widening the feature distances between pairs of samples from different locations. This approach ultimately learns a shared embedding space for the UAV and satellite domain. However, the large viewpoint difference between oblique UAV and orthophoto satellite imagery results in inaccurate shared embeddings learned by current metric learning methods, which in turn reduces the model's ability to generalize to new locations during testing. Existing contrastive learning-based methods fail to fully exploit the complementarity between multiple UAV views of the same scene. Therefore, we propose viewpoint consistency feature learning. This constructs multi-view UAV image features that are robust to changes in the UAV's viewpoint and contrasts them with orthophoto satellite image features to learn a shared embedding space for the UAV and satellite domain, thereby improving matching accuracy.

[0005] The cross-domain issue refers to the differences in imaging conditions and spectral characteristics between drone and satellite imagery, which leads to significant feature distribution shifts and presents another major challenge in achieving precise positioning. Addressing this inconsistent cross-domain feature distribution and alleviating the inter-domain gap between drone and satellite imagery remains a key issue that needs to be addressed. Summary of the Invention

[0006] In order to solve the above problems, the present invention proposes a multi-perspective UAV visual positioning method and system, which effectively addresses the three major problems of scale, perspective and cross-domain in UAV visual positioning technology, and improves the adaptability and accuracy of the UAV visual positioning system.

[0007] The specific plan is as follows:

[0008] On the one hand, the multi-perspective UAV visual positioning method includes:

[0009] S1, obtain a satellite map of the target geographical area;

[0010] S2, cutting the satellite map into multiple satellite map blocks of the same size, and building a satellite library based on the satellite map blocks; wherein adjacent satellite map blocks have overlapping areas;

[0011] S3, obtaining images taken by a drone from multiple perspectives within the target geographical area, and constructing a multi-perspective drone image based on the taken images;

[0012] S4, input the multi-view UAV image into the multi-view UAV visual positioning network, extract the single-view UAV image features from the multi-view UAV image through the embedding coding module of the multi-view UAV visual positioning network, and input the single-view UAV image features into the view consistency feature learning module of the multi-view UAV visual positioning network to extract the multi-view UAV image features;

[0013] S5, the satellite map blocks in the satellite library are input into the multi-view UAV visual positioning network, and the satellite library features are extracted through the embedding coding module of the multi-view UAV visual positioning network;

[0014] S6, calculating the cosine similarity between the multi-view UAV image features and the satellite library features, and determining the geographical coordinates of the center point corresponding to the satellite map block with the highest similarity as the current position of the UAV.

[0015] In S4, the multi-view UAV visual positioning network further includes: a prototype construction and alignment module; the prototype construction and alignment module narrows the UAV-satellite domain gap by constructing and aligning a UAV prototype library and a satellite prototype library.

[0016] Furthermore, the UAV prototype library and the satellite prototype library respectively include the feature vectors of all locations in the UAV domain and the satellite domain in the training set. The feature vectors of all locations are set to all zero vectors during initialization. During the model training process, the UAV prototype library and the satellite prototype library are updated batch by batch in a moving average manner. The update formula is as follows:

[0017] ;

[0018] ;

[0019] in represents the i-th multi-view UAV image feature in a batch, represents the i-th satellite image feature in a batch, express The prototype vector corresponding to the real location, express The prototype vector corresponding to the real location; Indicates update parameters.

[0020] Furthermore, in S4, the embedded coding module includes: an adapted DINOv2 feature extraction unit and a first multi-layer perceptron; the output of the adapted DINOv2 feature extraction unit is directly connected to the input of the first multi-layer perceptron.

[0021] Furthermore, the adapted DINOv2 feature extraction unit includes 12 DINOv2 blocks connected in sequence; the DINOv2 block includes: a first layer normalization, a multi-head self-attention mechanism, a common adapter, a first residual connection unit, a second layer normalization, a second multi-layer perceptron, a multi-scale convolution adapter, and a second residual connection unit; the input original image features are sequentially input into the first layer normalization, the multi-head self-attention mechanism, and the common adapter to obtain the image features processed by the common adapter, and the image features processed by the common adapter are sequentially combined with the input original image features through the first residual connection unit. The image features after addition are obtained, and the added image features are input into the second layer normalization to obtain the second layer normalized image features. The second layer normalized image features are respectively input into the multi-scale convolution adapter and the second multi-layer perceptron. The image features processed by the multi-scale convolution adapter are multiplied by the scaling factor s and then added element-by-element with the image features processed by the second multi-layer perceptron to obtain the quadratic added image features. The image features processed by the first residual connection unit and the output image features after the quadratic addition are added element-by-element through the second residual connection unit to obtain the final output image features.

[0022] Furthermore, the ordinary adapter includes a first dimensionality reduction linear layer, a RELU activation function, a first dimensionality increase linear layer and a third residual connection unit; the image features after processing by the multi-head self-attention mechanism are first processed by the first dimensionality reduction linear layer to obtain the image features after dimensionality reduction; the image features after dimensionality reduction are nonlinearly activated by the RELU activation function to obtain the image features after nonlinear activation; the image features after nonlinear activation are dimensionally increased by the first dimensionality increase linear layer to obtain the image features after dimensionality increase; the image features after dimensionality increase and the image features after processing by the multi-head self-attention mechanism are element-by-element added through the third residual connection unit to obtain the image features processed by the ordinary adapter.

[0023] Furthermore, the multi-scale convolution adapter includes a second dimensionality reduction linear layer, a RELU activation function, a multi-scale convolution unit, a second dimensionality increase linear layer and a fourth residual connection unit; wherein, the image features after the second layer normalization processing are first input into the second dimensionality reduction linear layer for dimensionality reduction processing to obtain the image features after dimensionality reduction; the image features after dimensionality reduction are input into the RELU activation function for nonlinear activation processing to obtain the image features after nonlinear activation; after removing the class code from the image features after nonlinear activation, they are reshaped into feature maps and input into the multi-scale convolution unit for processing to obtain the feature maps processed by the multi-scale convolution unit; after reshaping the feature maps processed by the multi-scale convolution unit into image features, the class codes are re-spliced ​​to obtain the image features processed by the multi-scale convolution unit; the image features processed by the multi-scale convolution unit are element-wise added to the image features after nonlinear activation through the fourth residual connection unit to obtain the image features processed by the fourth residual connection unit; the image features processed by the fourth residual connection unit are dimensionality increased through the second dimensionality increase linear layer, and finally the image features processed by the multi-scale convolution adapter are output.

[0024] Furthermore, in S4, the perspective consistency feature learning module is implemented by a perspective-invariant attention network, which includes a single-head self-attention layer, a fifth residual connection unit and an average pooling layer; the single-view UAV image feature is first subjected to cross-view feature weight distribution through the single-head self-attention layer to obtain the image feature after weight distribution, and the image feature after weight distribution is added element by element with the input single-view UAV image feature through the fifth residual connection unit to obtain the image feature that retains the original information, and then the image feature that retains the original information is feature compressed and dimensionally regularized through the average pooling layer to obtain the multi-view UAV image feature.

[0025] On the other hand, the multi-perspective UAV visual positioning system includes:

[0026] A satellite map acquisition module is used to obtain a satellite map of a target geographical area;

[0027] A satellite map clipping module is used to clip the satellite map into multiple satellite map blocks of the same size and build a satellite library based on the satellite map blocks; wherein adjacent satellite map blocks have overlapping areas;

[0028] A multi-view UAV image construction module is used to obtain images taken by UAVs from multiple viewpoints within the target geographical area and construct a multi-view UAV image based on the captured images;

[0029] A multi-view UAV embedding extraction module is used to input multi-view UAV images into a multi-view UAV visual positioning network, extract single-view UAV image features through the embedding coding module of the multi-view UAV visual positioning network, and input the single-view UAV image features into the view consistency feature learning module of the multi-view UAV visual positioning network to extract multi-view UAV image features;

[0030] The satellite library embedding and extraction module is used to cut satellite maps in the satellite library into pieces and input them into the multi-view UAV visual positioning network. The satellite library features are extracted through the embedding coding module of the multi-view UAV visual positioning network.

[0031] The UAV positioning module is used to calculate the cosine similarity between the multi-view UAV image features and the satellite library features, and determine the geographical coordinates of the center point corresponding to the satellite map block with the highest similarity as the current position of the UAV.

[0032] The present invention adopts the above technical solution and has the following beneficial effects:

[0033] (1) The present invention adopts a parameter-efficient transfer learning strategy, which only fine-tunes the dual adapters while keeping the parameters of the pre-trained backbone network unchanged. This design not only significantly reduces training time and computational cost, but also introduces multi-scale local information for specific scenarios while retaining the global learning ability of the pre-trained model. Through this design, the model can better cope with the mid-scale problem of UAV visual positioning and achieve fast and robust visual positioning.

[0034] (2) This paper proposes perspective consistency feature learning to construct a feature representation that is robust to changes in drone viewpoints. It uses a single-head self-attention mechanism to perform weighted interaction on image features from different perspectives of the same location, effectively capturing the correlation between multiple perspectives and aggregating them into multi-perspective drone image features through average pooling operations. Multi-perspective drone image features can represent the discriminative features of a location that are invariant to the perspective, and improve the accuracy of cross-perspective retrieval by comparative learning with satellite image features.

[0035] (3) This paper effectively alleviates the problem of inconsistent cross-domain feature distribution by introducing a dynamically maintained prototype library. The prototype library adopts a moving average update mechanism and is dynamically adjusted according to the training data to ensure that it can gradually learn and reflect the feature distribution of location discrimination within the domain. In addition, combined with a prototype-based domain alignment mechanism, the domain gap between UAV and satellite image features is further reduced. This design significantly enhances the expressive power of cross-domain features and improves the positioning accuracy of the model in complex cross-domain scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a flow chart of a multi-perspective UAV visual positioning method according to an embodiment of the present invention;

[0037] Figure 2 This is a training flow chart of a model according to an embodiment of the present invention;

[0038] Figure 3 Schematic diagram of a multi-perspective UAV image group according to a multi-perspective UAV visual positioning method according to an embodiment of the present invention;

[0039] Figure 4 This is a schematic diagram of the search results of the multi-view UAV visual positioning method according to an embodiment of the present invention;

[0040] Figure 5 This is a system diagram of a multi-perspective UAV visual positioning method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0041] The present invention will be described in further detail below with reference to the examples and accompanying drawings, but the embodiments of the present invention are not limited thereto. Figure 1 As shown, the multi-perspective UAV visual positioning method of the present invention includes:

[0042] S1, obtain a satellite map of the target geographical area;

[0043] S2, cutting the satellite map into multiple satellite map blocks of the same size, and building a satellite library based on the satellite map blocks; wherein adjacent satellite map blocks have overlapping areas;

[0044] S3, obtaining images taken by a drone from multiple perspectives within the target geographical area, and constructing a multi-perspective drone image based on the taken images;

[0045] S4, input the multi-view UAV image into the multi-view UAV visual positioning network, extract the single-view UAV image features from the multi-view UAV image through the embedding coding module of the multi-view UAV visual positioning network, and input the single-view UAV image features into the view consistency feature learning module of the multi-view UAV visual positioning network to extract the multi-view UAV image features;

[0046] Specifically, the multi-view UAV visual positioning network also includes a prototype construction and alignment module, which narrows the UAV-satellite domain gap by constructing and aligning a UAV prototype library and a satellite prototype library.

[0047] In this embodiment, the prototype construction and alignment process is to respectively construct a drone prototype library and a satellite prototype library using the multi-view drone image features output by the view consistency feature learning module and the satellite image features output by the embedded coding module in a moving average manner. They represent the discriminative features within the drone-satellite domain, and then achieve cross-domain alignment by aligning the drone prototype library and the satellite prototype library.

[0048] Specifically, the embedded coding module includes: an adapted DINOv2 feature extraction unit and a first multi-layer perceptron; the output of the adapted DINOv2 feature extraction unit is directly connected to the input end of the first multi-layer perceptron.

[0049] Specifically, the adapted DINOv2 feature extraction unit includes 12 DINOv2 blocks connected in sequence; the DINOv2 block includes: a first layer normalization, a multi-head self-attention mechanism, a common adapter, a first residual connection unit, a second layer normalization, a second multi-layer perceptron, a multi-scale convolution adapter, and a second residual connection unit; the input original image features are sequentially input into the first layer normalization, the multi-head self-attention mechanism, and the common adapter to obtain the image features processed by the common adapter, and the image features processed by the common adapter are sequentially combined with the input original image features through the first residual connection unit. The image features after addition are obtained, and the added image features are input into the second layer normalization to obtain the second layer normalized image features. The second layer normalized image features are respectively input into the multi-scale convolution adapter and the second multi-layer perceptron. The image features processed by the multi-scale convolution adapter are multiplied by the scaling factor s and then added element-by-element with the image features processed by the second multi-layer perceptron to obtain the quadratic added image features. The image features processed by the first residual connection unit and the output image features after the quadratic addition are added element-by-element through the second residual connection unit to obtain the final output image features.

[0050] Specifically, when processing an input image, it is first sliced ​​into blocks of a specified size. These blocks are then flattened into one-dimensional vectors. These one-dimensional vectors are then processed through a linear layer to generate a series of linearly transformed one-dimensional vectors, known as tokens. A learnable classification vector, known as a class token, is inserted at the beginning of the token sequence. To preserve positional information, each token is also added with a learnable positional code of the same dimension to obtain the original features of the input image. Specifically, the standard adapter comprises a first dimensionality reduction linear layer, a Relative Unity (RELU) activation function, a first dimensionality increase linear layer, and a third residual connection unit. Image features processed by a multi-head self-attention mechanism are first processed through the first dimensionality reduction linear layer to obtain reduced-dimensionality image features. These reduced-dimensionality image features are then nonlinearly activated through a RELU activation function to obtain nonlinearly activated image features. The nonlinearly activated image features are then dimensionalized through a first dimensionality increase linear layer to obtain increased-dimensionality image features. The increased-dimensionality image features and the image features processed by the multi-head self-attention mechanism are then element-wise added through the third residual connection unit to obtain image features processed by the standard adapter.

[0051] Specifically, the multi-scale convolution adapter includes a second dimensionality reduction linear layer, a RELU activation function, a multi-scale convolution unit, a second dimensionality increase linear layer and a fourth residual connection unit; wherein, the image features after the second layer normalization processing are first input into the second dimensionality reduction linear layer for dimensionality reduction processing to obtain the image features after dimensionality reduction; the image features after dimensionality reduction are processed by the RELU activation function for nonlinear activation to obtain the image features after nonlinear activation; after removing the class code from the image features after nonlinear activation, they are reshaped into feature maps and input into the multi-scale convolution unit for processing to obtain the feature maps processed by the multi-scale convolution unit; after reshaping the feature maps processed by the multi-scale convolution unit into image features, the class codes are re-spliced ​​to obtain the image features processed by the multi-scale convolution unit; the image features processed by the multi-scale convolution unit are element-by-element added to the image features after nonlinear activation through the fourth residual connection unit to obtain the image features processed by the fourth residual connection; the image features processed by the fourth residual connection are dimensionality increased through the second dimensionality increase linear layer, and finally the image features processed by the multi-scale convolution adapter are output.

[0052] Specifically, the multi-scale convolution unit consists of three convolution layers and channel splicing. The first convolution layer includes a convolution operation unit with a convolution kernel size of 1*1, a number of convolution kernels of 192, a stride of 1, and a padding size of 0. The second convolution layer includes two convolution operation units connected in sequence. The convolution kernel size of the first convolution operation unit is 1*1, the number of convolution kernels is 24, the stride of 1, and the padding size is 0. The convolution kernel size of the second convolution operation unit is 3*3, the number of convolution kernels is 96, the stride of 1, and the padding size is 1. The third convolution layer includes two convolution operation units connected in sequence. The convolution kernel size of the first convolution operation unit is 1*1, the number of convolution kernels is 24, the stride of 1, and the padding size is 0. The second convolution operation unit has a convolution kernel size of 5*5, 96 convolution kernels, a stride of 1, and a padding size of 2. The outputs of the three convolution layers are channel-wise concatenated to obtain the output of the multi-scale convolution unit. The convolution operation unit includes convolution operations, batch normalization, and a RELU activation function.

[0053] Specifically, after removing the ordinary adapter and the multi-scale convolution adapter, the remaining structure of the DINOv2 block constitutes a standard Transformer encoding layer; the first multi-layer perceptron includes a first linear layer, a GELU activation function, a batch normalization layer, a second linear layer and an L2 normalization layer connected in sequence, wherein the first linear layer is used to increase the input feature dimension from 768 to 1536, and the second linear layer is used to reduce the input feature dimension from 1536 to 768; the class code in the image feature output by the adapted DINOv2 feature extraction unit is extracted, and the class code is used as the input of the first multi-layer perceptron; the class code is sequentially subjected to a 2-fold dimensionality increase processing by the first linear layer, and then subjected to nonlinear activation by the GELU activation function, and then normalized by the batch normalization layer, and then subjected to a 2-fold dimensionality reduction processing by the second linear layer, and finally normalized by the L2 normalization layer to obtain the output of the first multi-layer perceptron.

[0054] Specifically, the perspective consistency feature learning module is implemented through a perspective-invariant attention network, which includes a single-head self-attention layer, a fifth residual connection unit and an average pooling layer; the single-view UAV image feature is first subjected to cross-view feature weight distribution through the single-head self-attention layer to obtain the image feature after weight distribution, and the image feature after weight distribution is added element by element with the input single-view UAV image feature through the fifth residual connection unit to obtain the image feature that retains the original information, and then the image feature that retains the original information is feature compressed and dimensionally regularized through the average pooling layer to obtain the multi-view UAV image feature.

[0055] Specifically, the UAV prototype library and the satellite prototype library respectively include the feature vectors of all locations in the training set. The feature vectors of all locations are set to all zero vectors during initialization. During the model training process, the UAV prototype library and the satellite prototype library are updated batch by batch in a moving average manner. The update formula is as follows:

[0056] ;

[0057] ;

[0058] in represents the i-th multi-view UAV image feature in a batch, represents the i-th satellite image feature in a batch, express The prototype vector corresponding to the real location, express The prototype vector corresponding to the real location; represents the update parameters. Specifically, The update process is: ;

[0059] in, is the current network training cycle, and epoch is the total network training cycle. The initial value is set to 0.99, the start value is set to 0.9, and the end value is set to 0.5.

[0060] Specifically, the multi-view UAV visual localization network also includes a classifier, which includes a third linear layer, a fourth linear layer, and a batch normalization layer. Image features first pass through the third linear layer, then undergo normalization processing through the batch normalization layer, and finally pass through the fourth linear layer to output the classification result. The third linear layer has an input feature dimension of 768 and an output feature dimension of 512. The fourth linear layer has an input feature dimension of 512 and an output feature dimension equal to the number of locations in the training set.

[0061] S5, the satellite map blocks in the satellite library are input into the multi-view UAV visual positioning network, and the satellite library features are extracted through the embedding coding module of the multi-view UAV visual positioning network;

[0062] S6, calculating the cosine similarity between the multi-view UAV image features and the satellite library features, and determining the geographical coordinates of the center point corresponding to the satellite map block with the highest similarity as the current position of the UAV.

[0063] In this example, during the training of the multi-view UAV visual localization network, a shared weight classifier is used to map single-view UAV image features, multi-view UAV image features, satellite image features, UAV prototype library, and satellite prototype library to a location-specific embedding space. However, during the testing phase, the classifier no longer works and is therefore removed. The UAV prototype library and the satellite prototype library represent the feature representations of location discrimination in the UAV domain and the satellite domain in the training set, respectively. Through the classification loss and domain alignment loss based on the prototype library, the model can extract feature representations of location discrimination and domain consistency, thereby effectively alleviating cross-domain differences. These prototype libraries are used to calculate the loss function to alleviate cross-domain differences and are removed during the model testing phase. Therefore, during model testing, when the input to the network is a multi-view UAV image of a new location, the UAV prototype library and the satellite prototype library will not participate in the calculation. Before calling the pre-trained multi-view UAV visual localization network to embed the images in the multi-view UAV image and the satellite library, the network needs to be trained. The training process is as follows Figure 2As shown in FIG, the specific steps are as follows: obtaining multi-view UAV and satellite training image pairs in the target area, including UAV images of multiple selected locations in the target area taken by UAV cameras from different perspectives, and constructing a multi-view UAV image library; obtaining satellite maps of the same geographical area from Google Maps; cropping satellite maps to ensure that adjacent satellite map slices overlap; pairing multi-view UAV images and satellite images of the same geographical location to construct a training dataset; calling the initial multi-view UAV visual positioning network to preprocess the above multi-view UAV and satellite training image pairs, the specific steps are as follows: first, randomly shuffling the locations in the training dataset, and inputting the multi-view UAV and corresponding satellite images of B locations in each batch into the embedding coding module of the shared weight multi-view UAV visual positioning network, which includes an adapted DINOv2 feature extraction unit and a first multi-layer perceptron connected in sequence to generate single-view UAV image features and satellite image features; inputting the single-view UAV image features of B locations into the perspective consistency feature learning module of the multi-view UAV visual positioning network to obtain multi-view UAV image features of B locations; and updating the UAV prototype library and satellite prototype library respectively by using the multi-view UAV image features and corresponding satellite image features of B locations in a moving average manner. During initialization, both prototype libraries contain C all-zero vectors, where C is the number of locations in the training set. The single-view UAV image features, multi-view UAV image features, satellite image features, and their respective original scores (Logits) obtained by the classifier are processed with the UAV prototype library, satellite prototype library, and their respective original scores (Logits) obtained by the classifier through a loss function. The parameters of the initial multi-view UAV visual positioning network are optimized to obtain a trained multi-view UAV visual positioning network. The specific details of the loss function are as follows:

[0064] Assume that a batch consists of multi-view drone and satellite training image pairs from B locations, with V multi-view drone images and 1 satellite image sampled for each location. The loss function of the multi-view drone visual localization network includes a viewpoint invariance loss, an embedding classification loss, a loss comparing multi-view drone image features with satellite image features, a prototype classification loss, and a prototype domain alignment loss. The prototype classification loss and prototype domain alignment loss are designed to further reduce the domain gap.

[0065] The view invariant loss constrains the average feature distance between the V single-view drone image features of each location in a batch and the multi-view drone image features of the location within a given threshold, thereby achieving that the V single-view drone image features of each location are closely clustered around the multi-view drone image features, thereby effectively improving the accuracy of single-view retrieval (coarse positioning). for:

[0066] ;

[0067] ;

[0068] ;

[0069] Where C is the number of locations in the training set; It is Location Single-view drone image features, It is Multi-view UAV image features of locations, represents the inner product operation, because before the calculation and It has been L2 normalized, so the inner product is equivalent to cosine similarity; is a set threshold.

[0070] Embedding classification loss includes single-view drone embedding classification loss, satellite embedding classification loss, and multi-view drone embedding classification loss. Each classification loss uses the cross entropy loss function. The calculation formula is:

[0071] ;

[0072] ;

[0073] ;

[0074] ;

[0075] Among them, C is the number of locations in the training set, N is the number of drone images in the training set; for single-view drone image features, is a sign function, taking values ​​of 0 or 1. When c is the real location of the i-th single-view UAV image feature, Take 1, otherwise take 0; is the probability that the i-th single-view UAV image feature is predicted by the classifier to be the c-th location. For multi-view UAV image features and satellite image features, the true location indicator and the predicted probability are respectively 、 、 and express; Embedding classification loss for single-view drones, Embedding loss for multi-view UAV classification, is the satellite embedding classification loss, is the sum of the three, i.e., the embedding classification loss.

[0076] The InfoNCE loss is used for the contrastive loss between multi-view drone image features and satellite image features. This loss aims to learn a shared embedding space for the drone and satellite domain by narrowing the distance between multi-view drone image features and satellite image features at the same location in the embedding space while simultaneously widening the distance between features at different locations. Unlike traditional triplet losses, this loss considers all negative examples of anchor points within a batch, enabling more comprehensive contrastive learning. The loss is calculated as:

[0077] ;

[0078] ;

[0079] ;

[0080] Where C is the number of locations in the training set, and B is the number of locations included in a batch. is the multi-view UAV image feature of the i-th location, is the corresponding positive sample satellite image feature. is the satellite image feature of the i-th location, is the corresponding positive sample multi-view drone image feature. is a logarithmic scaling factor. and They are InfoNCE contrast loss in two directions, multi-view UAV image feature and satellite image feature contrast loss It is the sum of the two.

[0081] The prototype classification loss includes the drone prototype classification loss and the satellite prototype classification loss. Each classification loss uses the cross entropy loss function. The calculation formula is:

[0082] ;

[0083] ;

[0084] ;

[0085] Where C is the number of locations in the training set. is a sign function (0 or 1), when k is the When the real location of the drone prototype vector is k (i.e. k is equal to i), =1, otherwise = 0, and For the The prototype vector of the drone at location k is predicted by the classifier to be the probability of location k. Similarly, if k is the When the true category of the satellite prototype vector of the location is (that is, k is equal to i), =1, otherwise = 0, and For the The satellite prototype vector of a location is predicted by the classifier to be the probability of the kth location. Classification loss for the drone prototype, is the satellite prototype classification loss, prototype classification loss It is the sum of the two.

[0086] The prototype domain alignment loss aims to align the UAV prototype library and the satellite prototype library to further alleviate the domain differences. The calculation formula is:

[0087] ; ;

[0088] ;

[0089] Where N is the number of drone images in the training set. represents the i-th single-view drone image feature, represents the prototype vector of the drone at the c-th location, represents the satellite prototype vector of the c-th location, represents the satellite image features of the i-th location, represents the UAV domain alignment loss, represents the satellite domain alignment loss, is the overall domain alignment loss, which is the sum of the drone domain alignment loss and the satellite domain alignment loss.

[0090] The total loss function is:

[0091] ;

[0092] in, is the view invariant loss, is the embedding classification loss, is the contrast loss between multi-view UAV image features and satellite image features, is the prototype classification loss, Prototype domain alignment loss, 、 、 、 is a hyperparameter, and is a dynamic trade-off parameter that increases with the number of training rounds t. Based on the total loss, the parameters of the initial multi-view UAV visual localization network are optimized.

[0093] Specifically, and The update formula is as follows:

[0094]

[0095]

[0096] in, is the current network training cycle. Preferably, max1 is set to one tenth of the epoch, and max2 is set to one half of the epoch.

[0097] like Figure 4 As shown, the trained multi-view UAV visual positioning network is shown. Figure 3 The figure shows the retrieval results for three sets of multi-view drone images. The figure lists the top five satellite images, ranked from highest to lowest by cosine similarity, and annotates the images with numbers. Green numbers indicate correct retrievals, while red numbers indicate incorrect retrievals. Since each location corresponds to only one satellite image, the ideal retrieval result for the model is to mark the first image green (indicating a correct retrieval) and the remaining images red (indicating incorrect retrievals).

[0098] Specifically, during the testing process, multi-view drone images are first fed into the embedding coding module of the multi-view drone visual localization network to obtain single-view drone image features. These features are then fed into the view consistency feature learning module of the multi-view drone visual localization network to obtain multi-view drone image features. Simultaneously, images from the satellite library are also processed by the embedding coding module of the multi-view drone visual localization network to obtain corresponding satellite library features. Localization can be performed in two ways: single-view search (coarse localization): single-view drone image features are used to search within the satellite library features to find the satellite feature with the highest cosine similarity. The center of the corresponding satellite map slice is used as the drone's location. Multi-view search (robust localization): multi-view drone image features are used to search within the satellite library features to find the satellite feature with the highest cosine similarity. The center of the corresponding satellite map slice is used as the drone's location.

[0099] In this embodiment, the experimental environment is an Intel(R) Core(TM) i9-12900KF CPU@3.19GHz, with 128G memory, and the GPU processor is an NVIDA GeForce RTX 3090 24G; the training data comes from the University-1652 public dataset, which contains multi-view drone images and satellite image pairs of 701 locations. Each location has a multi-view drone image consisting of 54 drone images and 1 satellite image, and the size is 512×512 pixels; during training, the multi-view drone images of each location are divided into three groups according to three different flight altitudes, specifically, the 1st to 18th, 19th to 36th, and 37th to 54th images correspond to the first, second, and third groups respectively, and 4 images are randomly selected from each group for training. The network structure of the DINOv2 pre-trained model we used is ViT-B, where the patch size is 14×14 and the token dimension is 768. The training uses the PyTorch deep learning framework, the optimizer chooses stochastic gradient descent (SGD), and the batch size is set to 16, which means that each batch contains 192 multi-view drone images and 16 satellite images from 16 locations, a total of 208 images. The epoch is set to 30, the initial learning rate is 0.01, and the learning rate is adjusted to 0.001 for the last 10 epochs. The momentum and weight decay are set to 0.9 and 0.0005, respectively. In addition, the parameters α and β in the loss function are set to 5 and 0.01, respectively. Set to 0.5, Set to 0.5. The scaling factor s is 0.2, is 0.7.

[0100] like Figure 5 As shown, this embodiment also discloses a multi-perspective UAV visual positioning system, including:

[0101] A satellite map acquisition module 51 is used to acquire a satellite map of a target geographical area;

[0102] The satellite map clipping module 52 is used to clip the satellite map into a plurality of satellite map blocks of the same size, and to construct a satellite library based on the satellite map blocks; wherein adjacent satellite map blocks have overlapping areas;

[0103] A multi-view UAV image construction module 53 is used to obtain images taken by a UAV from multiple viewpoints within the target geographical area and construct a multi-view UAV image;

[0104] a multi-view UAV embedding extraction module 54 for inputting the multi-view UAV image into the multi-view UAV visual positioning network, extracting single-view UAV image features from the multi-view UAV image through the embedding coding module of the multi-view UAV visual positioning network, and inputting the single-view UAV image features into the view consistency feature learning module of the multi-view UAV visual positioning network to extract multi-view UAV image features;

[0105] The satellite library embedding and extraction module 55 is used to cut the satellite map blocks in the satellite library into the multi-view UAV visual positioning network, and extract the satellite library features through the embedding coding module of the multi-view UAV visual positioning network;

[0106] The UAV positioning module 56 is used to calculate the cosine similarity between the multi-view UAV image features and the satellite library features, and determine the geographical coordinates of the center point corresponding to the satellite map block with the highest similarity as the current position of the UAV.

[0107] The specific implementation of the multi-perspective UAV visual positioning system is the same as the multi-perspective UAV visual positioning method, and will not be repeated in this embodiment.

[0108] Although the present invention has been particularly shown and described in conjunction with preferred embodiments, it will be understood by those skilled in the art that various changes in form and details may be made to the present invention without departing from the spirit and scope of the invention as defined in the appended claims, and all such changes are within the scope of protection of the present invention.

Claims

1. A multi-view UAV visual positioning method, characterized in that: include: S1, obtain a satellite map of the target geographical area; S2, cutting the satellite map into multiple satellite map blocks of the same size, and building a satellite library based on the satellite map blocks; wherein adjacent satellite map blocks have overlapping areas; S3, obtaining images taken by a drone from multiple perspectives within the target geographical area, and constructing a multi-perspective drone image based on the taken images; S4, inputting the multi-view UAV image into the multi-view UAV visual positioning network, extracting single-view UAV image features from the multi-view UAV image through the embedding coding module of the multi-view UAV visual positioning network, and inputting the single-view UAV image features into the perspective consistency feature learning module of the multi-view UAV visual positioning network to extract multi-view UAV image features; the perspective consistency feature learning module is implemented by a perspective-invariant attention network, which includes a single-head self-attention layer, a fifth residual connection unit and an average pooling layer; the single-view UAV image features are first subjected to cross-view feature weight distribution through the single-head self-attention layer to obtain image features after weight distribution, and the image features after weight distribution are added element-by-element to the input single-view UAV image features through the fifth residual connection unit to obtain image features that retain the original information, and then the image features that retain the original information are subjected to feature compression and dimension regularization through the average pooling layer to obtain multi-view UAV image features; S5, the satellite map blocks in the satellite library are input into the multi-view UAV visual positioning network, and the satellite library features are extracted through the embedding coding module of the multi-view UAV visual positioning network; S6, calculating the cosine similarity between the multi-view UAV image features and the satellite library features, and determining the geographical coordinates of the center point corresponding to the satellite map block with the highest similarity as the current position of the UAV.

2. The multi-view UAV visual positioning method according to claim 1, characterized in that: In S4, the multi-view UAV visual positioning network further includes: a prototype construction and alignment module; the prototype construction and alignment module narrows the gap between the UAV domain and the satellite domain by constructing and aligning the UAV prototype library and the satellite prototype library.

3. The multi-view UAV visual positioning method according to claim 2, characterized in that: The UAV prototype library and the satellite prototype library respectively include the feature vectors of all locations in the UAV domain and the satellite domain in the training set. The feature vectors of all locations are set to all zero vectors during initialization. During the model training process, the UAV prototype library and the satellite prototype library are updated batch by batch in a moving average manner. The update formula is as follows: where d i represents the i-th multi-view UAV image feature in a batch, s i represents the i-th satellite image feature in a batch, Indicates d i The prototype vector corresponding to the real location, Indicates s i The prototype vector corresponding to the real location; α(t) represents the update parameter.

4. The multi-view UAV visual positioning method according to claim 1, characterized in that: In S4, the embedded coding module includes: an adapted DINOv2 feature extraction unit and a first multi-layer perceptron; the output of the adapted DINOv2 feature extraction unit is directly connected to the input end of the first multi-layer perceptron.

5. The multi-view UAV visual positioning method according to claim 4, characterized in that: The adapted DINOv2 feature extraction unit includes 12 DINOv2 blocks connected in sequence; the DINOv2 block includes: a first layer normalization, a multi-head self-attention mechanism, a common adapter, a first residual connection unit, a second layer normalization, a second multi-layer perceptron, a multi-scale convolution adapter, and a second residual connection unit; the input original image features are sequentially input into the first layer normalization, the multi-head self-attention mechanism, and the common adapter to obtain the image features processed by the common adapter, and the image features processed by the common adapter are element-wise compared with the input original image features through the first residual connection unit. The image features after addition are obtained, and the added image features are input into the second layer normalization to obtain the second layer normalized image features. The second layer normalized image features are respectively input into the multi-scale convolution adapter and the second multi-layer perceptron. The image features processed by the multi-scale convolution adapter are multiplied by the scaling factor s and then added element-by-element with the image features processed by the second multi-layer perceptron to obtain the quadratic added image features. The image features processed by the first residual connection unit and the output image features after the quadratic addition are added element-by-element through the second residual connection unit to obtain the final output image features.

6. The multi-view UAV visual positioning method according to claim 5, characterized in that: The common adapter includes a first dimensionality reduction linear layer, a RELU activation function, a first dimensionality increase linear layer and a third residual connection unit; the image features after being processed by the multi-head self-attention mechanism are first processed by the first dimensionality reduction linear layer to obtain the image features after dimensionality reduction; the image features after dimensionality reduction are nonlinearly activated by the RELU activation function to obtain the image features after nonlinear activation; the image features after nonlinear activation are dimensionally increased by the first dimensionality increase linear layer to obtain the image features after dimensionality increase; the image features after dimensionality increase and the image features after being processed by the multi-head self-attention mechanism are element-by-element added through the third residual connection unit to obtain the image features processed by the common adapter.

7. The multi-view UAV visual positioning method according to claim 5, characterized in that: The multi-scale convolution adapter includes a second dimensionality reduction linear layer, a RELU activation function, a multi-scale convolution unit, a second dimensionality increase linear layer and a fourth residual connection unit; wherein, the image features after the second layer normalization processing are first input into the second dimensionality reduction linear layer for dimensionality reduction processing to obtain the image features after dimensionality reduction; the image features after dimensionality reduction are input into the RELU activation function for nonlinear activation processing to obtain the image features after nonlinear activation; after removing the class code from the image features after nonlinear activation, they are reshaped into feature maps and input into the multi-scale convolution unit for processing to obtain the feature maps processed by the multi-scale convolution unit; after reshaping the feature maps processed by the multi-scale convolution unit into image features, the class codes are re-spliced ​​to obtain the image features processed by the multi-scale convolution unit; the image features processed by the multi-scale convolution unit are element-wise added to the image features after nonlinear activation through the fourth residual connection unit to obtain the image features processed by the fourth residual connection unit; the image features processed by the fourth residual connection unit are dimensionality increased through the second dimensionality increase linear layer, and finally the image features processed by the multi-scale convolution adapter are output.

8. A multi-view UAV visual positioning system, characterized by: include: A satellite map acquisition module is used to obtain a satellite map of a target geographical area; A satellite map clipping module is used to clip the satellite map into multiple satellite map blocks of the same size and build a satellite library based on the satellite map blocks; wherein adjacent satellite map blocks have overlapping areas; A multi-view UAV image construction module is used to obtain images taken by UAVs from multiple viewpoints within the target geographical area and construct a multi-view UAV image based on the captured images; A multi-view drone embedding and extraction module is used to input the multi-view drone image into the multi-view drone visual positioning network, extract the single-view drone image features through the embedding coding module of the multi-view drone visual positioning network, and input the single-view drone image features into the perspective consistency feature learning module of the multi-view drone visual positioning network to extract the multi-view drone image features; the perspective consistency feature learning module is implemented by a perspective-invariant attention network, which includes a single-head self-attention layer, a fifth residual connection unit and an average pooling layer; the single-view drone image features are first subjected to cross-view feature weight distribution through the single-head self-attention layer to obtain the image features after weight distribution, and the image features after weight distribution are added element-by-element to the input single-view drone image features through the fifth residual connection unit to obtain the image features that retain the original information, and then the image features that retain the original information are subjected to feature compression and dimension regularization through the average pooling layer to obtain the multi-view drone image features; The satellite library embedding and extraction module is used to cut satellite maps in the satellite library into pieces and input them into the multi-view UAV visual positioning network. The satellite library features are extracted through the embedding coding module of the multi-view UAV visual positioning network. The UAV positioning module is used to calculate the cosine similarity between the multi-view UAV image features and the satellite library features, and determine the geographical coordinates of the center point corresponding to the satellite map block with the highest similarity as the current position of the UAV.

Images