Multi-view-angle-based unmanned aerial vehicle visual positioning method and system

Abstract

The invention discloses a multi-view-angle-based unmanned aerial vehicle visual positioning method and system, and relates to the field of unmanned aerial vehicle vision, and the method comprises the steps: obtaining a satellite map of a target region, and cutting the satellite map into overlapped blocks to construct a satellite library; acquiring images shot by the unmanned aerial vehicle from a plurality of visual angles in the target geographic area range, and constructing a multi-visual-angle unmanned aerial vehicle image based on the shot images; the images are processed through a multi-view-angle unmanned aerial vehicle visual positioning network, and the network comprises an embedded coding module, a view angle consistency feature learning module and a prototype construction and alignment module and is used for extracting single-view-angle and multi-view-angle unmanned aerial vehicle image features and satellite library features; and determining the current position of the unmanned aerial vehicle by calculating the cosine similarity between the multi-view unmanned aerial vehicle image features and the satellite library features. According to the method, DINOv2 feature extraction, adapter design and the view angle invariant attention network are fused, so that efficient information processing and feature extraction are ensured, and the positioning precision and reliability are improved.

Description

Technical Field

[0001] The present invention relates to the field of unmanned aerial vehicle vision, and in particular to a multi-viewing-based unmanned aerial vehicle vision positioning method and system. Background Art

[0002] In recent years, UAV visual positioning technology has made significant progress, and its application value in environments where global navigation satellite system (GNSS) signals are limited or completely ineffective has become increasingly prominent. As an important supplement to GNSS positioning, this technology has shown broad application prospects in emergency rescue, smart cities, agricultural monitoring, military reconnaissance and other fields. Especially in the field of emergency rescue, UAV visual positioning technology can effectively break through the limitations of GPS signal attenuation or interruption in complex environments, provide accurate positioning support for life-saving operations, and significantly improve rescue efficiency. However, since UAV visual positioning involves complex visual scenes and diverse feature changes, existing technologies still face significant challenges in dealing with scale issues, perspective issues, and cross-domain issues.

[0003] The scale problem refers to the fact that during the flight of a drone, the images captured by its onboard camera have large scale differences due to changes in flight altitude, which will aggravate the scale inconsistency between drone and satellite images, posing a severe challenge to visual positioning. In addition. Existing methods usually rely on parameter fine-tuning of the pre-trained backbone network, which not only increases the computational cost, but also damages the general feature extraction ability of the pre-trained model. Therefore, a parameter efficient transfer learning strategy based on dual adapters is introduced to achieve efficient multi-scale feature extraction. This strategy significantly reduces the training overhead by freezing the parameters of the pre-trained backbone network and only fine-tuning the adapter. This method not only retains the global feature learning ability of the pre-trained model, but also enables the model to effectively deal with the scale problem in drone visual positioning by introducing multi-scale local information for specific scenes, and ultimately achieves fast and robust visual positioning effects.

[0004] The perspective problem refers to the large perspective difference between oblique drone images and orthophoto satellite images, which brings great challenges to visual positioning. Existing deep metric learning methods use contrastive learning to shorten the feature distance between drone and satellite image pairs at the same location in the embedding space, while increasing the feature distance between sample pairs at different locations, thereby learning a shared embedding space for the drone-satellite domain. However, the large perspective difference between oblique drone images and orthophoto satellite images will lead to inaccurate shared embedding space learned by current metric learning methods, which in turn reduces the generalization ability of the model to new location samples during testing. Existing contrastive learning-based methods fail to fully exploit the complementarity between multiple perspectives of drones in the same scene. Therefore, we propose perspective consistency feature learning, which constructs multi-perspective drone image features that are robust to drone perspective changes and contrasts them with orthophoto satellite image features to learn the shared embedding space for the drone-satellite domain, thereby improving matching accuracy.

[0005] The cross-domain problem refers to the differences in imaging conditions and spectral characteristics between drone and satellite images, which leads to significant feature distribution shift, which is another major challenge in achieving precise positioning. How to solve the problem of inconsistent cross-domain feature distribution and alleviate the inter-domain gap between drone and satellite images is still a key issue that needs to be solved urgently. Summary of the invention

[0006] In order to solve the above problems, the present invention proposes a UAV visual positioning method and system based on multi-perspective, which improves the adaptability and accuracy of the UAV visual positioning system by effectively addressing the three major problems of scale, perspective and cross-domain in UAV visual positioning technology.

[0007] The specific plan is as follows:

[0008] On the one hand, the multi-view UAV visual positioning method includes:

[0009] S1, obtain a satellite map of the target geographic area;

[0010] S2, cutting the satellite map into a plurality of satellite map blocks of the same size, and constructing a satellite library based on the satellite map blocks; wherein adjacent satellite map blocks have overlapping areas;

[0011] S3, obtaining images taken by the drone from multiple perspectives within the target geographical area, and constructing a multi-perspective drone image based on the taken images;

[0012] S4, input the multi-view UAV image into the multi-view UAV visual positioning network, extract the single-view UAV image features from the multi-view UAV image through the embedding coding module of the multi-view UAV visual positioning network, and input the single-view UAV image features into the view consistency feature learning module of the multi-view UAV visual positioning network to extract the multi-view UAV image features;

[0013] S5, cutting the satellite map blocks in the satellite library into the multi-view UAV visual positioning network, and extracting the satellite library features through the embedding coding module of the multi-view UAV visual positioning network;

[0014] S6, calculating the cosine similarity between the multi-view UAV image features and the satellite library features, and determining the geographical coordinates of the center point corresponding to the satellite map block with the highest similarity as the current position of the UAV.

[0015] In S4, the multi-view UAV visual positioning network further includes: a prototype construction and alignment module; the prototype construction and alignment module narrows the UAV-satellite domain gap by constructing and aligning a UAV prototype library and a satellite prototype library.

[0016] Furthermore, the UAV prototype library and the satellite prototype library respectively include the feature vectors of all locations in the UAV domain and the satellite domain in the training set. The feature vectors of all locations are set to all zero vectors during initialization. During the model training process, the UAV prototype library and the satellite prototype library are updated batch by batch in a moving average manner. The update formula is as follows:

[0017] ;

[0018] ;

[0019] in represents the i-th multi-view UAV image feature in a batch, represents the i-th satellite image feature in a batch, express The prototype vector corresponding to the real location, express The prototype vector corresponding to the real location; Indicates update parameters.

[0020] Further, in S4, the embedded coding module includes: an adapted DINOv2 feature extraction unit and a first multi-layer perceptron; the output of the adapted DINOv2 feature extraction unit is directly connected to the input end of the first multi-layer perceptron.

[0021] Furthermore, the adapted DINOv2 feature extraction unit includes 12 DINOv2 blocks connected in sequence; the DINOv2 block includes: a first layer normalization, a multi-head self-attention mechanism, a common adapter, a first residual connection unit, a second layer normalization, a second multi-layer perceptron, a multi-scale convolution adapter, and a second residual connection unit; the input original image features are sequentially input into the first layer normalization, the multi-head self-attention mechanism, and the common adapter to obtain image features processed by the common adapter, and the image features processed by the common adapter are sequentially compared with the input original image features through the first residual connection unit. The image features after the addition are obtained by adding the image features, the image features after the addition are input into the second layer for normalization to obtain the image features after the second layer normalization, the image features after the second layer normalization are input into the multi-scale convolution adapter and the second multi-layer perceptron respectively, the image features processed by the multi-scale convolution adapter are multiplied by the scaling factor s and then added element by element with the image features processed by the second multi-layer perceptron to obtain the image features after the secondary addition, the image features processed by the first residual connection unit and the output image features after the secondary addition are added element by element through the second residual connection unit to obtain the final output image features.

[0022] Furthermore, the common adapter includes a first dimensionality reduction linear layer, a RELU activation function, a first dimensionality increase linear layer and a third residual connection unit; the image features after being processed by the multi-head self-attention mechanism are first processed by the first dimensionality reduction linear layer to obtain the image features after dimensionality reduction; the image features after dimensionality reduction are nonlinearly activated by the RELU activation function to obtain the image features after nonlinear activation; the image features after nonlinear activation are dimensionally upgraded by the first dimensionality increase linear layer to obtain the image features after dimensionality increase; the image features after dimensionality increase and the image features after being processed by the multi-head self-attention mechanism are element-by-element added through the third residual connection unit to obtain the image features processed by the common adapter.

[0023] Furthermore, the multiscale convolution adapter includes a second dimension reduction linear layer, a RELU activation function, a multiscale convolution unit, a second dimension increase linear layer and a fourth residual connection unit; wherein the image features after the second layer normalization processing are first input into the second dimension reduction linear layer for dimension reduction processing to obtain the image features after dimension reduction; the image features after dimension reduction are input into the RELU activation function for nonlinear activation processing to obtain the image features after nonlinear activation; after removing the class code from the image features after nonlinear activation, they are reshaped into feature maps and input into the multiscale convolution unit for processing to obtain the feature maps processed by the multiscale convolution unit; after reshaping the feature maps processed by the multiscale convolution unit into image features, the class codes are reconnected to obtain the image features processed by the multiscale convolution unit; the image features processed by the multiscale convolution unit are element-by-element added to the image features after nonlinear activation through the fourth residual connection unit to obtain the image features processed by the fourth residual connection unit; the image features processed by the fourth residual connection unit are dimensionally enhanced through the second dimension increase linear layer, and finally the image features processed by the multiscale convolution adapter are output.

[0024] Furthermore, in S4, the perspective consistency feature learning module is implemented by a perspective invariant attention network, which includes a single-head self-attention layer, a fifth residual connection unit and an average pooling layer; the single-view UAV image feature is firstly cross-view feature weights are assigned through the single-head self-attention layer to obtain the image feature after weight assignment, and the image feature after weight assignment is added element by element with the input single-view UAV image feature through the fifth residual connection unit to obtain the image feature that retains the original information, and then the image feature that retains the original information is compressed and dimensionally regularized through the average pooling layer to obtain the multi-view UAV image feature.

[0025] On the other hand, the multi-view UAV visual positioning system includes:

[0026] A satellite map acquisition module is used to acquire a satellite map of a target geographical area;

[0027] A satellite map clipping module is used to clip the satellite map into multiple satellite map blocks of the same size, and to construct a satellite library based on the satellite map blocks; wherein adjacent satellite map blocks are provided with overlapping areas;

[0028] A multi-view UAV image construction module is used to obtain images taken by a UAV from multiple viewpoints within a target geographical area, and to construct a multi-view UAV image based on the taken images;

[0029] A multi-view UAV embedding extraction module is used to input the multi-view UAV image into the multi-view UAV visual positioning network, extract the single-view UAV image features through the embedding coding module of the multi-view UAV visual positioning network, and input the single-view UAV image features into the view consistency feature learning module of the multi-view UAV visual positioning network to extract the multi-view UAV image features;

[0030] The satellite library embedding extraction module is used to cut satellite maps in the satellite library into blocks and input them into the multi-view UAV visual positioning network, and extract satellite library features through the embedding coding module of the multi-view UAV visual positioning network;

[0031] The UAV positioning module is used to calculate the cosine similarity between the multi-view UAV image features and the satellite library features, and determine the geographical coordinates of the center point corresponding to the satellite map block with the highest similarity as the current position of the UAV.

[0032] The present invention adopts the above technical solution and has the following beneficial effects:

[0033] (1) The present invention adopts a parameter-efficient transfer learning strategy, which only fine-tunes the dual adapters while keeping the parameters of the pre-trained backbone network unchanged. This design not only significantly reduces the training time and computing cost, but also introduces multi-scale local information for specific scenarios while retaining the global learning ability of the pre-trained model. Through this design, the model can better cope with the mid-scale problem of UAV visual positioning and achieve fast and robust visual positioning.

[0034] (2) The present invention proposes perspective-consistent feature learning to construct a feature representation that is robust to changes in drone viewpoints. The single-head self-attention mechanism is used to perform weighted interactions on image features of different perspectives of the same location, effectively capturing the correlation between multiple perspectives and aggregating them into multi-perspective drone image features through average pooling operations. Multi-perspective drone image features can represent the discriminative features of a location that are invariant to the perspective, and the accuracy of cross-perspective retrieval is improved by comparative learning with satellite image features.

[0035] (3) This invention effectively alleviates the problem of inconsistent cross-domain feature distribution by introducing a dynamically maintained prototype library. The prototype library adopts a moving average update mechanism and is dynamically adjusted according to the training data to ensure that it can gradually learn and reflect the feature distribution of location discrimination within the domain; in addition, combined with the prototype-based domain alignment mechanism, the domain gap between drone and satellite image features is further reduced. This design significantly enhances the expression ability of cross-domain features and improves the positioning accuracy of the model in complex cross-domain scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a flow chart of a multi-view UAV visual positioning method according to an embodiment of the present invention;

[0037] Figure 2 is a training flow chart of a model according to an embodiment of the present invention;

[0038] Figure 3 A schematic diagram of a multi-perspective UAV image group of a multi-perspective UAV visual positioning method according to an embodiment of the present invention;

[0039] Figure 4 A schematic diagram of retrieval results of a multi-view UAV visual positioning method according to an embodiment of the present invention;

[0040] Figure 5 This is a system diagram of a multi-perspective based UAV visual positioning method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0041] The present invention is further described in detail below in conjunction with the embodiments and drawings, but the embodiments of the present invention are not limited thereto. Figure 1 As shown, the multi-view UAV visual positioning method of the present invention includes:

[0042] S1, obtain a satellite map of the target geographic area;

[0043] S2, cutting the satellite map into a plurality of satellite map blocks of the same size, and constructing a satellite library based on the satellite map blocks; wherein adjacent satellite map blocks have overlapping areas;

[0044] S3, obtaining images taken by the drone from multiple perspectives within the target geographical area, and constructing a multi-perspective drone image based on the taken images;

[0045] S4, input the multi-view UAV image into the multi-view UAV visual positioning network, extract the single-view UAV image features from the multi-view UAV image through the embedding coding module of the multi-view UAV visual positioning network, and input the single-view UAV image features into the view consistency feature learning module of the multi-view UAV visual positioning network to extract the multi-view UAV image features;

[0046] Specifically, the multi-view UAV visual positioning network also includes a prototype construction and alignment module, which narrows the UAV-satellite domain gap by constructing and aligning a UAV prototype library and a satellite prototype library.

[0047] In this embodiment, the prototype construction and alignment process is to respectively construct a drone prototype library and a satellite prototype library by a moving average method using the multi-view drone image features output by the view consistency feature learning module and the satellite image features output by the embedded coding module. They represent the discriminative features within the drone-satellite domain, and then achieve cross-domain alignment by aligning the drone prototype library and the satellite prototype library.

[0048] Specifically, the embedded coding module includes: an adapted DINOv2 feature extraction unit and a first multi-layer perceptron; the output of the adapted DINOv2 feature extraction unit is directly connected to the input end of the first multi-layer perceptron.

[0049] Specifically, the adapted DINOv2 feature extraction unit includes 12 DINOv2 blocks connected in sequence; the DINOv2 block includes: a first layer normalization, a multi-head self-attention mechanism, a common adapter, a first residual connection unit, a second layer normalization, a second multi-layer perceptron, a multi-scale convolution adapter, and a second residual connection unit; the input original image features are sequentially input into the first layer normalization, the multi-head self-attention mechanism, and the common adapter to obtain the image features processed by the common adapter, and the image features processed by the common adapter are sequentially compared with the input original image features through the first residual connection unit. The image features after the addition are obtained by adding the image features, the image features after the addition are input into the second layer for normalization to obtain the image features after the second layer normalization, the image features after the second layer normalization are input into the multi-scale convolution adapter and the second multi-layer perceptron respectively, the image features processed by the multi-scale convolution adapter are multiplied by the scaling factor s and then added element by element with the image features processed by the second multi-layer perceptron to obtain the image features after the secondary addition, the image features processed by the first residual connection unit and the output image features after the secondary addition are added element by element through the second residual connection unit to obtain the final output image features.

[0050] Specifically, when processing the input image, it is first cut into blocks according to the specified size, and then these image blocks are flattened into one-dimensional vectors. These one-dimensional vectors are processed by a linear layer to generate a series of one-dimensional vectors after linear transformation, namely tokens. At the starting position of these tokens sequences, a learnable classification vector, namely a class code (classtoken), is inserted. In order to retain the position information, each token also needs to add a learnable position code of the same dimension to obtain the original features of the input image. Specifically, the ordinary adapter includes a first dimensionality reduction linear layer, a RELU activation function, a first dimensionality increase linear layer and a third residual connection unit; the image features processed by the multi-head self-attention mechanism are first processed by the first dimensionality reduction linear layer to obtain the image features after dimensionality reduction; the image features after dimensionality reduction are nonlinearly activated by the RELU activation function to obtain the image features after nonlinear activation; the image features after nonlinear activation are dimensionally increased by the first dimensionality increase linear layer to obtain the image features after dimensionality increase; the image features after dimensionality increase and the image features processed by the multi-head self-attention mechanism are element-by-element added through the third residual connection unit to obtain the image features processed by the ordinary adapter.

[0051] Specifically, the multi-scale convolution adapter includes a second dimension reduction linear layer, a RELU activation function, a multi-scale convolution unit, a second dimension increase linear layer and a fourth residual connection unit; wherein the image features after the second layer normalization processing are first input into the second dimension reduction linear layer for dimension reduction processing to obtain the image features after dimension reduction; the image features after dimension reduction are processed by the RELU activation function for nonlinear activation to obtain the image features after nonlinear activation; after removing the class code from the image features after nonlinear activation, they are reshaped into feature maps and input into the multi-scale convolution unit for processing to obtain the feature maps processed by the multi-scale convolution unit; after reshaping the feature maps processed by the multi-scale convolution unit into image features, the class codes are reconnected to obtain the image features processed by the multi-scale convolution unit; the image features processed by the multi-scale convolution unit are element-by-element added to the image features after nonlinear activation through the fourth residual connection unit to obtain the image features processed by the fourth residual connection; the image features processed by the fourth residual connection are processed by the second dimension increase linear layer for dimensionality enhancement, and finally the image features processed by the multi-scale convolution adapter are output.

[0052] Specifically, the multi-scale convolution unit consists of three convolution layers and channel splicing. The first convolution layer includes a convolution operation unit, the convolution kernel size is 1*1, the number of convolution kernels is 192, the step size is 1, and the padding size is 0; the second convolution layer includes two convolution operation units connected in sequence, the convolution kernel size of the first convolution operation unit is 1*1, the number of convolution kernels is 24, the step size is 1, and the padding size is 0. The convolution kernel size of the second convolution operation unit is 3*3, the number of convolution kernels is 96, the step size is 1, and the padding size is 1; the third convolution layer includes two convolution operation units connected in sequence, the convolution kernel size of the first convolution operation unit is 1*1, the number of convolution kernels is 24, the step size is 1, and the padding size is 0. The convolution kernel size of the second convolution operation unit is 5*5, the number of convolution kernels is 96, the step size is 1, and the padding size is 2; the output of the three-way convolution layer is channel-joined to obtain the output of the multi-scale convolution unit. The convolution operation unit includes convolution operation, batch normalization and RELU activation function.

[0053] Specifically, after removing the ordinary adapter and the multi-scale convolution adapter, the remaining structure of the DINOv2 block constitutes a standard Transformer encoding layer; the first multi-layer perceptron includes a first linear layer, a GELU activation function, a batch normalization layer, a second linear layer and an L2 normalization layer connected in sequence, wherein the first linear layer is used to increase the input feature dimension from 768 to 1536, and the second linear layer is used to reduce the input feature dimension from 1536 to 768; the class code in the image feature output by the adapted DINOv2 feature extraction unit is extracted, and the class code is used as the input of the first multi-layer perceptron; the class code is sequentially subjected to a 2-fold dimensionality increase process by the first linear layer, then subjected to a nonlinear activation by the GELU activation function, then subjected to a normalization process by the batch normalization layer, then subjected to a 2-fold dimensionality reduction process by the second linear layer, and finally subjected to a normalization process by the L2 normalization layer to obtain the output of the first multi-layer perceptron.

[0054] Specifically, the perspective consistency feature learning module is implemented through a perspective invariant attention network, which includes a single-head self-attention layer, a fifth residual connection unit and an average pooling layer; the single-view UAV image feature is firstly subjected to cross-view feature weight allocation through the single-head self-attention layer to obtain the image feature after weight allocation, and the image feature after weight allocation is added element by element with the input single-view UAV image feature through the fifth residual connection unit to obtain the image feature that retains the original information, and then the image feature that retains the original information is compressed and dimensionally regularized through the average pooling layer to obtain the multi-view UAV image feature.

[0055] Specifically, the UAV prototype library and the satellite prototype library respectively include the feature vectors of all locations in the training set, and the feature vectors of all locations are set to all zero vectors during initialization. During the model training process, the UAV prototype library and the satellite prototype library are updated batch by batch in a moving average manner, and the update formula is as follows:

[0056] ;

[0057] ;

[0058] in represents the i-th multi-view UAV image feature in a batch, represents the i-th satellite image feature in a batch, express The prototype vector corresponding to the real location, express The prototype vector corresponding to the real location; represents the update parameter. Specifically, The update process is: ;

[0059] in, is the current network training cycle, and epoch is the total network training cycle. Preferably, The initial value is set to 0.99, the start is set to 0.9, and the end is set to 0.5.

[0060] Specifically, the multi-view UAV visual positioning network also includes a classifier, which includes a third linear layer, a fourth linear layer and a batch normalization layer; the image features first pass through the third linear layer, then pass through the batch normalization layer for normalization, and finally pass through the fourth linear layer to output the classification result. The input feature dimension of the third linear layer is 768, and the output feature dimension is 512; the input feature dimension of the fourth linear layer is 512, and the output feature dimension is the number of locations in the training set.

[0061] S5, cutting the satellite map blocks in the satellite library into the multi-view UAV visual positioning network, and extracting the satellite library features through the embedding coding module of the multi-view UAV visual positioning network;

[0062] S6, calculating the cosine similarity between the multi-view UAV image features and the satellite library features, and determining the geographical coordinates of the center point corresponding to the satellite map block with the highest similarity as the current position of the UAV.

[0063] In this example, during the training process of the multi-view UAV visual localization network, a shared weight classifier is used to map single-view UAV image features, multi-view UAV image features, satellite image features, UAV prototype library, and satellite prototype library to a location-specific embedding space. However, during the testing phase, the classifier no longer works and is therefore removed. The UAV prototype library and the satellite prototype library represent the feature representations of location discrimination in the UAV domain and the satellite domain in the training set, respectively. Through the classification loss and domain alignment loss based on the prototype library, the model is able to extract feature representations of location discrimination and domain consistency, thereby effectively alleviating cross-domain differences. These prototype libraries are used to calculate the loss function to alleviate cross-domain differences and are removed during the model testing phase. Therefore, during model testing, when the input to the network is a multi-view UAV image of a new location, the UAV prototype library and the satellite prototype library will not participate in the calculation. Before calling the pre-trained multi-view UAV visual localization network to embed and encode the images in the multi-view UAV image and the satellite library, the network needs to be trained. The training process is as follows Figure 2As shown in FIG. 1 , the specific steps are as follows: obtaining multi-view UAV and satellite training image pairs in the target area, including UAV images of selected multiple locations in the target area taken by UAV cameras from different perspectives, and constructing a multi-view UAV image library; obtaining satellite maps of the same geographical area from Google Maps; cropping satellite maps to ensure that adjacent satellite map slices overlap; pairing multi-view UAV images with satellite images of the same geographical location to construct a training data set; calling the initial multi-view UAV visual positioning network to preprocess the above multi-view UAV and satellite training image pairs, and the specific steps are as follows: first, randomly shuffle the locations in the training data set, and input the multi-view UAVs and corresponding satellite images of B locations in each batch into the embedding coding module of the multi-view UAV visual positioning network with shared weights, respectively, which includes an adapted DINOv2 feature extraction unit and a first multi-layer perceptron connected in sequence to generate single-view UAV image features and satellite image features; inputting the single-view UAV image features of B locations into the perspective consistency feature learning module of the multi-view UAV visual positioning network to obtain multi-view UAV image features of B locations; and updating the UAV prototype library and the satellite prototype library respectively by using the multi-view UAV image features and the corresponding satellite image features of B locations in a moving average manner. During initialization, both prototype libraries contain C all-zero vectors, where C is the number of locations in the training set. The single-view UAV image features, multi-view UAV image features, satellite image features and their original scores Logits obtained by the classifier are processed with the UAV prototype library, satellite prototype library and their original scores Logits obtained by the classifier respectively. The parameters of the initial multi-view UAV visual positioning network are optimized to obtain the trained multi-view UAV visual positioning network. The specific details of the loss function are as follows:

[0064] Assume that a batch consists of multi-view UAV and satellite training image pairs of B locations, and each location samples V multi-view UAV images and 1 satellite image. The loss function of the multi-view UAV visual localization network includes view invariance loss, embedding classification loss, multi-view UAV image feature and satellite image feature comparison loss, prototype classification loss, and prototype domain alignment loss. The prototype classification loss and prototype domain alignment loss are intended to further reduce the domain gap.

[0065] The view invariant loss constrains the average feature distance between the V single-view UAV image features of each location in a batch and the multi-view UAV image features of the location within a given threshold, so that the V single-view UAV image features of each location are closely clustered around the multi-view UAV image features, thereby effectively improving the accuracy of single-view retrieval (rough positioning). for:

[0066] ;

[0067] ;

[0068] ;

[0069] Where C is the number of locations in the training set; It is Location Single-view UAV image features, It is The multi-view UAV image features of the locations, represents the inner product operation, because before the calculation and It has been L2 normalized, so the inner product is equivalent to cosine similarity; is a set threshold.

[0070] The embedding classification loss includes the single-view drone embedding classification loss, satellite embedding classification loss, and multi-view drone embedding classification loss. Each classification loss uses the cross entropy loss function. The calculation formula is:

[0071] ;

[0072] ;

[0073] ;

[0074] ;

[0075] Among them, C is the number of locations in the training set, and N is the number of drone images in the training set; for single-view drone image features, is a sign function, taking values ​​of 0 or 1. When c is the real location of the i-th single-view UAV image feature, Take 1, otherwise take 0; is the probability that the i-th single-view UAV image feature is predicted by the classifier to be the c-th location. For multi-view UAV image features and satellite image features, the true location indicator and predicted probability are respectively , , and express; Embedding classification loss for single-view drones, Embedding loss for multi-view UAV classification, is the satellite embedding classification loss, It is the sum of the three, i.e., the embedding classification loss.

[0076] The InfoNCE loss is used for the contrast loss between multi-view drone image features and satellite image features. It aims to learn a shared embedding space for drone-satellite domain by narrowing the distance between multi-view drone image features and satellite image features of the same location in the embedding space, while widening the feature distance of different locations. Unlike the traditional triplet loss, this loss considers all negative samples of anchor points in a batch, so as to conduct more comprehensive contrastive learning. The loss is calculated as:

[0077] ;

[0078] ;

[0079] ;

[0080] Among them, C is the number of locations in the training set, and B is the number of locations included in a batch. is the multi-view UAV image feature of the i-th location, is the corresponding positive sample satellite image feature. is the satellite image feature of the ith location, is the corresponding positive sample multi-view drone image feature. is a logarithmic scaling factor. and They are InfoNCE contrast loss in two directions, multi-view UAV image feature and satellite image feature contrast loss. It is the sum of the two.

[0081] The prototype classification loss includes the drone prototype classification loss and the satellite prototype classification loss. Each classification loss uses the cross entropy loss function. The calculation formula is:

[0082] ;

[0083] ;

[0084] ;

[0085] Where C is the number of locations in the training set. is a sign function (0 or 1), when k is the When the real location of the drone prototype vector of locations (i.e. k is equal to i), =1, otherwise = 0, and For the The probability that the prototype vector of the drone at location k is predicted by the classifier is the probability of being at location k. Similarly, if k is When the true category of the satellite prototype vector of the location is (that is, k is equal to i), =1, otherwise = 0, and For the The satellite prototype vector of a location is predicted by the classifier as the probability of the kth location. Classify the loss for the drone prototype, is the satellite prototype classification loss, prototype classification loss It is the sum of the two.

[0086] The prototype domain alignment loss aims to align the UAV prototype library and the satellite prototype library to further alleviate the domain differences. The calculation formula is:

[0087] ; ;

[0088] ;

[0089] Where N is the number of drone images in the training set. represents the i-th single-view drone image feature, represents the prototype vector of the drone at the cth location, represents the satellite prototype vector of the c-th location, represents the satellite image features of the ith location, represents the drone domain alignment loss, represents the satellite domain alignment loss, is the overall domain alignment loss, which is the sum of the drone domain alignment loss and the satellite domain alignment loss.

[0090] The total loss function is:

[0091] ;

[0092] in, is the view invariant loss, is the embedding classification loss, is the comparison loss between multi-view UAV image features and satellite image features, is the prototype classification loss, Prototype domain alignment loss, , , , is a hyperparameter, and is a dynamic trade-off parameter that increases with the number of training rounds t. is the total loss. Based on the loss, the parameters of the initial multi-view UAV visual localization network are optimized.

[0093] Specifically, and The update formula is as follows:

[0094]

[0095]

[0096] in, is the cycle of the current network training. Preferably, max1 is set to one tenth of an epoch, and max2 is set to one half of an epoch.

[0097] like Figure 4 As shown in the figure, the trained multi-view UAV visual localization network is shown. Figure 3 The retrieval results of the three sets of multi-view drone images shown in the figure. The figure lists the top five satellite images sorted from high to low according to cosine similarity, and numbers are marked on the images, where green numbers indicate correct retrieval and red numbers indicate incorrect retrieval. Since each location corresponds to only one satellite image, the ideal retrieval result of the model is that the first image is marked green (indicating correct retrieval) and the remaining images are marked red (indicating incorrect retrieval).

[0098] Specifically, in the test process, first, the multi-view UAV image is input into the embedding coding module of the multi-view UAV visual positioning network to obtain the single-view UAV image features, and then it is input into the perspective consistency feature learning module of the multi-view UAV visual positioning network to obtain the multi-view UAV image features. At the same time, the image in the satellite library is also processed by the embedding coding module of the multi-view UAV visual positioning network to obtain the corresponding satellite library features. Positioning can be divided into two ways: single-view retrieval (rough positioning): use the single-view UAV image features to search in the satellite library features, find the satellite features with the highest cosine similarity, and the corresponding center point position of the satellite map block is the position of the UAV. Multi-view retrieval (robust positioning): use the multi-view UAV image features to search in the satellite library features, find the satellite features with the highest cosine similarity, and the corresponding center point position of the satellite map block is the positioning result of the UAV.

[0099] In this embodiment, the experimental environment is Intel(R) Core(TM) i9-12900KF CPU@3.19GHz, the memory is 128G, and the GPU processor is an NVIDA GeForce RTX 3090 24G; the training data comes from the University-1652 public data set, which contains multi-perspective drone images and satellite image pairs of 701 locations, and each location has a multi-perspective drone image consisting of 54 drone images and 1 satellite image, and the size is 512×512 pixels; during training, the multi-perspective drone images of each location are divided into three groups according to three different flight altitudes, specifically, the 1st to 18th, 19th to 36th, and 37th to 54th images correspond to the first group, the second group, and the third group, respectively, and 4 images are randomly selected from each group for training. The network structure of the DINOv2 pre-trained model we used is ViT-B, where the size of the patch is 14×14 and the dimension of the token is 768; the training uses the PyTorch deep learning framework, the optimizer selects stochastic gradient descent (SGD), and the batch size is set to 16, which means that each batch contains 192 multi-view drone images and 16 satellite images from 16 locations, a total of 208 images. The epoch is set to 30, the initial learning rate is 0.01, and the learning rate is adjusted to 0.001 for the last 10 epochs. The momentum and weight decay are set to 0.9 and 0.0005, respectively. In addition, the parameters α and β in the loss function are set to 5 and 0.01, respectively. Set to 0.5, Set to 0.5. The scaling factor s is 0.2, is 0.7.

[0100] like Figure 5 As shown, this embodiment also discloses a multi-view UAV visual positioning system, including:

[0101] A satellite map acquisition module 51 is used to acquire a satellite map of a target geographical area;

[0102] The satellite map clipping module 52 is used to clip the satellite map into a plurality of satellite map blocks of the same size, and to construct a satellite library based on the satellite map blocks; wherein adjacent satellite map blocks are provided with overlapping areas;

[0103] A multi-view UAV image construction module 53, used to obtain images taken by a UAV from multiple viewpoints within the target geographical area, and construct a multi-view UAV image;

[0104] A multi-view UAV embedding extraction module 54 is used to input the multi-view UAV image into the multi-view UAV visual positioning network, extract the single-view UAV image features from the multi-view UAV image through the embedding coding module of the multi-view UAV visual positioning network, and input the single-view UAV image features into the view consistency feature learning module of the multi-view UAV visual positioning network to extract the multi-view UAV image features;

[0105] The satellite library embedding extraction module 55 is used to cut the satellite map blocks in the satellite library into the multi-view UAV visual positioning network, and extract the satellite library features through the embedding coding module of the multi-view UAV visual positioning network;

[0106] The drone positioning module 56 is used to calculate the cosine similarity between the multi-view drone image features and the satellite library features, and determine the geographical coordinates of the center point corresponding to the satellite map block with the highest similarity as the current position of the drone.

[0107] The specific implementation of the UAV visual positioning system based on multiple perspectives is the same as the UAV visual positioning method based on multiple perspectives, and will not be repeated in this embodiment.

[0108] Although the present invention has been specifically shown and described in conjunction with the preferred embodiments, it should be understood by those skilled in the art that various changes may be made to the present invention in form and details without departing from the spirit and scope of the present invention as defined by the appended claims, all of which are within the scope of protection of the present invention.

Claims

1. A multi-view UAV visual positioning method, characterized in that: include: S1, obtain a satellite map of the target geographic area; S2, cutting the satellite map into a plurality of satellite map blocks of the same size, and constructing a satellite library based on the satellite map blocks; wherein adjacent satellite map blocks have overlapping areas; S3, obtaining images taken by the drone from multiple perspectives within the target geographical area, and constructing a multi-perspective drone image based on the taken images; S4, input the multi-view UAV image into the multi-view UAV visual positioning network, extract the single-view UAV image features from the multi-view UAV image through the embedding coding module of the multi-view UAV visual positioning network, and input the single-view UAV image features into the view consistency feature learning module of the multi-view UAV visual positioning network to extract the multi-view UAV image features; S5, cutting the satellite map blocks in the satellite library into the multi-view UAV visual positioning network, and extracting the satellite library features through the embedding coding module of the multi-view UAV visual positioning network; S6, calculating the cosine similarity between the multi-view UAV image features and the satellite library features, and determining the geographical coordinates of the center point corresponding to the satellite map block with the highest similarity as the current position of the UAV.

2. The multi-view UAV visual positioning method according to claim 1, characterized in that: In S4, the multi-view UAV visual positioning network also includes: a prototype construction and alignment module; the prototype construction and alignment module narrows the gap between the UAV domain and the satellite domain by constructing and aligning the UAV prototype library and the satellite prototype library.

3. The multi-view UAV visual positioning method according to claim 2 is characterized in that: The UAV prototype library and the satellite prototype library respectively include the feature vectors of all locations in the UAV domain and the satellite domain in the training set. The feature vectors of all locations are set to all zero vectors during initialization. During the model training process, the UAV prototype library and the satellite prototype library are updated batch by batch in a moving average manner. The update formula is as follows: ； ； in represents the i-th multi-view UAV image feature in a batch, represents the i-th satellite image feature in a batch, express The prototype vector corresponding to the real location, express The prototype vector corresponding to the real location; Indicates update parameters.

4. The multi-view UAV visual positioning method according to claim 1, characterized in that: In S4, the embedded coding module includes: an adapted DINOv2 feature extraction unit and a first multi-layer perceptron; the output of the adapted DINOv2 feature extraction unit is directly connected to the input end of the first multi-layer perceptron.

5. The multi-view UAV visual positioning method according to claim 4 is characterized in that: The adapted DINOv2 feature extraction unit includes 12 DINOv2 blocks connected in sequence; the DINOv2 block includes: a first layer normalization, a multi-head self-attention mechanism, a common adapter, a first residual connection unit, a second layer normalization, a second multi-layer perceptron, a multi-scale convolution adapter, and a second residual connection unit; the input original image features are sequentially input into the first layer normalization, the multi-head self-attention mechanism, and the common adapter to obtain the image features processed by the common adapter, and the image features processed by the common adapter are element-by-element compared with the input original image features through the first residual connection unit. The image features after addition are obtained, the added image features are input into the second layer normalization to obtain the second layer normalized image features, the second layer normalized image features are respectively input into the multi-scale convolution adapter and the second multi-layer perceptron, the image features processed by the multi-scale convolution adapter are multiplied by the scaling factor s and then added element by element with the image features processed by the second multi-layer perceptron to obtain the image features after secondary addition, the image features processed by the first residual connection unit and the output image features after secondary addition are added element by element through the second residual connection unit to obtain the final output image features.

6. The multi-view UAV visual positioning method according to claim 5, characterized in that: The common adapter includes a first dimensionality reduction linear layer, a RELU activation function, a first dimensionality increase linear layer and a third residual connection unit; the image features after being processed by the multi-head self-attention mechanism are first processed by the first dimensionality reduction linear layer to obtain the image features after dimensionality reduction; the image features after dimensionality reduction are nonlinearly activated by the RELU activation function to obtain the image features after nonlinear activation; the image features after nonlinear activation are dimensionally upgraded by the first dimensionality increase linear layer to obtain the image features after dimensionality increase; the image features after dimensionality increase and the image features after being processed by the multi-head self-attention mechanism are element-by-element added through the third residual connection unit to obtain the image features processed by the common adapter.

7. The multi-view UAV visual positioning method according to claim 5, characterized in that: The multiscale convolution adapter includes a second dimension reduction linear layer, a RELU activation function, a multiscale convolution unit, a second dimension increase linear layer and a fourth residual connection unit; wherein the image features after the second layer normalization processing are first input into the second dimension reduction linear layer for dimension reduction processing to obtain the image features after dimension reduction; the image features after dimension reduction are input into the RELU activation function for nonlinear activation processing to obtain the image features after nonlinear activation; after removing the class code from the image features after nonlinear activation, they are reshaped into feature maps and input into the multiscale convolution unit for processing to obtain the feature maps processed by the multiscale convolution unit; after reshaping the feature maps processed by the multiscale convolution unit into image features, the class codes are reconnected to obtain the image features processed by the multiscale convolution unit; the image features processed by the multiscale convolution unit are element-by-element added to the image features after nonlinear activation through the fourth residual connection unit to obtain the image features processed by the fourth residual connection unit; the image features processed by the fourth residual connection unit are dimensionally enhanced through the second dimension increase linear layer, and finally the image features processed by the multiscale convolution adapter are output.

8. The multi-view UAV visual positioning method according to claim 1, characterized in that: In S4, the perspective consistency feature learning module is implemented through a perspective invariant attention network, which includes a single-head self-attention layer, a fifth residual connection unit and an average pooling layer; the single-view UAV image feature is firstly cross-view feature weights are assigned through the single-head self-attention layer to obtain the image feature after weight assignment, and the image feature after weight assignment is added element by element with the input single-view UAV image feature through the fifth residual connection unit to obtain the image feature that retains the original information, and then the image feature that retains the original information is compressed and dimensionally regularized through the average pooling layer to obtain the multi-view UAV image feature.

9. A multi-view UAV visual positioning system, characterized in that: include: A satellite map acquisition module is used to acquire a satellite map of a target geographical area; A satellite map clipping module is used to clip the satellite map into multiple satellite map blocks of the same size, and to construct a satellite library based on the satellite map blocks; wherein adjacent satellite map blocks are provided with overlapping areas; A multi-view UAV image construction module is used to obtain images taken by a UAV from multiple viewpoints within a target geographical area, and to construct a multi-view UAV image based on the taken images; A multi-view UAV embedding extraction module is used to input the multi-view UAV image into the multi-view UAV visual positioning network, extract the single-view UAV image features through the embedding coding module of the multi-view UAV visual positioning network, and input the single-view UAV image features into the view consistency feature learning module of the multi-view UAV visual positioning network to extract the multi-view UAV image features; The satellite library embedding extraction module is used to cut satellite maps in the satellite library into blocks and input them into the multi-view UAV visual positioning network, and extract satellite library features through the embedding coding module of the multi-view UAV visual positioning network; The UAV positioning module is used to calculate the cosine similarity between the multi-view UAV image features and the satellite library features, and determine the geographical coordinates of the center point corresponding to the satellite map block with the highest similarity as the current position of the UAV.

Images