Unsupervised cross-view scene matching localization based on visual large model, system and medium

Abstract

The embodiment of the application discloses a kind of based on visual large model's uncontrolled cross view scene matching positioning, system and medium, method includes: first, the image to be positioned unmanned aerial vehicle and satellite image library are obtained, two are as input image;Again, the multiple intermediate layer feature maps of input image are extracted by pre-training DINOv3 visual large model, feature vector is generated by generalized mean pooling, and geometric correction feature is obtained by doing geometric correction;Then, the feature vectors of preset intermediate layer level are spliced, and structure enhancement features are obtained by frequency domain transformation and filtering;Finally, two kinds of features are spliced to generate global descriptor, which considers geometric alignment and structural stability, can make the similarity calculation of unmanned aerial vehicle and satellite image more accurate, and determine the absolute geographic position of unmanned aerial vehicle according to similarity sorting.The positioning accuracy of the method is high, and the efficiency is greatly improved, and can meet the demand of positioning real-time and accuracy in unmanned aerial vehicle autonomous navigation, emergency rescue, geographic surveying and mapping and other engineering scenarios.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) matching navigation and positioning technology, and in particular to an unsupervised cross-view scene matching and positioning system and medium based on a large visual model. Background Technology

[0002] In unmanned aerial vehicle (UAV) matching navigation and positioning, accurate positioning is crucial. When UAVs fly in environments with poor or interfered GNSS (Global Navigation Satellite System) signal coverage, traditional positioning techniques may fail to provide reliable location information. To address this issue, unsupervised cross-view scene matching positioning technology has emerged, aiming to determine the precise location of the UAV by comparing real-world images captured by the UAV with satellite remote sensing orthophotos. The core of this technology lies in its ability to match images based on similarity, without relying on pre-labeled data or additional auxiliary equipment.

[0003] Existing cross-view scene matching and localization methods are typically based on symmetric linear projection models. These methods perform poorly when dealing with complex geometric distortions between UAV and satellite imagery (such as viewpoint tilt, rotation, and scale changes). Especially when UAVs are captured in complex flight attitudes such as hovering or rotating, traditional methods struggle to effectively correct these nonlinear deformations, leading to increased matching failure rates, reduced localization accuracy, and weak generalization capabilities. Furthermore, while some clustering-based pseudo-label generation methods can improve matching robustness to some extent, noise interference can easily cause feature alignment failures and decreased localization accuracy.

[0004] Specifically, traditional methods often use a single network or linear adapter to process images during feature extraction. Such designs struggle to simultaneously address local perspective distortion and overall rotational distortion. While methods based on statistical whitening or dimensionality reduction can reduce the correlation between features in the temporal or spatial domains, they also struggle to distinguish useful information from useless environmental noise. These issues render existing technologies inadequate when faced with complex geometric distortions and diverse visual environments, making it difficult to meet the navigation needs of UAVs under various complex conditions. Summary of the Invention

[0005] Based on this, it is necessary to address the above problems by proposing an unsupervised cross-view scene matching and localization system and medium based on a large visual model.

[0006] An unsupervised cross-view scene matching and localization method based on a large visual model, the method comprising: Acquire images of the drone to be located and a satellite image library, and use the current satellite images of the drone to be located and the satellite image library as input images; The input image is used to extract features using a pre-trained DINOv3 visual large model to obtain multiple intermediate layer feature maps, and generalized mean pooling is performed on each layer feature map to generate feature vectors. Perform geometric correction on the feature vector to obtain geometrically corrected features; The feature vectors of the UAV image and the current satellite image at a preset intermediate level are stitched together, and the stitched feature vectors are subjected to frequency domain transformation and filtering to obtain structural enhancement features; The geometric correction features and structural enhancement features are spliced ​​together to generate a global descriptor. The similarity between the global descriptors of the UAV image to be located and each image in the satellite image library is calculated. The location result of the UAV is determined based on the similarity ranking result.

[0007] Specifically, performing geometric correction on the feature vector to obtain geometrically corrected features includes: The feature vectors corresponding to the satellite images are encoded to obtain satellite coded features; The feature vector corresponding to the UAV image is encoded to obtain UAV encoded features, and the UAV encoded features are corrected to obtain corrected features; The correction features and satellite coding features are used as geometric correction features.

[0008] Specifically, the step of correcting the UAV coding features to obtain corrected features includes: according to The UAV coding features are corrected to obtain corrected features, wherein, Encoding features for drones, This is the GeLU activation function. These are the weight parameters for the fusion layer. For the expansion rate One-dimensional convolution operation, To correct features.

[0009] Specifically, the step of stitching together the feature vectors of the UAV image and satellite image at a preset intermediate level, and then performing frequency domain transformation and filtering on the stitched feature vectors to obtain structural enhancement features includes: The feature vectors of UAV images and satellite images at a preset intermediate level are stitched together respectively; Perform a Fourier transform on the concatenated feature vectors to obtain the spectral representation, and decompose the spectral representation into an amplitude spectrum and a phase spectrum; The amplitude spectrum is filtered to obtain the corrected spectral representation; The modified spectral representation is subjected to inverse Fourier transform to obtain the structural enhancement features.

[0010] Specifically, the process of splicing the geometric correction features and structural enhancement features to generate a global descriptor, calculating the cosine similarity between the global descriptors of the UAV image to be located and each image in the satellite image database, and determining the UAV's location result based on the ranking of the cosine similarity results includes: according to The corrected features and the structural enhancement features corresponding to the UAV image are stitched together to determine the first stitched feature, wherein, To correct features, Structural enhancement features for drone images, This is the first splicing feature; according to The satellite coding features and the structural enhancement features corresponding to the satellite image are spliced ​​together to determine the second splicing feature, wherein, For satellite coding features, To enhance the structural features of satellite images, This is the second splicing feature; according to Determine the features of the queried drone image, where, To query drone image features; according to Identify all satellite features in the satellite image database, among which, The feature of the i-th satellite in the satellite image database; according to Determine the cosine similarity between the features of the queried UAV image and all satellite features in the satellite image database, wherein, To query the cosine similarity between UAV image features and the i-th satellite feature in the satellite image database; The cosine similarity scores are sorted according to a preset order, and the satellite image with the highest cosine similarity score is the UAV's positioning result.

[0011] An unsupervised cross-view scene matching and localization system based on a large visual model, the system comprising: The image acquisition module is used to acquire images of the drone to be located and a satellite image library, and to use the images of the drone to be located and the current satellite images in the satellite image library as input images; The feature extraction module is used to extract features from the input image using a pre-trained DINOv3 visual large model, obtain multiple intermediate layer feature maps, and perform generalized mean pooling on each layer feature map to generate feature vectors. The geometric correction module is used to perform geometric correction on the feature vector and obtain geometric correction features; The intrinsic structure mining module is used to stitch together the feature vectors of UAV images and current satellite images at a preset intermediate level, and then perform frequency domain transformation and filtering on the stitched feature vectors to obtain structure enhancement features. The UAV positioning module is used to stitch together the geometric correction features and structural enhancement features to generate a global descriptor, calculate the similarity between the global descriptors of the UAV image to be located and each image in the satellite image library, and determine the UAV positioning result based on the similarity ranking result.

[0012] The system also includes: The training module acquires UAV image datasets and satellite image datasets, selecting an equal number of UAV and satellite images as a training batch. Images from the same training batch are input into the feature extraction module, which outputs feature vectors. The feature vectors are then input into the geometric correction module, which outputs geometrically corrected features. The transmission cost matrix is ​​determined based on the cosine similarity of the geometrically corrected features. Soft pseudo-labels are generated from the transmission cost matrix using the Sinkhorn algorithm. A loss value is determined based on the cosine similarity and the soft pseudo-labels. The parameters of the geometric correction module are optimized based on the loss value.

[0013] Specifically, the process of determining the transmission cost matrix based on the cosine similarity of the geometric correction features, generating soft pseudo-labels from the transmission cost matrix using the Sinkhorn algorithm, determining the loss value based on the cosine similarity and the soft pseudo-labels, and optimizing the parameters of the geometric correction module based on the loss value includes: In a training batch, the cosine similarity between the UAV features and the satellite features is calculated to obtain a cosine similarity matrix; The transmission cost matrix is ​​determined based on the cosine similarity matrix. The optimal transmission scheme is taken as the task objective. An optimal transmission model is constructed based on the transmission cost matrix. The optimal transmission model is solved iteratively using the Sinkhorn algorithm to generate soft pseudo-labels. The transmission scheme is the pairing scheme of the UAV images and satellite images in a training batch. The total loss function is determined based on the cosine similarity matrix and the soft pseudo-labels. The loss value is determined based on the total loss function, and the parameters of the geometric correction module are optimized based on the loss value.

[0014] Specifically, determining the total loss function based on the cosine similarity matrix and the soft pseudo-labels includes: according to Determine the loss function for two-way soft distillation, where, This is the loss function for two-way soft distillation. Cosine similarity matrix Distribution after Softmax Soft fake tags The row normalized value, Let B be the KL divergence and B be the training batch. according to Determine the cross-entropy loss function, where, Let cross-entropy be the loss function. The distribution of the similarity matrix from the UAV to the satellite direction after applying Softmax is the corresponding term. The distribution of the similarity matrix from satellite to UAV direction after applying Softmax represents the corresponding terms. In soft fake tags Matching pairs where both the middle row and column are maximum values Samples that are similar to a threshold; according to Determine the reconstruction loss function, where, To reconstruct the loss function, Original features The reconstructed features are calculated after the features are mapped back to the original space using a decoder. according to Determine the total loss function, where L is the total loss function.

[0015] A computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to perform the steps of the method described above.

[0016] The embodiments of the present invention have the following beneficial effects: First, this application only needs to obtain images of the UAV to be located and a satellite image library covering the same geographical area to complete the entire process of feature extraction, matching and positioning. This effectively solves the problems of large workload, high cost and positioning error affecting positioning accuracy of traditional cross-view positioning methods, which greatly reduces the threshold for implementation and can be widely adapted to UAV positioning needs in unlabeled scenarios.

[0017] Furthermore, the pre-trained DINOv3 visual large model is used to quickly extract features from the input image, obtain multiple intermediate layer feature maps, and perform generalized mean pooling on each layer feature map, which reduces model parameters, preserves key details, and improves computational efficiency.

[0018] Furthermore, geometric correction is performed on the feature vectors to effectively correct the nonlinear geometric distortion caused by low-altitude tilt shooting of UAV images, so that they can be accurately aligned with the geometric features of satellite images (orthogonal top-down view, without geometric distortion). At the same time, by stitching together preset intermediate-level feature vectors and performing frequency domain transformation and filtering, structural enhancement features are extracted, high-frequency noise in UAV images (such as interference caused by changes in light and shadow and shooting shake) is suppressed, and the core structural information of ground objects is preserved. This achieves stable matching between UAV images and satellite images across viewpoints and scales, and significantly improves the robustness and accuracy of matching.

[0019] Finally, by splicing geometric correction features and structural enhancement features to generate a global descriptor, both geometric alignment and structural stability are taken into account. This makes the similarity calculation between UAV images and images in the satellite image library more accurate. The absolute geographical location of the UAV determined based on the similarity ranking results not only has higher positioning accuracy but also significantly improves positioning efficiency. This can meet the real-time and accurate positioning requirements of practical engineering scenarios such as UAV autonomous navigation, emergency rescue, and geographic surveying. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] in: Figure 1 This is a flowchart illustrating an embodiment of an unsupervised cross-view scene matching and localization method based on a large visual model provided by the present invention. Figure 2 A schematic diagram of the structure of an embodiment of an unsupervised cross-view scene matching and localization system for a large visual model provided by the present invention; Figure 3 This is a schematic diagram of the feature extraction module based on a large visual model provided by the present invention; Figure 4 This is a schematic diagram of the geometric correction module provided by the present invention; Figure 5 A schematic diagram of the intrinsic structure mining module provided by the present invention; Figure 6 This is a schematic diagram of another embodiment of the unsupervised cross-view scene matching and localization system based on a large visual model provided by the present invention. Figure 7 A schematic diagram illustrating the generation and distribution optimization of soft pseudo-tags based on alignment using the optimal transmission algorithm provided by this invention; Figure 8 A schematic diagram of the structure of an embodiment of the medium provided by the present invention. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] As shown in the figure Figure 1 This is a flowchart illustrating an embodiment of an unsupervised cross-view scene matching and localization method based on a large visual model provided by the present invention. The method includes: S101: Obtain the drone image to be located and the satellite image library, and use the drone image to be located and the current satellite image in the satellite image library as input images.

[0024] For example, a drone equipped with a high-definition visible light camera performs a low-altitude flight photography mission over an area to be located (such as urban building clusters, mountainous terrain, industrial parks, etc.) to acquire drone images for positioning. Simultaneously, the satellite image library uses satellite remote sensing orthophotos covering the entire area to be located. These satellite images are sourced from publicly available satellite remote sensing databases (such as the Gaofen satellite database and the Landsat satellite database) or customized satellite imagery. The satellite images are optical orthophotos, and undergo professional geometric correction processing to eliminate geometric distortions during satellite imaging, ensuring a regular geometric structure and accurately reflecting the actual spatial location and proportions of ground features. The resolution of the satellite images matches the resolution of the drone images to be located.

[0025] S102: Use the pre-trained DINOv3 visual large model to extract features from the input image, obtain multiple intermediate layer feature maps, and perform generalized mean pooling on each layer feature map to generate feature vectors.

[0026] For example, in conjunction with reference Figure 2 and Figure 3 , Figure 2 This is a schematic diagram of an embodiment of an unsupervised cross-view scene matching and localization system based on a large visual model provided by the present invention. Figure 3This is a schematic diagram of the feature extraction module based on a large visual model provided by this invention. The ViT-L DINOv3 model is selected as the feature extraction base. To balance high-level semantic information and low-level geometric texture information, the output of the last layer of the model is not used; instead, the intermediate layer feature maps of layers 21 and 23 are extracted. The pre-trained DINOv3 large visual model is used to extract features from the input drone image and the current satellite image, respectively. For a single input image... (Drone image or satellite image), DINOv3's The layer output feature map is denoted as ,in, This represents the spatial resolution of the feature map. For the channel dimension (for the ViT-L model selected in this invention, ).

[0027] To filter background noise and focus on salient target regions, instead of standard average pooling, generalized mean pooling is applied to each feature map layer. The feature vector after generalized mean pooling... The calculation formula is as follows: ; in, Representing the spatial location of the feature map The feature vector at that location, In this embodiment, the pooling parameter is set as follows: This operation can effectively enhance the response of features to salient targets and suppress irrelevant background.

[0028] For each drone and satellite image, feature vectors (1×1024 dimension) of layers 21 and 23 are extracted and calculated, and these vectors are saved as offline files to the feature database. This eliminates the need for cumbersome forward inference of large models during training; only lightweight feature vectors need to be loaded, greatly improving training efficiency.

[0029] S103: Perform geometric correction on the feature vector to obtain the geometrically corrected features.

[0030] For example, the feature vector extracted from the 23rd intermediate layer is selected as input. (See also...) Figure 4 , Figure 4 This is a schematic diagram of the geometric correction module provided by the present invention. It converts the feature vector of the satellite image... As a geometric reference, only a single shared feature encoder is used for encoding to maintain the stability of its distribution and obtain satellite-coded features. For the feature vector of a drone image The UAV coded features are obtained in the shared feature encoder. Building upon this foundation, a geometric correction module was constructed. This module utilizes one-dimensional convolution operations with different dilation rates to expand the receptive field without increasing the number of parameters. Small dilation rates focus on local detail fine-tuning, while large dilation rates capture global structural correction. This allows the module to capture multi-scale contextual information, from local detail fine-tuning to global rotation correction, within the one-dimensional sequence of feature vectors. This enables a more refined and accurate fit to real manifold distortions, significantly improving the matching success rate under complex flight attitudes.

[0031] Assume the feature vector of the input drone image is The coded features of the UAV after processing by the shared feature encoder are: The geometric correction module will The data is split into three parallel processing branches, each using a different expansion rate. One-dimensional convolutional layers are used to capture distortion features at different scales: Branch 1 (Local Perception) ): Focus on the nearest neighbor correlation between feature channels and correct for minor differences.

[0032] Branch 2 (Mid-range perception) ): Focus on feature dependencies with medium spans and correct for medium differences.

[0033] Branch 3 (Global Awareness) ): Focus on long-distance channel correlation and correct large-angle distortion differences.

[0034] The outputs of the three branches are processed by the GeLU activation function, then concatenated and fused. A 1×1 point convolutional layer is then used to map the dimensions back to the original space, and finally the result is superimposed onto the input as a residual. Above. Corrected drone features. The calculation formula is: ; in, Encoding features for drones, This is the GeLU activation function. These are the weight parameters for the fusion layer. For the expansion rate One-dimensional convolution operation, To correct the features. This multi-scale parallel residual design enables the model to adaptively fit complex nonlinear manifold distortions, outputting geometric correction features with a dimension of 2048.

[0035] S104: Separately stitch the feature vectors of the UAV image and the current satellite image at the preset intermediate level, perform frequency domain transformation and filtering on the stitched feature vectors, and obtain structural enhancement features.

[0036] For example, in conjunction with reference Figure 5 , Figure 5 This is a schematic diagram of the intrinsic structure mining module provided by the present invention. This module uses Fast Fourier Transform (FFT) to transform features to the frequency domain, and mines the essential structural information of the image by suppressing high-frequency noise components, thereby removing redundancy and enhancing the robustness of the features.

[0037] The feature vectors of the UAV image and the current satellite image are respectively stitched together at a preset intermediate level (levels 21 and 23 are selected in this example), and used as input, denoted as... First, the concatenated feature vectors are... Perform a Fourier transform to map it to the frequency domain to obtain its spectral representation. And decompose it into amplitude spectrum and phase spectrum : ; Based on the theoretical assumption that amplitude spectrum contains low-level style and noise information, and phase spectrum contains high-level semantic structure information, this invention designs a learnable spectrum-gated mask. An adaptive filter is applied to the amplitude spectrum. This mask tends to suppress high-frequency components representing redundant textures and random noise, while retaining low-frequency components representing the main structure. The amplitude spectrum is filtered based on a frequency-domain gated mask to obtain a corrected spectral representation. : ; in, This indicates element-wise multiplication, where j represents the imaginary unit. The phase factor refers to the high-level semantic structure information in the image features.

[0038] Subsequently, the corrected spectrum was obtained by inverse Fourier transform. By restoring the feature space to the real number space, we obtain the enhanced structure features. .

[0039] To further ensure that the enhanced features possess statistical independence and high discriminative power, this invention introduces orthogonality constraints at the output end, forcing the feature matrix to... The column vectors should be as orthogonal as possible to prevent feature dimension collapse: ; in, It is the identity matrix. It is a collection of all structural enhancement features within a training batch. Finally, through the combined effect of frequency domain filtering and orthogonal constraints, a structural enhancement feature (1024 dimensions) is output, which removes high-frequency interference and preserves the intrinsic structure.

[0040] It should be noted that when acquiring drone image datasets and satellite image datasets, an equal number of drone images and satellite images are selected as a training batch.

[0041] S105: Generate a global descriptor by splicing geometric correction features and structural enhancement features, calculate the similarity between the global descriptors of the UAV image to be located and each image in the satellite image library, and determine the UAV's location result based on the similarity ranking result.

[0042] For example, in conjunction with reference Figure 2 The output features of the geometric correction module and the intrinsic structure mining module are fused to obtain a global descriptor that has both geometric robustness and structural discriminative power.

[0043] Specifically, let the correction features of the UAV image output by the geometric correction module be... The satellite coding features of satellite images are The structural enhancement features of the UAV images output by the intrinsic structure mining module are: The structural enhancement features of satellite images are .

[0044] First, the features output by the geometric correction module and the intrinsic structure mining module are L2 normalized separately, and then concatenated along the channel dimensions: ; ; in, This is the first splicing feature. This is the second splicing feature.

[0045] Next, the first splicing feature was analyzed again. Second splicing features L2 normalization is performed to eliminate the influence of dimensions, thereby obtaining the features of the queried UAV image. Compared with all features in the candidate satellite image library The calculation process is as follows: ; ; Finally, calculate the features of the query drone image. Compared with all features in the candidate satellite image library Cosine similarity score between them: ; Satellite images are sorted from highest to lowest based on similarity scores, and the top-ranked satellite image is the location result for the drone.

[0046] As can be seen from the above description, firstly, the present invention only needs to acquire images of the UAV to be located and a satellite image library covering the same geographical area to complete the entire process of feature extraction, matching and positioning. It effectively solves the problems of large workload, high cost and positioning error affecting positioning accuracy of traditional cross-view positioning methods, greatly reducing the threshold for implementation of the method and making it widely adaptable to UAV positioning needs in unlabeled scenarios.

[0047] Furthermore, the pre-trained DINOv3 visual large model is used to quickly extract features from the input image, obtain multiple intermediate layer feature maps, and perform generalized mean pooling on each layer feature map, which reduces model parameters, preserves key details, and improves computational efficiency.

[0048] Furthermore, geometric correction is performed on the feature vectors to effectively correct the nonlinear geometric distortion caused by low-altitude tilt shooting in UAV images, enabling precise alignment with the geometric features of satellite images (orthogonal top-down view, without geometric distortion). Simultaneously, by stitching together preset intermediate-level feature vectors and performing frequency domain transformation and filtering, structural enhancement features are extracted, suppressing high-frequency noise in UAV images (such as interference caused by changes in light and shadow and shooting shake), and preserving the core structural information of ground features. This achieves stable matching between UAV images and satellite images across perspectives and scales, significantly improving the robustness and accuracy of the matching.

[0049] Finally, by splicing geometric correction features and structural enhancement features to generate a global descriptor, both geometric alignment and structural stability are taken into account. This makes the similarity calculation between UAV images and images in the satellite image library more accurate. The absolute geographical location of the UAV determined based on the similarity ranking results not only has higher positioning accuracy but also significantly improves positioning efficiency. This can meet the real-time and accurate positioning requirements of practical engineering scenarios such as UAV autonomous navigation, emergency rescue, and geographic surveying.

[0050] like Figure 6 As shown, Figure 6 This is a schematic diagram of another embodiment of the unsupervised cross-view scene matching and localization system based on a large visual model provided by the present invention. An unsupervised cross-view scene matching and localization system 10 based on a large visual model includes: The image acquisition module 11 is used to acquire images of the UAV to be located and a satellite image library, and uses the images of the UAV to be located and the current satellite images in the satellite image library as input images.

[0051] The feature extraction module 12 is used to extract features from the input image using the pre-trained DINOv3 visual large model, obtain multiple intermediate layer feature maps, and perform generalized mean pooling on each layer feature map to generate feature vectors.

[0052] The geometric correction module 13 is used to perform geometric correction on the feature vector and obtain the geometric correction features.

[0053] The intrinsic structure mining module 14 is used to stitch together the feature vectors of the UAV image and the current satellite image at a preset intermediate level, and to perform frequency domain transformation and filtering on the stitched feature vectors to obtain structure enhancement features.

[0054] The UAV positioning module 15 is used to stitch together geometric correction features and structural enhancement features to generate a global descriptor, calculate the similarity between the global descriptor of the UAV image to be located and each image in the satellite image library, and determine the UAV positioning result based on the similarity ranking result.

[0055] For example, in the image acquisition module 11, the image of the UAV to be located and a satellite image library are acquired, and the current satellite image in the UAV image to be located and the satellite image library are used as input images. In the feature extraction module 12, features are extracted from the input image using a pre-trained DINOv3 visual large model to obtain multiple intermediate layer feature maps, and generalized mean pooling is performed on each layer feature map to generate feature vectors. In the geometric correction module 13, the feature vectors corresponding to the satellite image are encoded to obtain satellite encoded features; based on the satellite encoded features, the feature vectors corresponding to the UAV image are corrected to obtain corrected features; the corrected features and satellite encoded features are used as geometric correction features. In the intrinsic structure mining module 14, the feature vectors of the UAV image and the satellite image at the preset intermediate layers are spliced ​​together; the spliced ​​feature vectors are subjected to Fourier transform to obtain a spectral representation, and the spectral representation is decomposed into an amplitude spectrum and a phase spectrum; the amplitude spectrum is filtered to obtain a corrected spectral representation; the corrected spectral representation is subjected to an inverse Fourier transform to obtain structural enhancement features.

[0056] Finally, in the UAV positioning module 15, the corrected features and the structural enhancement features corresponding to the UAV image are concatenated according to the formula shown below to determine the first concatenated feature: ; in, For geometric correction features, Structural enhancement features for drone images, This is the first splicing feature.

[0057] The satellite-coded features and the corresponding structural enhancement features of the satellite image are concatenated according to the formula shown below to determine the second concatenated feature: ; in, For satellite coding features, To enhance the structural features of satellite images, This is the second splicing feature.

[0058] The features of the queried drone image are determined using the formula shown below: ; in, To query the features of drone images.

[0059] All satellite features in the satellite image database are determined using the formula shown below: ; in, Let i be the feature of the i-th satellite in the satellite image database.

[0060] The cosine similarity between the features of the queried UAV image and all satellite features in the satellite image database is determined using the formula shown below: ; in, To query the cosine similarity between UAV image features and the i-th satellite feature in the satellite image database.

[0061] The cosine similarity scores are sorted according to a preset order, and the satellite image with the highest cosine similarity score is the drone's positioning result.

[0062] The system also includes a training module for acquiring UAV image datasets and satellite image datasets. An equal number of UAV and satellite images are selected as a training batch. Images from the same training batch are input into a feature extraction module, which outputs feature vectors. These feature vectors are then input into a geometric correction module, which outputs geometrically corrected features. The transmission cost matrix is ​​determined based on the cosine similarity of the geometrically corrected features. Soft pseudo-labels are generated from the transmission cost matrix using the Sinkhorn algorithm. The loss value is determined based on the cosine similarity and the soft pseudo-labels. The parameters of the geometric correction module are then optimized based on the loss value.

[0063] For example, in the training module, firstly, a drone image dataset and a satellite image dataset of the same target geographical area are obtained, and an equal number of drone images and satellite images are selected as a training batch. The images of the same training batch are then input into the feature extraction module.

[0064] Furthermore, in conjunction with reference Figure 2The drone and satellite images from the training batch are input into the pre-trained DINOv3 feature extraction module to extract the intermediate layer feature maps of the model. Generalized mean pooling is then performed on these intermediate layer feature maps to convert them into 1×1024-dimensional feature vectors. Specifically, the ViT-L model of DINOv3 is selected as the feature extraction base. To balance high-level semantic information and low-level geometric texture information, the output of the last layer of the model is not used; instead, the intermediate layer feature maps of layers 21 and 23 are extracted. The pre-trained DINOv3 large-scale visual model is used to extract features from the input drone and satellite images respectively. For a single input image... (Drone image or satellite image), DINOv3's The layer output feature map is denoted as ,in, This represents the spatial resolution of the feature map. For the channel dimension (for the ViT-L model selected in this invention, ).

[0065] To filter background noise and focus on salient target regions, instead of standard average pooling, generalized mean pooling is applied to each feature map layer. The feature vector after generalized mean pooling... The calculation formula is as follows: ; in, Representing the spatial location of the feature map The feature vector at that location, In this embodiment, the pooling parameter is set as follows: This operation can effectively enhance the response of features to salient targets and suppress irrelevant background.

[0066] For each drone and satellite image, feature vectors (1×1024 dimensions) at layers 21 and 23 are extracted and calculated, and these vectors are saved as offline files to the feature database. This eliminates the need for cumbersome forward inference of large models during training; only lightweight feature vectors need to be loaded, greatly improving training efficiency.

[0067] Furthermore, in conjunction with reference Figure 4 For the feature vectors of satellite images within a batch, only a shared feature encoder is used for encoding, outputting 2048-dimensional satellite encoded features as a reference for geometric alignment; for the feature vectors of UAV images within a batch, the geometric correction module is input into the geometric correction module. Based on the encoded features obtained by shared linear projection, the geometric correction module corrects the nonlinear geometric distortion of UAV images caused by tilted shooting, outputting 2048-dimensional geometric correction feature vectors.

[0068] Specifically, the feature vector extracted from the 23rd intermediate layer is selected as input. The feature vector from the satellite image is then used as input. As a geometric reference, only a single shared feature encoder is used for encoding to maintain the stability of its distribution and obtain satellite-coded features. For the feature vector of a drone image The UAV coded features are obtained in the shared feature encoder. Building upon this foundation, a geometric correction module was constructed. This module utilizes one-dimensional convolution operations with different dilation rates to expand the receptive field without increasing the number of parameters. Small dilation rates focus on local detail fine-tuning, while large dilation rates capture global structural correction. This allows the module to capture multi-scale contextual information, from local detail fine-tuning to global rotation correction, within the one-dimensional sequence of feature vectors. This enables a more refined and accurate fit to real manifold distortions, significantly improving the matching success rate under complex flight attitudes.

[0069] Assume the feature vector of the input drone image is The coded features of the UAV after processing by the shared feature encoder are: The geometric correction module will The data is split into three parallel processing branches, each using a different expansion rate. One-dimensional convolutional layers are used to capture distortion features at different scales: Branch 1 (Local Perception) ): Focus on the nearest neighbor correlation between feature channels and correct for minor differences.

[0070] Branch 2 (Mid-range perception) ): Focus on feature dependencies with medium spans and correct for medium differences.

[0071] Branch 3 (Global Awareness) ): Focus on long-distance channel correlation and correct large-angle distortion differences.

[0072] The outputs of the three branches are processed by the GeLU activation function, then concatenated and fused. A 1×1 point convolutional layer is then used to map the dimensions back to the original space, and finally the result is superimposed onto the input as a residual. Above. Corrected drone features. The calculation formula is: ; in, Encoding features for drones, This is the GeLU activation function. These are the weight parameters for the fusion layer. For the expansion rate One-dimensional convolution operation, For feature correction. This multi-scale parallel residual design enables the model to adaptively fit complex nonlinear manifold distortions, outputting geometric correction features with a dimension of 2048, which include satellite-coded features and correction features.

[0073] Furthermore, in conjunction with reference Figure 7 , Figure 7 This invention illustrates the generation and distribution optimization of soft pseudo-labels based on the optimal transport algorithm alignment. Soft pseudo-labels are generated using the optimal transport algorithm, and unsupervised training is performed based on a distribution-aware optimization strategy, aiming to solve the model convergence problem in the absence of ground truth labels. To overcome the instability of training caused by traditional greedy clustering generating pseudo-labels, this invention utilizes the Sinkhorn algorithm from optimal transport theory to generate soft pseudo-labels satisfying double random constraints from a global perspective.

[0074] Specifically, extract drone images and satellite images from a batch respectively. One corrected feature and Satellite coding features In a training batch, containing One corrected feature and Satellite coding features First, calculate the cosine similarity matrix between them. And construct the transmission cost matrix The task objective is to find an optimal transmission scheme. (i.e., soft pseudo-labels) minimizes the total transmission cost, while introducing an entropy regularization term to smooth the distribution: ; in, It is the entropy regularization term. This is the regularization coefficient. This optimization problem is solved iteratively using the Sinkhorn-Knopp algorithm: ; in It uses a Gibbs core and employs dynamic adjustment. The course learning strategies, as the training progresses, The value gradually decreased from 0.2 to 0.05, making the soft pseudo-label... The smooth global distribution gradually converges into a sharp, deterministic match.

[0075] Then, based on cosine similarity and the generated soft pseudo-labels A distribution-aware mutual optimization strategy was designed, which includes the following loss function: A. Two-way soft distillation loss function: similarity distribution predicted using KL divergence constraints. To approximate the soft pseudo-tags generated by Sinkhorn Regarding the direction from the drone to the satellite: ; in, Soft fake tags The row normalized value, Cosine similarity matrix The distribution after Softmax is calculated similarly. The loss from the satellite to the UAV direction is calculated similarly.

[0076] B. Cross-entropy loss function: To consolidate high-confidence matches, only those that... Matching pairs where both the middle row and column are maximum values And the similarity is higher than the threshold. For the samples, calculate the cross-entropy loss function: ; in, The distribution of the similarity matrix from the UAV to the satellite direction after applying Softmax is the corresponding term. The distribution of the similarity matrix from satellite to UAV direction after applying Softmax represents the corresponding terms. In soft fake tags Matching pairs where both the middle row and column are maximum values Samples with similarity higher than the threshold.

[0077] C. Reconstruction Loss Function: To prevent semantic loss due to overfitting by the geometry correction module, a decoder is used to map features back to the original space, and the symmetric JS divergence is calculated as the reconstruction loss function. ; in, Original features The reconstructed features are calculated after the features are mapped back to the original space using a decoder.

[0078] The total loss function is determined based on the above loss function, as shown in the following formula: .

[0079] Furthermore, the loss value is determined based on the total loss function, and the parameters of the geometric correction module are optimized based on the loss value.

[0080] As described above, this invention utilizes the Sinkhorn algorithm to generate soft pseudo-labels that satisfy global double random constraints, and combines this with dynamic regularization coefficients for unsupervised training to optimize the parameters of the geometric correction module. This global perspective effectively avoids local extremum interference, achieving efficient cold start and stable convergence of the model without any ground truth supervision.

[0081] like Figure 8 As shown, Figure 8 This is a schematic diagram of the structure of an embodiment of the medium provided by the present invention. The medium 20 stores at least one computer program 21, which is executed by a processor to perform the following... Figure 1 The method shown is detailed above and will not be repeated here. In one embodiment, the medium 20 can be a storage chip, hard disk, portable hard disk, USB flash drive, optical disk, or other read / write storage device, or even a server, etc.

[0082] Furthermore, the processes depicted in the accompanying drawings do not necessarily have to be performed in the specific or sequential order shown to achieve the desired result. In some implementations, multitasking and parallel processing are possible or may be advantageous.

[0083] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, devices, and non-volatile computer-readable storage media are basically similar to the method embodiments, and therefore described more simply; relevant parts can be referred to the descriptions of the method embodiments.

[0084] The apparatus, device, non-volatile computer-readable storage medium and method provided in the embodiments of this specification are corresponding. Therefore, the apparatus, device and non-volatile computer storage medium also have similar beneficial technical effects as the corresponding method. Since the beneficial technical effects of the method have been described in detail above, the beneficial technical effects of the corresponding apparatus, device and non-volatile computer storage medium will not be repeated here.

[0085] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or any combination of these devices.

[0086] For ease of description, the above apparatus is described by dividing it into various functional units. Of course, in implementing this specification, the functions of each unit can be implemented in one or more software and / or hardware components. Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, the embodiments of this specification can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, the embodiments of this specification can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0087] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0088] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0089] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0090] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0091] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0092] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0093] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0094] This specification can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This specification can also be practiced in distributed computing environments, where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0095] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0096] The above description discloses only preferred embodiments of the present invention and should not be construed as limiting the scope of the present invention. Therefore, equivalent variations made in accordance with the claims of the present invention are still within the scope of the present invention.

Claims

1. An unsupervised cross-view scene matching and localization method based on a large visual model, characterized in that, The method includes: Acquire images of the drone to be located and a satellite image library, and use the current satellite images of the drone to be located and the satellite image library as input images; The input image is used to extract features using a pre-trained DINOv3 visual large model to obtain multiple intermediate layer feature maps, and generalized mean pooling is performed on each layer feature map to generate feature vectors. Perform geometric correction on the feature vector to obtain geometrically corrected features; The feature vectors of the UAV image and the current satellite image at a preset intermediate level are stitched together, and the stitched feature vectors are subjected to frequency domain transformation and filtering to obtain structural enhancement features; The geometric correction features and structural enhancement features are spliced ​​together to generate a global descriptor. The similarity between the global descriptors of the UAV image to be located and each image in the satellite image library is calculated. The location result of the UAV is determined based on the similarity ranking result.

2. The unsupervised cross-view scene matching and localization method based on a large visual model according to claim 1, characterized in that, The step of performing geometric correction on the feature vector to obtain geometrically corrected features specifically includes: The feature vectors corresponding to the satellite images are encoded to obtain satellite coded features; The feature vector corresponding to the UAV image is encoded to obtain UAV encoded features, and the UAV encoded features are corrected to obtain corrected features; The correction features and satellite coding features are used as geometric correction features.

3. The unsupervised cross-view scene matching and localization method based on a large visual model according to claim 2, characterized in that, The step of correcting the UAV coding features to obtain corrected features specifically includes: according to The UAV coding features are corrected to obtain corrected features, wherein, Encoding features for drones, This is the GeLU activation function. These are the weight parameters for the fusion layer. For the expansion rate One-dimensional convolution operation, To correct features.

4. The unsupervised cross-view scene matching and localization method based on a large visual model according to claim 1, characterized in that, The process of stitching together the feature vectors of the UAV image and satellite image at a preset intermediate level, performing frequency domain transformation and filtering on the stitched feature vectors to obtain structure-enhanced features, specifically includes: The feature vectors of UAV images and satellite images at a preset intermediate level are stitched together respectively; Perform a Fourier transform on the concatenated feature vectors to obtain the spectral representation, and decompose the spectral representation into an amplitude spectrum and a phase spectrum; The amplitude spectrum is filtered to obtain the corrected spectral representation; The modified spectral representation is subjected to inverse Fourier transform to obtain the structural enhancement features.

5. The unsupervised cross-view scene matching and localization method based on a large visual model according to claim 2 or 4, characterized in that, The process involves concatenating the geometric correction features and structural enhancement features to generate a global descriptor, calculating the cosine similarity between the global descriptors of the UAV image to be located and each image in the satellite image database, and determining the UAV's location result based on the ranking of the cosine similarity scores. Specifically, this includes: according to The corrected features and the structural enhancement features corresponding to the UAV image are stitched together to determine the first stitched feature, wherein, To correct features, Structural enhancement features for drone images, This is the first splicing feature; according to The satellite coding features and the structural enhancement features corresponding to the satellite image are spliced ​​together to determine the second splicing feature, wherein, For satellite coding features, To enhance the structural features of satellite images, This is the second splicing feature; according to Determine the features of the queried drone image, where, To query drone image features; according to Identify all satellite features in the satellite image database, among which, The feature of the i-th satellite in the satellite image database; according to Determine the cosine similarity between the features of the queried UAV image and all satellite features in the satellite image database, wherein, To query the cosine similarity between UAV image features and the i-th satellite feature in the satellite image database; The cosine similarity scores are sorted according to a preset order, and the satellite image with the highest cosine similarity score is the UAV's positioning result.

6. An unsupervised cross-view scene matching and localization system based on a large visual model, characterized in that, The system includes: The image acquisition module is used to acquire images of the drone to be located and a satellite image library, and to use the images of the drone to be located and the current satellite images in the satellite image library as input images; The feature extraction module is used to extract features from the input image using a pre-trained DINOv3 visual large model, obtain multiple intermediate layer feature maps, and perform generalized mean pooling on each layer feature map to generate feature vectors. The geometric correction module is used to perform geometric correction on the feature vector and obtain geometric correction features; The intrinsic structure mining module is used to stitch together the feature vectors of UAV images and current satellite images at a preset intermediate level, and then perform frequency domain transformation and filtering on the stitched feature vectors to obtain structure enhancement features. The UAV positioning module is used to stitch together the geometric correction features and structural enhancement features to generate a global descriptor, calculate the similarity between the global descriptors of the UAV image to be located and each image in the satellite image library, and determine the UAV positioning result based on the similarity ranking result.

7. The unsupervised cross-view scene matching and localization system based on a large visual model according to claim 6, characterized in that, The system also includes: The training module acquires UAV image datasets and satellite image datasets, selecting an equal number of UAV and satellite images as a training batch. Images from the same training batch are input into the feature extraction module, which outputs feature vectors. The feature vectors are then input into the geometric correction module, which outputs geometrically corrected features. The transmission cost matrix is ​​determined based on the cosine similarity of the geometrically corrected features. Soft pseudo-labels are generated from the transmission cost matrix using the Sinkhorn algorithm. A loss value is determined based on the cosine similarity and the soft pseudo-labels. The parameters of the geometric correction module are optimized based on the loss value.

8. The unsupervised cross-view scene matching and localization system based on a large visual model according to claim 7, characterized in that, The process of determining the transmission cost matrix based on the cosine similarity of the geometric correction features, generating soft pseudo-labels from the transmission cost matrix using the Sinkhorn algorithm, determining the loss value based on the cosine similarity and the soft pseudo-labels, and optimizing the parameters of the geometric correction module based on the loss value specifically includes: In a training batch, the cosine similarity between the UAV features and the satellite features is calculated to obtain a cosine similarity matrix; The transmission cost matrix is ​​determined based on the cosine similarity matrix. The optimal transmission scheme is taken as the task objective. An optimal transmission model is constructed based on the transmission cost matrix. The optimal transmission model is solved iteratively using the Sinkhorn algorithm to generate soft pseudo-labels. The transmission scheme is the pairing scheme of the UAV images and satellite images in a training batch. The total loss function is determined based on the cosine similarity matrix and the soft pseudo-labels. The loss value is determined based on the total loss function, and the parameters of the geometric correction module are optimized based on the loss value.

9. The unsupervised cross-view scene matching and localization system based on a large visual model according to claim 8, characterized in that, The determination of the total loss function based on the cosine similarity matrix and soft pseudo-labels specifically includes: according to Determine the loss function for two-way soft distillation, where, This is the loss function for two-way soft distillation. Cosine similarity matrix Distribution after Softmax Soft fake tags The row normalized value, Let B be the KL divergence and B be the training batch. according to Determine the cross-entropy loss function, where, Let cross-entropy be the loss function. The distribution of the similarity matrix from the UAV to the satellite direction after applying Softmax is the corresponding term. The distribution of the similarity matrix from satellite to UAV direction after applying Softmax represents the corresponding terms. In soft fake tags Matching pairs where both the middle row and column are maximum values Samples with similarity higher than the threshold; according to Determine the reconstruction loss function, where, To reconstruct the loss function, Original features The reconstructed features are calculated after the features are mapped back to the original space using a decoder. according to Determine the total loss function, where L is the total loss function.

10. A computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to perform the steps of the method as claimed in any one of claims 1 to 5.

Images