An unmanned system cross-view positioning method based on soft label distillation and multi-scale feature fusion

Abstract

The application discloses an unmanned system cross-view positioning method based on soft label distillation and multi-scale feature fusion. The method comprises the following steps: acquiring a ground view image and an aerial view image, inputting the images into a visual Transformer backbone network to extract multi-scale features composed of patch tokens and class tokens (CLS Token); based on attention weights, the patch tokens are windowed and weighted pooled to obtain local features, which are input into a local-global fusion module together with the global feature CLS Token, cross-scale feature interaction is realized through a Transformer structure to obtain fused multi-scale feature representation; a teacher model trained with a panoramic field of view is used to calculate the continuous visual similarity between the ground image and the aerial image, a soft label similarity matrix is formed, and bidirectional soft label distribution is obtained by performing Softmax normalization on the rows and columns respectively; bidirectional Kullback-Leibler divergence constraints are applied to the bidirectional similarity distribution and the soft label distribution output by the student model to form a soft label distillation loss, and the student model is jointly trained with the original hard label loss, so that the student model generates discriminative cross-view features under unknown orientation and limited field of view conditions; the trained model is used for cross-view image retrieval, geographical position matching results are obtained according to feature similarity ranking, and cross-view unmanned system visual positioning and geographical positioning are realized.

Description

Technical Field

[0001] This invention belongs to the fields of computer vision, image retrieval and geographic information processing technology, and specifically relates to a cross-view positioning method for unmanned systems based on soft label distillation and multi-scale feature fusion. Background Technology

[0002] Cross-view geolocation missions aim to locate images captured in satellite or aerial imagery databases using terrestrial imagery. This task is highly challenging due to the significant differences in perspective, scale, and appearance between terrestrial and aerial views.

[0003] The existing technology has the following main problems:

[0004] 1. Binary labels limit model capabilities

[0005] Traditional methods encode matching relationships as binary values ​​of 0 / 1, making it difficult to represent true visual similarity in a continuous space, resulting in inaccurate feature learning.

[0006] 2. Limited field of view leads to information loss.

[0007] In real-world applications, a 360° field of view is rare and too ideal. In narrow fields of view such as 180°, 90°, and 70°, image semantics are prematurely truncated, which greatly increases the difficulty of cross-view matching.

[0008] 3. Lack of unified local-global modeling

[0009] Transformers typically output only global CLS features, ignoring fine-grained structural information in local regions.

[0010] Therefore, a solution is needed that can simultaneously utilize multi-scale feature fusion and soft label distillation to improve the robustness of the model in real-world scenarios. Summary of the Invention

[0011] The present invention aims to solve the following problems: alleviate the problem that binary labels cannot express visual continuity, alleviate the problem of semantic loss caused by narrow field of view, improve the robustness of retrieval, and improve retrieval accuracy through local-global feature fusion.

[0012] To address the aforementioned technical problems, this invention discloses a cross-view localization method for unmanned systems that combines soft tag distillation and multi-scale feature fusion, comprising the following steps:

[0013] Step 1: Acquire ground view images and aerial view images, input them into the visual Transformer backbone network, and extract multi-scale features composed of Patch Tokens and CLS Tokens;

[0014] Step 2: Based on attention weights, the Patch Tokens are windowed and weighted pooled to obtain local features. These local features are then input together with the global CLS features into the local-global fusion module. Cross-scale feature interaction is achieved through the Transformer structure to obtain the fused multi-scale feature representation.

[0015] Step 3: Train the teacher model using panoramic field of view input, and use the trained teacher model to calculate the continuous visual similarity between any ground map-aerial map pair to form a soft label similarity matrix;

[0016] Step 4: Apply bidirectional Kullback-Leibler divergence constraints to the similarity distribution and soft label distribution of the student model output to form the soft label distillation loss;

[0017] Step 5: Train the student model by combining soft-label distillation loss with the original hard-label loss, enabling it to generate discriminative cross-view features under unknown orientation and limited field of view conditions;

[0018] Step 6: Use the trained model for cross-view image retrieval. First, generate cross-view feature vectors, then sort them according to similarity to obtain the geographic location matching results, thus achieving cross-view geolocation.

[0019] Furthermore, the window weighted pooling in step 2 includes:

[0020] All Patch Tokens generated by the backbone network are average pooled, and then importance weights are generated through a multilayer perceptron. These weights are then weighted within a fixed-size sliding window. Traditional feature extraction methods (such as direct global average pooling (GAP), global max pooling (GMP), or simple patch concatenation) often ignore the semantic differences in image content and treat invalid backgrounds such as the sky and ground equally with key landmarks. Furthermore, directly using all original Patch Tokens results in excessive computation and redundancy; direct global pooling completely loses the relative positional relationships between features. The window-weighted pooling proposed in this invention, by introducing an importance weight mechanism generated by a multilayer perceptron, can adaptively identify and significantly suppress low-value background noise, making feature extraction highly focused on discriminative landmark regions. Simultaneously, combined with a fixed-size sliding window strategy, this method effectively preserves the relative positions and topological structures between adjacent image patches when aggregating features, thus overcoming the spatial information loss problem caused by the "one-size-fits-all" approach of traditional global pooling. This generates local feature representations with stronger anti-interference capabilities and rich spatial structure.

[0021] Furthermore, the local-global fusion module described in step 2 includes two cascaded Transformer layers for modeling the cross-scale relationship between local and global features. Existing technologies typically involve simple concatenation, linear superposition, or independent processing of local and global features, resulting in semantic fragmentation and difficulty in achieving deep information complementarity. The local-global fusion module proposed in this invention constructs a cross-scale interaction mechanism by introducing two cascaded Transformer layers, utilizing a self-attention mechanism to achieve deep coupling and dynamic calibration between local details and global macro-semantics. This design not only overcomes the limitations of traditional shallow fusion in effectively integrating information but also effectively compensates for semantic deficiencies caused by narrow field of view, unknown orientation, or object occlusion in ground-view scenarios using global contextual information, thereby significantly enhancing the feature representation integrity and retrieval robustness of the model in complex cross-view scenarios.

[0022] Furthermore, InfoNCE is used as the loss function during the training of the teacher model in step 3.

[0023] Set temperature coefficient To control the discriminative power of the neural network model against negative samples, let the image features of the Anchor be... , No. Images from different perspectives constitute the sample. The positive samples that match the Anchor are Then the loss function for:

[0024]

[0025] in, Indicates the number of samples. Represented by the natural logarithm An exponential function with base 0. This represents similarity calculation. Compared to common methods in cross-view geolocation such as cross-entropy or triplet loss, this method can simultaneously utilize a large number of negative samples for global discriminative learning within a single batch, and also uses temperature coefficients... By flexibly adjusting the level of attention given to hard-to-bear samples, more compact and discriminative cross-view feature representations can be learned, making it more suitable for solving matching problems with significant differences between ground and aerial viewpoints.

[0026] Furthermore, the soft labels in step 3 are generated by the teacher model from the panoramic input. Let the feature extraction function of the teacher model be... The i-th ground image in the batch is The j-th aerial image is The cross-perspective similarity calculated by the teacher model is: ,but:.

[0027]

[0028] Set the temperature coefficient to The similarity matrix is ​​then subjected to Softmax normalization row by row to obtain the soft label distribution in the Ground→Aerial direction. Furthermore, the similarity matrix is ​​subjected to Softmax normalization column-wise to obtain the soft label distribution in the Aerial→Ground direction. Compared to hard-label supervision, soft-label distillation generated by the teacher model not only provides positive sample matching information but also preserves the similarity between samples. This enables the student model, which is constrained by the ground perspective, to learn a smoother distribution that better conforms to the cross-perspective semantic structure.

[0029] Furthermore, in step 4, the student model undergoes bidirectional distillation training under the soft-label supervision of the teacher model. Let the feature extraction function of the student model be... The cross-view similarity matrix calculated by the student model is: ,but:

[0030]

[0031] The student model similarity matrix is ​​obtained by performing Softmax normalization on both the row and column sides. and Kullback–Leibler divergence To measure the consistency between student and teacher distributions, a two-way soft-label distillation loss is constructed. :

[0032]

[0033] Among them, the The bidirectional distillation loss improves the robustness and discriminativeness of cross-view retrieval by simultaneously constraining the matching distribution in both the Ground→Aerial and Aerial→Ground directions. Furthermore, by simultaneously constraining the soft label distribution in both the ground-to-air and air-to-ground directions, bidirectional distillation enables the student model to establish a more consistent and complete similarity structure in the dual-view space. This not only enhances the symmetry and complementarity of feature representation but also effectively mitigates the semantic gap caused by viewpoint differences, thereby achieving more stable retrieval performance in complex geographical scenarios.

[0034] Furthermore, the visual Transformer backbone network is a DINOv2 pre-trained model, or other Transformer backbones with Patch Token output capability.

[0035] Furthermore, the training data for the student model includes ground view images with different fields of view, including 360°, 180°, 90°, and 70°. The teacher model uses panoramic ground images as training data and input to obtain cross-view similarity with strong semantic continuity.

[0036] Furthermore, the similarity calculation in step 6 uses the dot product or cosine similarity between feature vectors.

[0037] Furthermore, the final cross-view feature vector in step 6 is obtained by flattening and dimensionality reduction of the fused feature sequence.

[0038] Beneficial effects:

[0039] 1. Based on standard-to-general knowledge transfer, robust cross-view localization capabilities are significantly improved in real-world complex scenarios, especially those with limited field of view:

[0040] This invention employs a standard-to-conventional transfer framework for teacher and student models, utilizing more reliable cross-view similarity knowledge from standard ground images to guide student model learning. This enables the model to achieve stable retrieval and localization performance even in conventional scenarios where ground images have unknown orientations or limited fields of view, thereby significantly improving applicability and robustness in real-world environments.

[0041] 2. Based on importance discrimination, the hierarchical feature aggregation model has strong feature discrimination ability: The hierarchical feature aggregation model obtains local tokens through importance estimation and window-level weighted aggregation, and interacts and merges them with global tokens. This enables the model to highlight the local structural information that is more critical for cross-view positioning, overcomes the problem that simple global pooling is not sensitive to fine-grained clues, and thus improves the matching ability and retrieval accuracy in complex scenes.

[0042] 3. More stable and reliable training, low incidence of feature collapse, and low risk of performance degradation caused by distillation noise: This invention introduces both contrastive learning during teacher model training and knowledge distillation during student model training. By combining InfoNCE loss and bidirectional KL, it improves the accuracy of basic matching by improving feature quality, while effectively suppressing noise accumulation and error propagation caused by imperfect teacher predictions, thereby improving the stability of the training process and the reliability of the final model performance. Attached Figure Description

[0043] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments, and the advantages of the present invention in the above and / or other aspects will become clearer.

[0044] Figure 1 This is a schematic diagram of the overall process in this invention.

[0045] Figure 2 This is a schematic diagram of the multi-scale feature fusion network structure in this invention.

[0046] Figure 3 This is a schematic diagram of the comparative learning and knowledge distillation training process in this invention.

[0047] Figure 4 This is a schematic diagram of the process for generating soft tags using the teacher model in this invention. Detailed Implementation

[0048] This invention provides a feature learning and training method for cross-view geolocation, used for retrieval-based localization between ground-view images and aerial-view images, especially suitable for scenes with unknown orientation and limited field of view. It improves retrieval accuracy through multi-scale feature fusion and enhances retrieval robustness by using a teacher model with an ideal known view to guide the training of a student model with random limited view.

[0049] The following describes "A Cross-View Localization Method for Unmanned Systems Based on Soft Tag Distillation and Multi-Scale Feature Fusion" with reference to preferred embodiments of the present invention. It should be understood that the following embodiments are only for explaining the present invention. Those skilled in the art can make equivalent substitutions or modifications to the network structure, parameter settings, and implementation without departing from the concept of the present invention, and all such substitutions or modifications should fall within the protection scope of the present invention.

[0050] I. Symbols and Definitions (for ease of understanding, not as a limitation)

[0051] 1. The ground view image is denoted as Aerial view image denoted as Ground view images can be panoramic (e.g., 360°) or limited (e.g., 180°, 90°, 70°, etc.).

[0052] 2. The visual Transformer backbone network outputs a token sequence containing a global CLS Token and local Patch Tokens, where the CLS Token... Patch Tokens , H and W represent the height and width of the patch grid, and D represents the feature dimension.

[0053] 3. The feature extraction function for the teacher model is denoted as... The feature extraction function for the student model is denoted as

[0054] 4. The similarity calculation function is denoted as: , which can be the dot product or cosine similarity. This implementation uses the dot product as an example, but it is not a limitation.

[0055] A cross-view localization method for unmanned systems based on soft-tag distillation and multi-scale feature fusion, the method comprising the following steps:

[0056] Step 1: Data input and multi-scale feature extraction.

[0057] Ground view images and aerial view images are acquired and input into a visual Transformer backbone network, respectively, to extract multi-scale features composed of Patch Tokens and CLS Tokens. The visual Transformer backbone network is preferably a DINOv2 pre-trained model, or other Transformer backbone networks with Patch Token output capability.

[0058] In an alternative implementation, the first few layers of the backbone network can be frozen to improve training stability, and the normalized output of the last layer can be used as the input for subsequent modules.

[0059] In terms of constructing training data for student models, ground view images with different fields of view can be included, including but not limited to 360°, 180°, 90°, and 70°. In an optional implementation, the ground image can be randomly rotated and cropped to the target FoV to simulate unknown orientation and limited field of view conditions, and enhancement methods such as compression, color perturbation, blurring, discarding, sharpening, and synchronous rotation / flipping can be superimposed.

[0060] Step 2: Multi-scale feature fusion.

[0061] First, the Patch Tokens are windowed and weighted pooled to obtain local features. Specifically, this includes:

[0062] (1) Rearrange into feature maps Rearranged into feature maps according to spatial location .

[0063] (2) Generation of importance weights: An importance weight vector is generated for each patch using a multilayer perceptron (MLP). And mapped to (0,1) by the Sigmoid activation function. A preferred form is: ,in These are learnable parameters. Then... Rearranged into a weighted graph .

[0064] (3) Weighted pooling within a fixed sliding window: Weighted pooling is performed within a fixed-size sliding window, with the window step size denoted as . Output aggregated local feature map One optimal implementation is as follows:

[0065]

[0066] in For element-wise multiplication, For normalization operations, Indicates step size is Average pooling.

[0067] (4) Local-Global Fusion Module

[0068] The local features obtained by window-weighted pooling and the global CLS features are input together into the local-global fusion module. A Transformer structure is used to achieve cross-scale feature interaction, resulting in a fused multi-scale feature representation. The local-global fusion module preferably includes two cascaded Transformer layers to model the cross-scale relationship between local and global features, for example:

[0069]

[0070] in, This represents the flattened local token sequence.

[0071] (5) Final cross-view feature vector generation)

[0072] The final cross-view feature vector is obtained by flattening and dimensionality reduction of the fused feature sequence. Dimensionality reduction can be achieved through linear layers or PCA, and the dimensionality reduction dimension can be set according to storage and retrieval speed requirements; this invention does not impose any limitations on this.

[0073] Step 3: Teacher Model Training and Soft Tag Similarity Matrix Generation

[0074] (1) Teacher Model Training (InfoNCE)

[0075] The teacher model was trained using panoramic field-of-view input, with InfoNCE as the loss function and a temperature coefficient set. The neural network model's ability to distinguish negative samples is controlled. Let the image features of the Anchor be... , No. Samples from different perspectives The positive samples that match the Anchor are The sample size is Similarity is calculated as Then the loss function is:

[0076] ,

[0077] in, Represented by the natural logarithm An exponential function with base 0.

[0078] (2) Generate soft tag similarity matrix and soft tag distribution

[0079] Soft labels are generated by the teacher model from the panoramic input. Let the feature extraction function of the teacher model be... The first batch One ground image is , No. One aerial image is The cross-perspective similarity calculated by the teacher model is:

[0080]

[0081] This forms a soft tag similarity matrix. .

[0082] Set temperature coefficient The similarity matrix is ​​then subjected to Softmax normalization row by row to obtain the Ground. Soft label distribution in the Aerial direction :

[0083]

[0084] Furthermore, the similarity matrix is ​​subjected to Softmax normalization column-by-column to obtain Aerial. Soft label distribution in the ground direction :

[0085]

[0086] Step 4: Two-way KL soft-label distillation loss

[0087] The student model is trained using bidirectional distillation under the soft-label supervision of the teacher model. Let the feature extraction function of the student model be... The cross-view similarity matrix calculated by the student model is: ,but:

[0088]

[0089] The student model similarity matrix is ​​obtained by performing Softmax normalization on both the row and column sides. and :

[0090]

[0091]

[0092] Kullback–Leibler divergence To measure the consistency between student and teacher distributions, a two-way soft-label distillation loss is constructed. :

[0093]

[0094] Step 5: Train the student model using a combination of soft-label distillation loss and hard-label loss.

[0095] The student model is trained by combining soft-label distillation loss with the original hard-label loss, enabling it to generate discriminative cross-view features under conditions of unknown orientation and limited field of view.

[0096] In a preferred implementation, the hard labels use an identity matrix. ,in:

[0097]

[0098] Hard-label loss can be implemented using a two-way KL constraint:

[0099]

[0100] The total loss is the weighted sum of the soft-label distillation loss and the hard-label loss:

[0101]

[0102] Step 6: Achieve geolocation through cross-view image retrieval

[0103] The trained model is used for cross-view image retrieval: first, cross-view feature vectors of query ground images are generated, and cross-view feature vectors of aerial image databases are generated; the similarity between query features and database features is calculated and sorted using dot product or cosine similarity, and the geographic location matching results are obtained, thereby achieving cross-view geolocation.

[0104] As a further improvement of the present invention, in step 3, the temperature coefficient can be adjusted. With weighting coefficients Adaptive or phased adjustments can be made to improve training stability and generalization performance. For example, a larger adjustment can be used in the early stages of training. To achieve a smoother distribution, the distribution is then gradually reduced. To enhance distinctiveness; and can The labels are gradually increased from small to large to rely more on continuous supervision with soft labels in the early stages of training and to strengthen constraints with hard labels in the later stages. This improvement is not limited to a specific scheduling strategy; linear scheduling, piecewise scheduling, or exponential scheduling are all acceptable.

[0105] As a further improvement to this invention, the local-global fusion module in step 2 can introduce residual enhancement, layer normalization, and / or feature gating mechanisms between or within two cascaded Transformer layers to improve gradient propagation and convergence speed. For example, it can be used to adjust the gradient propagation and convergence speed before and after fusion. and Perform L2 normalization, or perform L2 normalization on the fused output. Normalization is performed before flattening and dimensionality reduction to improve the consistency and stability of similarity calculation.

[0106] As a further improvement of the present invention, in step 2, the sliding window can adopt a multi-scale window combination, that is, setting multiple different window sizes and / or step sizes. Multiple local aggregated features are calculated separately and then stitched or weighted and fused to simultaneously capture coarse-grained structural information and fine-grained local details. This multi-scale window combination can improve the adaptability to scenes with different urban block scales and building densities.

[0107] As a further improvement of the present invention, in step 3, a similarity matrix can be constructed. and The process incorporates a hard sample mining strategy, prioritizing the sampling of highly similar but unmatched negative samples within a batch, or maintaining a dynamic queue between batches to expand the negative sample set, thereby enhancing the ability to distinguish hard negative samples. This improvement is compatible with InfoNCE training and soft-label distillation training.

[0108] As a further improvement to this invention, in step 5, the distillation intensity can be adjusted based on the soft-label confidence level of the sample pairs in the teacher model: when the teacher output distribution is too flat or uncertain, the intensity of the distillation for the corresponding sample pairs is reduced. The weights in the distillation process are adjusted; when the teacher output distribution is sharper and more consistent, the distillation weights are increased. This mechanism can reduce the adverse effects of teacher error on student training and improve overall robustness.

[0109] Example:

[0110] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0111] Example 1: As Figure 1 The method described above, a cross-view localization method for unmanned systems based on soft tag distillation and multi-scale feature fusion, includes the following steps:

[0112] Step 1: Data Acquisition and Processing.

[0113] Step 2: Multi-scale feature extraction.

[0114] Step 3: Multi-scale feature fusion.

[0115] Step 4: Teacher model training and soft label generation based on the teacher model.

[0116] Step 5: Soft label generation and student model training.

[0117] Step 6: Cross-perspective retrieval and geolocation.

[0118] The specific content of each stage is described below.

[0119] (1) Data acquisition and processing stage

[0120] First, the ground view image and aerial view image to be processed are acquired. During the training phase, in order to ensure the uniformity of feature map size, the aerial and ground images are simultaneously converted into 448*448 images. At the same time, image enhancement algorithms are applied to the ground and aerial images, including color perturbation, blurring, discarding, sharpening, and random ground / aerial synchronous rotation / flipping.

[0121] To simulate the limited field of view of unmanned systems in real-world scenarios, the input data for the teacher model has a field of view of 360°, while the input data for the student model includes ground view images with different field of view ranges, including but not limited to 360°, 180°, 90°, and 70°. Aerial images are typically orthophoto images. (2) Multi-scale feature extraction

[0122] The acquired images are input into the Visual Transformer backbone network. In this embodiment, the backbone network preferably uses the DINOv2 pre-trained model. The same backbone network is used for both the ground image processing model and the aerial image processing model. After the images are processed by the backbone network, multi-scale features containing two parts are extracted. Patch Tokens contain feature sequences of fine-grained local image information; CLS Tokens contain feature vectors of global semantic information of the image.

[0123] (3) Multi-scale feature fusion

[0124] This step aims to address the problem of traditional methods utilizing only global features while ignoring local structural information. For example... Figure 2 As shown, the specific implementation of this step includes two sub-processes: "window weighted pooling" and "local-global fusion".

[0125] The window-weighted pooling process involves first processing the extracted Patch Tokens using a Multilayer Perceptron (MLP) to generate importance weights for each Patch, and then mapping these weights to the (0,1) interval using a Sigmoid function. Subsequently, within a fixed-size sliding window, the Patch Tokens are weighted and averaged according to the generated weights. Specifically, a given number of Patch Tokens are first averaged and then weighted and aggregated within the window. This operation effectively suppresses dynamic disturbances such as moving vehicles and pedestrians while highlighting the local features of key landmarks such as buildings.

[0126] After completing window weighted pooling, local-global fusion is performed: the local features obtained through weighted pooling are concatenated with the global CLS features extracted in step 1, and then input together into the "local-global fusion module". For example... Figure 2 As shown in the structure, this module contains two cascaded Transformer layers. Through the Transformer's self-attention mechanism, cross-scale interaction between local and global features is achieved: global features are supplemented with lost details, while local features gain constraints from the global context. Finally, the fused feature sequence is flattened and its dimensionality reduced to obtain a high-quality cross-view feature representation for subsequent computation.

[0127] (4) Teacher model training and soft label generation based on teacher model

[0128] To address the issue that binary (0 / 1) hard labels cannot represent continuous visual similarity between images, this step introduces a teacher model to generate soft labels. For example... Figure 4 As shown, the specific process is as follows:

[0129] The first step is teacher model construction and training. A panoramic (360°) ground image is used as input to train the teacher model. InfoNCE is used as the loss function during training, and the learning rate is set to [value missing]. A temperature coefficient τ (in this example, set as a learnable parameter) is set to control the model's ability to distinguish negative samples. The InfoNCE loss function enables the teacher model to have a strong feature discrimination ability for panoramic ground images by bringing positive sample pairs (matched ground-aerial images) closer and pushing negative sample pairs further away.

[0130] After training, the soft-label similarity matrix is ​​calculated: using the trained teacher model, the feature similarity (usually dot product or cosine similarity) between any ground image (panoramic) and aerial image within a batch is calculated, forming a continuous numerical similarity matrix. Let the feature extraction function of the teacher model be... The similarity is .

[0131] Finally, a bidirectional soft-label distribution is generated: to construct soft labels in the form of a probability distribution, the similarity matrix above is subjected to Softmax normalization by row and column respectively: Ground Aerial direction: Normalize by row to obtain the soft label distribution for retrieving aeronautical charts from ground maps; Aerial Ground direction: Normalize by column to obtain the soft label distribution of ground map for aeronautical chart retrieval.

[0132] This two-way soft label not only indicates the correct matching object, but also implies knowledge of "which non-matching images are visually highly similar", providing richer supervision signals for the student model.

[0133] (5) Soft label generation and student model training.

[0134] These two steps constitute the core of training the student model, such as Figure 3 As shown, the training process combines teacher-model contrastive learning with teacher-student knowledge distillation.

[0135] Figure 3 The student model on the right receives a limited field of view (such as a randomly cropped 90° image) as input and outputs its predicted similarity matrix. Similarly, the similarity matrix of the student model is subjected to Softmax normalization by row and column to obtain the bidirectional prediction distribution of the student model. The Kullback–Leibler divergence between the soft label distribution of the teacher model and the prediction distribution of the student model is calculated to construct the bidirectional soft label distillation loss. This loss forces the student model to approximate the discriminative distribution of the teacher model in a panoramic view, even with incomplete input information (limited field of view), thereby learning robust features under unknown orientations.

[0136] The soft-label distillation loss described above is combined with the original hard-label loss (0 / 1 labels based on Ground Truth) to jointly train the student model. The hard-label loss ensures basic retrieval accuracy, while the soft-label distillation loss improves the model's generalization ability to complex scenes. Ultimately, the student model is able to generate highly discriminative cross-viewpoint features under conditions of unknown orientation and limited field of view.

[0137] (6) Cross-perspective retrieval and geolocation

[0138] Deploy the trained teacher and student models for real-world cross-view geolocation tasks:

[0139] First, feature generation is performed by inputting the queried ground images and aerial images from the database into the model. After processing in steps 1 and 2, the final cross-view feature vector is generated.

[0140] Then, similarity ranking is performed, and the distance between the query image and the library images is calculated using the dot product or cosine similarity between feature vectors.

[0141] Finally, the location is determined by sorting the similarity scores from high to low and retrieving the most matching aerial image. The geographic coordinates of this aerial image are the geographic location result of the ground image.

[0142] In its specific implementation, this application provides a computer storage medium and a corresponding data processing unit. The computer storage medium is capable of storing a computer program, which, when executed by the data processing unit, can run the invention's content regarding a cross-view localization method for unmanned systems based on soft-tag distillation and multi-scale feature fusion, as well as some or all of the steps in various embodiments. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0143] Those skilled in the art will clearly understand that the technical solutions in the embodiments of the present invention can be implemented using computer programs and their corresponding general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of computer programs, i.e., software products. These computer program software products can be stored in a storage medium and include several instructions to cause a device containing a data processing unit (which may be a personal computer, server, microcontroller, MUU, or network device, etc.) to execute the methods described in various embodiments or certain parts of the embodiments of the present invention.

[0144] This invention provides a concept and method for cross-view localization of unmanned systems based on soft tag distillation and multi-scale feature fusion. Many methods and approaches exist for implementing this technical solution; the above description is merely a preferred embodiment of the invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention. All components not explicitly stated in this embodiment can be implemented using existing technologies.

Claims

1. A cross-view localization method for unmanned systems based on soft-tag distillation and multi-scale feature fusion, characterized in that, Includes the following steps: (1) Obtain ground view images and aerial view images, input them into the visual Transformer backbone network, and extract multi-scale features composed of image patch tokens and class tokens; (2) Based on attention weights, the Patch Tokens are divided into windows and weighted pooled to obtain local features. These features are then input into the local-global fusion module along with the global CLS features. Cross-scale feature interaction is achieved through the Transformer structure to obtain the fused multi-scale feature representation. (3) Use panoramic field of view input to train the teacher model, and use the trained teacher model to calculate the continuous visual similarity between any ground map-aerial map pair to form a soft label similarity matrix; (4) Apply bidirectional Kullback-Leibler divergence constraints to the similarity distribution and soft label distribution of the student model output to form soft label distillation loss; (5) Combine soft label distillation loss with original hard label loss to train student model so that it can generate discriminative cross-view features under unknown orientation and limited field of view conditions; (6) The trained model is used for cross-view image retrieval. First, cross-view feature vectors are generated, and the geographical location matching results are obtained by sorting according to similarity, so as to realize cross-view geolocation.

2. The cross-view localization method for unmanned systems based on soft tag distillation and multi-scale feature fusion according to claim 1, characterized in that, The window weighted pooling in step (2) includes: A given number of Patch Tokens are pooled on an average basis, and then importance weights are generated through a multilayer perceptron. The weights are then weighted within a sliding window of a fixed size.

3. The cross-view localization method for unmanned systems based on soft tag distillation and multi-scale feature fusion according to claim 1, characterized in that, The local-global fusion module in step (2) includes two cascaded Transformer layers for modeling cross-scale relationships between local and global features.

4. The cross-view localization method for unmanned systems based on soft tag distillation and multi-scale feature fusion according to claim 1, characterized in that, In step (3), the teacher model is trained using InfoNCE as the loss function. Set temperature coefficient To control the discriminative power of the neural network model against negative samples, let the image features of the anchor sample be... , No. Images from different perspectives constitute the sample. The positive samples that match the Anchor are Then the loss function for: in, Indicates the number of samples. Represented by the natural logarithm An exponential function with base 0. This indicates the calculation of cosine similarity.

5. The cross-view localization method for unmanned systems based on soft tag distillation and multi-scale feature fusion according to claim 1, characterized in that, The soft labels in step (3) are generated by the teacher model from the panoramic input. Let the feature extraction function of the teacher model be... The i-th ground image in the batch is The j-th aerial image is The cross-perspective similarity calculated by the teacher model is: ,but: Set the temperature coefficient to The similarity matrix is ​​then subjected to Softmax normalization row by row to obtain the soft label distribution in the Ground→Aerial direction. Softmax normalization is performed on the similarity matrix column by column to obtain the soft label distribution in the Aerial→Ground direction. .

6. The cross-view localization method for unmanned systems based on soft tag distillation and multi-scale feature fusion according to claim 1, characterized in that, In step (4), the student model undergoes bidirectional distillation training under the supervision of the teacher model's soft labels. Let the feature extraction function of the student model be... The cross-view similarity matrix calculated by the student model is: ,but: The student model similarity matrix is ​​obtained by performing Softmax normalization on both the row and column sides. and Kullback–Leibler divergence To measure the consistency between student and teacher distributions, a two-way soft-label distillation loss is constructed. : Among them, the , and Representing the probability distributions of students and teachers respectively, bidirectional distillation loss improves the robustness and discriminativeness of cross-perspective retrieval by simultaneously constraining the matching distributions in both the Ground→Aerial and Aerial→Ground directions.

7. The cross-view localization method for unmanned systems based on soft tag distillation and multi-scale feature fusion according to claim 1, characterized in that, The visual Transformer backbone network is a DINOv2 pre-trained model, or other Transformer backbones with Patch Token output capability.

8. The cross-view localization method for unmanned systems based on soft tag distillation and multi-scale feature fusion according to claim 1, characterized in that, The training data for the student model includes ground view images with different fields of view, including 360°, 180°, 90°, and 70°. The teacher model uses panoramic ground images as training data and input to obtain cross-view similarity with strong semantic continuity.

9. The cross-view localization method for unmanned systems based on soft tag distillation and multi-scale feature fusion according to claim 1, characterized in that, The similarity calculation in step (6) uses the dot product or cosine similarity between feature vectors.

10. The cross-view localization method for unmanned systems based on soft tag distillation and multi-scale feature fusion according to claim 1, characterized in that, The final cross-view feature vector is obtained by flattening and dimensionality reduction of the fused feature sequence.

Images