Cross-view-angle image scene matching method based on rotation robustness and knowledge distillation
By using twin networks, pre-trained EVA-02 backbone networks, and knowledge distillation techniques, the difficulties in deploying cross-view image matching on UAV devices and the problem of rotation sensitivity were solved, thereby improving matching accuracy and efficiency.
Patent Information
- Application Number
- CN202511380410.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-09-25
- Publication Date
- 2026-02-17
AI Technical Summary
Existing cross-view image matching technologies are difficult to deploy on UAV devices, are highly sensitive to rotation, have underutilized feature extraction potential, and involve large model parameters and high computational demands.
By employing a Siamese network structure, a pre-trained EVA-02 backbone network, a feature rotation encoding strategy, and knowledge distillation techniques, rotation robustness and matching accuracy are improved through preprocessing, feature extraction, rotation encoding, and model compression.
It achieves efficient and accurate cross-view image matching on UAV equipment, improves rotation robustness and feature extraction capabilities, and reduces the number of model parameters.
Smart Images
Figure CN121544918A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of computer vision and deep learning, and particularly relates to a cross-view image scene matching method based on rotation robustness and knowledge distillation. BACKGROUND
[0002] The purpose of cross-view image matching is to match images from different views of the same target scene, so as to infer the location of the place. In recent years, with the rapid development of unmanned aerial vehicle technology, how to help unmanned aerial vehicles to locate and navigate through cross-view matching technology has become a research hotspot. The common cross-view matching process is as follows: a picture to be searched is given in a certain view of the search data set, and the matching system can find the picture under the same place in the candidate data set of another view. For cross-view matching of satellite and unmanned aerial vehicle platforms, two tasks need to be solved: unmanned aerial vehicle positioning and unmanned aerial vehicle navigation. Unmanned aerial vehicle positioning: given an image in the view of the unmanned aerial vehicle, find the most matched satellite view image; unmanned aerial vehicle navigation: given an image in the view of the satellite, find the most matched unmanned aerial vehicle image. Therefore, the core of the cross-view matching technology is to learn the invariant features of images under different views through training of a deep learning model, so as to improve the accuracy and robustness of the matching system.
[0003] Cross-view matching algorithm has experienced a development process from the initial extraction of point feature descriptors of images to the extraction of global features of images through deep convolutional networks. The vector of locally aggregated descriptors (VLAD) is a classic image retrieval algorithm and can also be applied to the field of cross-view scene matching. VLAD constructs a vector by aggregating point local descriptors, and takes the vector as a global descriptor of the image. Then, the image is optimally matched according to the feature vector. Arandjelovic proposed NetVLAD in 2016, which improved VLAD by using a neural network. In NetVLAD, the local features described by the hand-craft descriptor in VLAD are replaced by global features extracted by a convolutional neural network, and the hard-assignment part in VLAD which is not differentiable is replaced by a differentiable softmax function, so that NetVLAD can optimize the parameters through training.
[0004] Thanks to the foundation laid by deep convolutional neural networks in recent years, deep learning-based models have been widely used to extract image features. Workman's research in 2015 found that features extracted by deep convolutional networks have the characteristics of high discriminability and easy computation. Experimental results showed that they performed better than point features on the Place dataset, pointing the way for subsequent cross-view matching research. With the release of the University1652 dataset, cross-view matching algorithms based on UAV and satellite views have also made some progress in the past year. In 2021, Ding proposed a cross-view matching method based on location classification (LCM). During the training phase, LCM treats the image retrieval problem as a classification problem and considers the impact of different feature sizes on the final result. The results show that LCM improves the performance by 5-10% compared to the baseline model of University1652.
[0005] In 2021, Wang proposed the Local Pattern Network (LPN) deep neural network, which considers the contextual information of neighboring regions. It employs a square ring feature segmentation strategy and enhances attention based on the distance to the image center. This strategy exhibits good robustness to rotational changes, significantly improving matching efficiency. In 2021, Tian proposed a method that comprehensively considers the spatial correspondence between satellite views and surrounding regional information. This method comprises two parts: converting the drone's tilted view into a vertical view through perspective transformation; and using a conditional generative adversarial network to make the drone's perspective image closer to the satellite image. Experimental results show that this method improves upon LPN by 5%. In 2021, Zhuang, to eliminate the differences between images acquired from different viewpoints, referenced the idea of multi-scale block attention (MSBA) and extracted features from different views through multi-branch results to improve feature extraction efficiency.
[0006] However, there are still many defects and shortcomings in the existing technology, including:
[0007] 1. Large number of model parameters: As models become increasingly complex, the number of parameters and computational load also increase, making it extremely challenging to deploy scene matching models on edge computing devices such as drones;
[0008] 2. Rotation sensitivity: When drones capture images, their posture changes frequently, and their shooting angle often differs from the fixed angle of the satellite image. This difference in rotation angle can seriously affect the accuracy and efficiency of matching.
[0009] 3. Feature extraction potential not fully explored: Although Transformer-based backbone networks have shown great potential, they require large-scale data for training and fine-tuning. How to fully utilize modern backbone networks pre-trained on large datasets to improve scene matching performance is a pressing issue that needs to be addressed.
[0010] Therefore, through beneficial exploration and research, the applicant has found a solution to the above problems, and the technical solution to be introduced below is the result of this research. Summary of the Invention
[0011] The technical problem to be solved by this invention is to provide a cross-view image scene matching method based on rotation robustness and knowledge distillation to address the shortcomings of the prior art. The aim is to improve the accuracy and rotation robustness of cross-view image matching, and to enable its efficient deployment on edge devices such as drones through model compression technology.
[0012] The technical problem to be solved by this invention can be achieved by the following technical solution:
[0013] A cross-view image scene matching method based on rotation robustness and knowledge distillation includes the following steps:
[0014] Step S10: Establish a scene matching model based on a twin network structure. This model contains two weight-sharing branches, which are used to process satellite view images and UAV view images, respectively.
[0015] Step S20: Preprocess the input image using the preprocessing layer of the scene matching model;
[0016] Step S30: Use a modern visual model EVA-02 that has been fully pre-trained on a large dataset as the backbone network for feature extraction;
[0017] Step S40: Introduce a feature rotation coding strategy to eliminate the negative impact of image rotation;
[0018] Step S50: Apply knowledge distillation technique to compress the model to achieve efficient reasoning;
[0019] In step S60, during the inference matching stage, the similarity between the feature vector of the query image and the feature vector of each image in the geographic reference image library is calculated using metrics such as cosine distance, so as to achieve the final scene matching.
[0020] In a preferred embodiment of the present invention, in step S10, the scene matching model includes a preprocessing layer, a feature extractor, and a matching layer. The feature extractor uses a pre-trained EVA-02 model as the backbone network for feature extraction.
[0021] In a preferred embodiment of the present invention, in step S20, the input image is preprocessed through the preprocessing layer of the scene matching model, and the specific process is as follows:
[0022] The image size is standardized, and image enhancement operations such as random slicing and random horizontal flipping are applied. The random slicing operation involves cropping the image at random locations according to a set size, while the random horizontal flipping operation involves horizontally flipping the image with a given probability.
[0023] In a preferred embodiment of the present invention, in step S30, the preprocessed image is input into the pre-trained EVA-02 model feature extractor for feature extraction. The specific process is as follows:
[0024] During the training phase, the vector output by the feature extractor is mapped to the number of target scenes to be classified through the neurons of the fully connected layer in the classification layer, and the output value is normalized through the softmax layer. The weight parameters obtained by the scene matching model during training are optimized based on the joint metric loss function of the cross-entropy loss function (CE) and the loss function infoNCE.
[0025] The formula for calculating the cross-entropy loss function is as follows:
[0026]
[0027] in, y represents the predicted value from the scene matching model, and y represents the label value. Indicates predicted value The predicted probability of being identified as class y. Indicates the column direction of the label. The column vector represents the sample, where i represents the index of the total number of scenes, j represents the j-th vector output by the scene matching model, c represents the c-th sample in this round of training, and C represents the number of training samples.
[0028] The formula for calculating the infoNCE loss function is as follows:
[0029]
[0030] Wherein, given a positive sample pair and a set of negative samples τ represents the sharpness coefficient used to control the distribution of the data, sim(q,k) represents the distance between samples, N represents the batch size, K represents the total number of candidate samples (including positive samples and all other negative samples), and q i This represents the i-th query vector. Let k represent the i-th corresponding positive sample. j Let j represent the j-th candidate sample (which may include both positive and negative samples).
[0031] In a preferred embodiment of the present invention, in step S40, a feature rotation encoding strategy is introduced to eliminate the negative impact of image rotation on matching. The specific process is as follows:
[0032] During the inference (testing) phase, to enhance the model's robustness to rotational changes, a feature rotation encoding strategy is adopted. A single input image is transformed to generate two transformed images. The original image and the transformed image are then fed into the pre-trained model to extract their respective feature vectors. The resulting multiple sets of feature vectors are superimposed or averaged to fuse them into a final, more robust feature representation that contains rotational information.
[0033] In a preferred embodiment of the present invention, in step S50, knowledge distillation technology is applied to compress the model to achieve efficient reasoning, and the specific process is as follows:
[0034] Use a well-trained, high-performance model with a large number of parameters as the teacher model and freeze its parameters;
[0035] A lightweight network model is selected as the student model. The KL divergence loss is used to guide the student model to learn the output distribution of the teacher model. At the same time, the student model also learns through its own loss function (cross-entropy + infoNCE loss). This approach significantly reduces the number of parameters while maintaining high matching accuracy.
[0036] The formula for calculating the KL divergence loss is as follows:
[0037]
[0038] in, Let represent the output probability of the i-th class in teacher model t (temperature T). Let T represent the output probability of the i-th class of student model s, where T>0, and let T be the temperature coefficient.
[0039] In a preferred embodiment of the present invention, in step S60, during the inference matching stage, the similarity between the feature vector of the query image and the feature vectors of each image in the geographic reference image database is calculated using metrics such as cosine distance, so as to achieve the final scene matching. The specific process is as follows:
[0040] The feature vectors of the obtained satellite images and the feature vectors of the UAV images are measured by cosine similarity. The feature vector pair with the closest distance is selected as the match. When performing UAV positioning tasks, the distance between a UAV image and all satellite images is calculated, and the vector pair with the closest distance is selected as the match pair. When performing UAV navigation tasks, the distance between a satellite image and all UAV images is calculated, and the vector pair with the closest distance is selected as the match pair.
[0041] The formula for calculating the cosine similarity is:
[0042]
[0043] Where CS represents cosine similarity, f A f represents the eigenvector matrix composed of all UAV images. B f represents the eigenvector matrix composed of all satellite images. Ai f represents the feature vector of the i-th drone image, i.e., the i-th scene. Bi This represents the feature vector of the i-th UAV map. Let L represent the L2 norm of the vector, and N represent the total number of scenarios.
[0044] Due to the adoption of the above technical solution, the beneficial effects of the present invention are as follows:
[0045] 1. This invention uses the EVA-02 model pre-trained on a massive dataset as the backbone network. This network itself has a powerful visual feature representation capability and can accurately extract key information in the scene without designing a complex attention module, which significantly improves the accuracy of cross-view matching.
[0046] 2. By using the feature rotation encoding strategy proposed in this invention, image features from multiple rotation angles are fused, so that the final features generated by the model have rotation invariance, effectively overcoming the matching difficulties between UAV and satellite images caused by different shooting angles;
[0047] 3. This invention employs knowledge distillation technology to successfully transfer knowledge from a large teacher model to a lightweight student model. This allows the model to achieve near-large-scale performance with a significantly reduced number of parameters, laying the foundation for real-time scene matching on edge devices with limited computing resources, such as drones. Attached Figure Description
[0048] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0049] Figure 1 This is a flowchart illustrating the present invention.
[0050] Figure 2 This is a schematic diagram of the scene matching model of the present invention.
[0051] Figure 3 The schematic diagram of the feature rotation encoding strategy of the present invention illustrates the process of enhancing feature representation by extracting features from images at different angles and accumulating them.
[0052] Figure 4 The schematic diagram of the invention's knowledge distillation framework illustrates how the teacher model guides the student model's learning through KL loss. Detailed Implementation
[0053] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below with reference to specific illustrations.
[0054] See Figure 1 This invention proposes a cross-view image scene matching method based on rotation robustness and knowledge distillation, comprising the following steps:
[0055] Step S10: Establish a scene matching model based on a twin network structure. This model contains two weight-sharing branches, which are used to process satellite view images and UAV view images, respectively.
[0056] Step S20: Preprocess the input image using the preprocessing layer of the scene matching model;
[0057] Step S30: Use a modern visual model EVA-02 that has been fully pre-trained on a large dataset as the backbone network for feature extraction;
[0058] Step S40: Introduce a feature rotation coding strategy to eliminate the negative impact of image rotation;
[0059] Step S50: Apply knowledge distillation technique to compress the model to achieve efficient reasoning;
[0060] In step S60, during the inference matching stage, the similarity between the feature vector of the query image and the feature vector of each image in the geographic reference image library is calculated using metrics such as cosine distance, so as to achieve the final scene matching.
[0061] In step S10, the scene matching model includes a preprocessing layer, a feature extractor, and a matching layer. The feature extractor uses the pre-trained modern visual model EVA-02 as the backbone network for feature extraction.
[0062] In step S20, the input image is preprocessed using the preprocessing layer of the scene matching model. The specific process is as follows:
[0063] See Figure 2 The image size is standardized, and image enhancement operations such as random slicing and random horizontal flipping are applied. The random slicing operation involves cropping the image at random positions according to a set size, and the random horizontal flipping operation involves horizontally flipping the image with a given probability.
[0064] In step S30, the preprocessed image is input into the pre-trained EVA-02 feature extractor for feature extraction. The specific process is as follows:
[0065] During the training phase, the vector output by the feature extractor is mapped to the number of target scenes to be classified through the neurons of the fully connected layer in the classification layer, and the output value is normalized through the softmax layer. The weight parameters obtained by the scene matching model during training are optimized based on the joint metric loss function of the cross-entropy loss function (CE) and the loss function infoNCE.
[0066] The formula for calculating the cross-entropy loss function is:
[0067]
[0068] in, y represents the predicted value from the scene matching model, and y represents the label value. Indicates predicted value The predicted probability of being identified as class y. Indicates the column direction of the label. The column vector represents the sample, where i represents the index of the total number of scenes, j represents the j-th vector output by the scene matching model, c represents the c-th sample in this round of training, and C represents the number of training samples.
[0069] The formula for calculating the infoNCE loss function is:
[0070]
[0071] Wherein, given a positive sample pair and a set of negative samples τ represents the sharpness coefficient used to control the distribution of the data, sim(q,k) represents the distance between samples, N represents the batch size, K represents the total number of candidate samples (including positive samples and all other negative samples), and q i This represents the i-th query vector. Let k represent the i-th corresponding positive sample. j Let j represent the j-th candidate sample (which may include both positive and negative samples).
[0072] In step S40, a feature rotation encoding strategy is introduced to eliminate the negative impact of image rotation on matching. The specific process is as follows:
[0073] See Figure 3 In the inference (testing) phase, to enhance the model's robustness to rotational changes, a feature rotation encoding strategy is adopted. A single input image is transformed, for example, by performing a "horizontal flip" and a "90-degree rotation" operation to generate two transformed images. The original image and the transformed image are then fed into the pre-trained model to extract their respective feature vectors. The resulting multiple sets of feature vectors are superimposed or averaged to fuse them into a final, more robust feature representation that contains rotational information.
[0074] In step S50, knowledge distillation technique is applied to compress the model to achieve efficient reasoning. The specific process is as follows:
[0075] See Figure 4 A well-trained, high-performance but parameter-intensive model is used as the teacher model, and its parameters are frozen. A lightweight network model, such as MobileViTv2, is selected as the student model. The student model learns the output distribution of the teacher model using KL divergence loss (KLLoss), while also learning using its own loss function (cross-entropy + infoNCE loss). This approach significantly reduces the number of parameters while maintaining high matching accuracy.
[0076] The formula for calculating the KL divergence loss is:
[0077]
[0078] in, Let represent the output probability of the i-th class in teacher model t (temperature T). Let T represent the output probability of the i-th class of student model s, where T>0, and let T be the temperature coefficient.
[0079] In step S60, during the inference matching stage, the similarity between the feature vector of the query image and the feature vectors of each image in the geographic reference image database is calculated using metrics such as cosine distance to achieve the final scene matching. The specific process is as follows:
[0080] The feature vectors of the obtained satellite images and the feature vectors of the UAV images are measured by cosine similarity. The feature vector pair with the closest distance is selected as the match. When performing UAV positioning tasks, the distance between a UAV image and all satellite images is calculated, and the vector pair with the closest distance is selected as the match pair. When performing UAV navigation tasks, the distance between a satellite image and all UAV images is calculated, and the vector pair with the closest distance is selected as the match pair.
[0081] The formula for calculating cosine similarity is:
[0082]
[0083] Where CS represents cosine similarity, f A f represents the eigenvector matrix composed of all UAV images. B f represents the eigenvector matrix composed of all satellite images. Ai f represents the feature vector of the i-th drone image, i.e., the i-th scene. Bi This represents the feature vector of the i-th UAV map. Let L represent the L2 norm of the vector, and N represent the total number of scenarios.
[0084] This invention addresses the technical problem that existing cross-view matching methods are not applicable to cross-view image matching between UAV and satellite perspectives. Based on the similarity between UAV and satellite images, it utilizes a pre-trained model to extract depth features to achieve mutual matching between satellite and UAV perspective images. By employing a feature rotation encoding strategy and knowledge distillation, the overall efficiency is improved, enabling UAVs to perform positioning and navigation at low altitudes. This invention's cross-view scene matching model, through the introduction of an advanced pre-trained backbone network, the design of a feature rotation encoding strategy, and the application of knowledge distillation technology, successfully proposes a novel method for high-precision, high-robustness, and high-efficiency cross-view scene matching.
[0085] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.
Claims
1. A cross-view image scene matching method based on rotation robustness and knowledge distillation, characterized in that, Includes the following steps: Step S10: Establish a scene matching model based on a twin network structure. This model contains two weight-sharing branches, which are used to process satellite view images and UAV view images, respectively. Step S20: Preprocess the input image using the preprocessing layer of the scene matching model; Step S30: Use a modern visual model EVA-02 that has been fully pre-trained on a large dataset as the backbone network for feature extraction; Step S40: Introduce a feature rotation coding strategy to eliminate the negative impact of image rotation; Step S50: Apply knowledge distillation technique to compress the model to achieve efficient reasoning; In step S60, during the inference matching stage, the similarity between the feature vector of the query image and the feature vector of each image in the geographic reference image library is calculated using metrics such as cosine distance, so as to achieve the final scene matching.
2. The cross-view image scene matching method based on rotation robustness and knowledge distillation as described in claim 1, characterized in that, In step S10, the scene matching model includes a preprocessing layer, a feature extractor, and a matching layer. The feature extractor uses a pre-trained EVA-02 model as the backbone network for feature extraction.
3. The cross-view image scene matching method based on rotation robustness and knowledge distillation as described in claim 2, characterized in that, In step S20, the input image is preprocessed using the preprocessing layer of the scene matching model. The specific process is as follows: The image size is standardized, and image enhancement operations such as random slicing and random horizontal flipping are applied. The random slicing operation involves cropping the image at random locations according to a set size, while the random horizontal flipping operation involves horizontally flipping the image with a given probability.
4. The cross-view image scene matching method based on rotation robustness and knowledge distillation as described in claim 1, characterized in that, In step S30, the preprocessed image is input into the pre-trained EVA-02 model feature extractor for feature extraction. The specific process is as follows: During the training phase, the vector output by the feature extractor is mapped to the number of target scenes to be classified through the neurons of the fully connected layer in the classification layer, and the output value is normalized through the softmax layer. The weight parameters obtained by the scene matching model during training are optimized based on the joint metric loss function of the cross-entropy loss function (CE) and the loss function infoNCE. The formula for calculating the cross-entropy loss function is as follows: in, y represents the predicted value from the scene matching model, and y represents the label value. Indicates predicted value The predicted probability of being identified as class y. Indicates the column direction of the label. The column vector represents the sample, where i represents the index of the total number of scenes, j represents the j-th vector output by the scene matching model, c represents the c-th sample in this round of training, and C represents the number of training samples. The formula for calculating the infoNCE loss function is as follows: Wherein, given a positive sample pair and a set of negative samples τ represents the sharpness coefficient used to control the distribution of the data, sim(q,k) represents the distance between samples, N represents the batch size, K represents the total number of candidate samples (including positive samples and all other negative samples), and q i This represents the i-th query vector. Let k represent the i-th corresponding positive sample. j Let j represent the j-th candidate sample (which may include both positive and negative samples).
5. The cross-view image scene matching method based on rotation robustness and knowledge distillation as described in claim 1, characterized in that, In step S40, a feature rotation encoding strategy is introduced to eliminate the negative impact of image rotation on matching. The specific process is as follows: During the inference (testing) phase, to enhance the model's robustness to rotational changes, a feature rotation encoding strategy is adopted. A single input image is transformed to generate two transformed images. The original image and the transformed image are then fed into the pre-trained model to extract their respective feature vectors. The resulting multiple sets of feature vectors are superimposed or averaged to fuse them into a final, more robust feature representation that contains rotational information.
6. The cross-view image scene matching method based on rotation robustness and knowledge distillation as described in claim 1, characterized in that, In step S50, knowledge distillation technique is applied to compress the model to achieve efficient reasoning. The specific process is as follows: Use a well-trained, high-performance model with a large number of parameters as the teacher model and freeze its parameters; A lightweight network model is selected as the student model. The KL divergence loss is used to guide the student model to learn the output distribution of the teacher model. At the same time, the student model also learns through its own loss function (cross-entropy + infoNCE loss). This approach significantly reduces the number of parameters while maintaining high matching accuracy. The formula for calculating the KL divergence loss is as follows: in, Let represent the output probability of the i-th class in teacher model t (temperature T). Let T represent the output probability of the i-th class of student model s, where T>0, and let T be the temperature coefficient.
7. The cross-view image scene matching method based on rotation robustness and knowledge distillation as described in claim 1, characterized in that, In step S60, during the inference matching stage, the similarity between the feature vector of the query image and the feature vectors of each image in the geographic reference image database is calculated using metrics such as cosine distance to achieve the final scene matching. The specific process is as follows: The distance between the feature vectors of the obtained satellite image and the feature vectors of the UAV image is measured by cosine similarity. The feature vector pair with the closest distance is selected as the match. When performing UAV positioning task, the distance between a UAV image and all satellite images is calculated, and the vector pair with the closest distance is selected as the match pair. When performing UAV navigation task, the distance between a satellite image and all UAV images is calculated, and the vector pair with the closest distance is selected as the match pair. The formula for calculating the cosine similarity is: Where CS represents cosine similarity, f A f represents the eigenvector matrix composed of all UAV images. B f represents the eigenvector matrix composed of all satellite images. Ai f represents the feature vector of the i-th drone image, i.e., the i-th scene. Bi This represents the feature vector of the i-th UAV map. Let L represent the L2 norm of the vector, and N represent the total number of scenarios.