Cross-view geographic positioning method based on image matching

Abstract

The invention discloses a cross-view-angle geographic positioning method based on image matching, and the method comprises the steps: firstly, constructing an image matching data set containing a low-resolution small-view-field unmanned plane image and a high-resolution satellite image, enabling the data set to cover a plurality of scenes of a plurality of geographic regions, and proposing four evaluation indexes for evaluating the performance; then, scanning the satellite image by adopting a sliding window method to generate a plurality of candidate areas, carrying out feature extraction and matching on the unmanned aerial vehicle image and the candidate areas by utilizing a Siamese network, and carrying out model training by calculating similarity measurement; then, image matching is carried out based on the trained model, and the most similar candidate area is screened out; and finally, determining the accurate position of the unmanned aerial vehicle image in the satellite image according to the optimal matching position, thereby completing a cross-view geographic positioning task. The method effectively improves the positioning precision of the low-resolution unmanned aerial vehicle image in the high-resolution satellite image, and is suitable for the fields of unmanned aerial vehicle geographic positioning and the like.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a cross-viewing angle geographic positioning method based on image matching. Background Art

[0002] Unmanned aerial vehicles (UAVs) have developed rapidly in the past few decades. With their advantages of high mobility, low cost, and flexible flight modes, they have been widely used in many fields such as industrial inspection, remote sensing, mapping, surveying, and rescue, and have played an important role in national defense and military applications. However, in some complex environments (such as GNSS-denied areas), due to the limited communication and perception capabilities of onboard sensors, UAVs cannot accurately perceive the surrounding environment and cannot work normally. Therefore, how to achieve UAV geolocation in the case of GNSS denial has become an urgent problem to be solved.

[0003] Cross-viewpoint geolocation technology based on image matching refers to the use of images of the flight area (drone images) taken by drone image sensors to match images with stored satellite images, and calculate positioning information by combining the matching results with drone flight parameters. This method has the advantages of avoiding error accumulation caused by signal drift and ensuring positioning accuracy and stability. This technology does not rely on external signals (such as GNSS) and can effectively improve the autonomy and reliability of drones in complex environments, especially in disaster response, environmental monitoring and urban planning. It can provide more accurate real-time geographic information.

[0004] However, cross-view geolocation faces many challenges, including how to deal with perspective differences, illumination changes, projection transformations, and image distortion. Existing research mainly focuses on matching drone images and satellite images with similar perspectives, and most of the currently available data sets lack annotation information about the real geographic location of the drone. These problems make it difficult for the models trained by existing technologies to be effectively integrated into drone systems in practical applications, although theoretically feasible, and unable to perform real-world positioning tasks. In summary, there are still many shortcomings in the current technical field. Summary of the invention

[0005] In order to overcome the above-mentioned deficiencies of the prior art, the present invention provides a cross-viewing geolocation method based on image matching.

[0006] The technical solution adopted by the present invention is as follows:

[0007] A cross-view geolocation method based on image matching comprises the following steps:

[0008] S1. Construct a dataset of low-resolution small-field-of-view drone images and high-resolution satellite images. The dataset contains multiple scenes in different geographical areas. In addition, a new evaluation system is proposed to evaluate the robustness and inference speed of the matching model under different positioning accuracies. The mean absolute error (MAE) and mean square error (MSE) are introduced as supplementary evaluation indicators to quantify the deviation between the predicted position and the actual position, and further comprehensively evaluate the performance of the matching model.

[0009] S2. According to the dataset proposed in S1, the satellite image patch position corresponding to the drone image is first calculated. Then, the local image patches of the drone image and the satellite image are input into the shared weight dual-branch Siamese network for feature extraction, and the network is trained by calculating the similarity and using the contrast loss function.

[0010] S3, extract multiple image blocks from the satellite image through a sliding window and input the drone image into the trained CNN network for feature extraction and calculate the distance between the features of the satellite image block and the drone image to evaluate their similarity. Finally, determine the position of the image block with the highest similarity, so as to achieve accurate positioning of the drone.

[0011] Preferably, the process of constructing a data set of low-resolution small-field-of-view drone images and high-resolution satellite images in step S1 includes:

[0012] S11. Construct a data set. Its contents are as follows:

[0013] The dataset includes 256 high-resolution images taken by drones during flight missions, as well as a high-resolution satellite image obtained from Bing Maps that details the drone's flight path.

[0014] Specifically, the present invention uses drone_001, drone_002, ... drone_256 to distinguish drone view images obtained at different locations, including a wide range of scene types, not limited to buildings, but also including roads, grasslands, rivers and composite scenes. In addition, each drone view has corresponding latitude and longitude, GPS altitude, pitch, yaw, roll angle and other posture information.

[0015] S12. Construct four evaluation indicators to evaluate the performance of sliding window-based matching methods in UAV image positioning, which are as follows:

[0016] The mean absolute error is used to measure the average deviation between the predicted position and the true position, reflecting the size of the overall positioning error. The mean square error is used to evaluate the square average of the prediction error. It is more sensitive to large errors and can better reflect the stability of positioning performance. The specific definitions of MAE and MSE are as follows:

[0017]

[0018]

[0019] Where N is the total number of test samples, d i It represents the Euclidean distance between the predicted position and the true position of the i-th sample, and the calculation formula is:

[0020]

[0021] MAE directly reflects the average deviation between the predicted position and the true position. A lower MAE indicates better overall positioning performance. Compared to MAE, the square operation of MSE will amplify larger errors, so it can provide higher sensitivity to extreme cases when analyzing model performance. MAE is in kilometers, and MSE is calculated and reported in square kilometers.

[0022] In addition, positioning accuracy is used to evaluate the proportion of samples whose predicted position and true position error is less than the threshold under a given distance threshold. This indicator can reflect the applicability of the system under different accuracy requirements. The definition is as follows:

[0023]

[0024] Where M represents the positioning error d i The number of samples less than the threshold T; N is the total number of test samples; T is the distance threshold, such as 400m, 600m, and 800m. By setting multiple thresholds (@400m, @600m, and @800m), the system performance can be comprehensively evaluated from the perspective of different accuracy requirements.

[0025] Finally, the inference speed is also used as an evaluation indicator to select a baseline model with relatively fast inference performance, and its inference time is used as the benchmark speed. The benchmark is set to 1.00, and the other inference speeds are recorded as 1.00×T(0 <T<+∞)。

[0026] Preferably, the training model based on the Siamese network described in step S2 has the following structure:

[0027] During the training phase, the network consists of two identical branches, each consisting of a Backbone network, which aims to extract deep features from the input image. During the training process, the input is a pair of images: one is a satellite image and the other is a drone image. The satellite image and the drone image are directly input to the corresponding Backbone network for feature extraction.

[0028] The network calculates similarity by comparing the extracted features, and then optimizes the model by minimizing the loss function. Label information (indicating whether the image pair matches) is used to calculate the loss to guide the model to learn more effective feature representation. During the entire training process, the two Backbone networks share weights, that is, they use the same parameters for feature extraction to ensure the consistency of image matching. The Siamese network is optimized by the contrastive loss function, so that the feature distance of similar image pairs is closer, and the feature distance of dissimilar image pairs is as far as possible. It is defined as:

[0029]

[0030] Where: y i Represents the label of the image pair: y i = 0 is a similar image pair, y i = 1 for non-similar image pairs; d i is the characteristic distance of the i-th sample, and the calculation formula is the Euclidean distance:

[0031]

[0032] Preferably, the process of constructing the UAV geolocation process through the sliding window described in step S3 includes:

[0033] First, multiple image blocks are extracted from the satellite image. This process is achieved through the sliding window technology. Each sliding window selects a local area from the satellite image as the image block to be matched. Then, these image blocks and drone images are input into two Backbone networks for feature extraction.

[0034] Unlike the training phase, the test phase no longer updates the weights. Instead, the distance between the features of the satellite image patch and the drone image is calculated to evaluate their similarity. Finally, the image patch position corresponding to the minimum distance is determined to achieve accurate matching between the satellite image patch and the drone image. The sliding window is based on the principles of geometry and camera calibration, and the size of the window is determined by calculating the camera's orientation and field of view.

[0035] The beneficial effects of the present invention are:

[0036] The cross-view geolocation method based on image matching proposed in the present invention can achieve accurate matching between drone images and satellite images. The constructed dataset of low-resolution small-field-of-view drone images and high-resolution satellite images can cover a variety of geographical scenes, improving the adaptability and generalization ability of the model. By using the Siamese network for feature extraction and matching, the similarity between image blocks can be stably and accurately calculated, thereby achieving high-precision geolocation. At the same time, the introduction of sliding window technology and a new evaluation system further improves the robustness and reasoning speed of positioning, ensuring the efficiency of the method under different accuracy requirements. Finally, through multi-dimensional evaluation indicators (such as MAE, MSE and positioning accuracy), the present invention can comprehensively evaluate the performance of the model in various scenarios, ensuring the stability and reliability of the method in practical applications.

[0037] Experimental results and analysis show that the present invention demonstrates its superior performance in the task of UAV geolocation. This method effectively solves the problems existing in the existing model, improves the accuracy of image matching, and is suitable for a variety of application scenarios such as UAV positioning and UAV navigation. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly explain the technical solution of the embodiment of the present invention and make the purpose, characteristics and advantages of the present invention more obvious and easy to understand, the specific implementation methods will be described in detail with reference to the accompanying drawings. It should be noted that the drawings described below are only some exemplary embodiments of the present invention. For ordinary technicians in this field, other drawings and implementation methods can be obtained based on these drawings without creative work.

[0039] Figure 1 is a schematic diagram of the overall architecture of cross-viewing geographical positioning based on image matching according to an embodiment of the present invention;

[0040] Figure 2 is a schematic diagram of a drone image and a satellite image in a geolocation data set according to an embodiment of the present invention;

[0041] Figure 3 1. It is a schematic diagram of a training and testing structure based on a Siamese network according to an embodiment of the present invention;

[0042] Figure 4 Schematic diagram of geographic positioning visualization result according to an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The present invention can be implemented in a variety of ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. In addition, the technical features in each embodiment can be combined accordingly without conflicting with each other.

[0044] According to an embodiment of the present invention, a cross-viewing geographical positioning method based on image matching can be used as follows in practical applications: Figure 1 As shown, the network structure of the present invention is deployed, including the following steps S1 to S3:

[0045] S1. Construct a dataset of low-resolution small-field-of-view drone images and high-resolution satellite images. The dataset contains multiple scenes in different geographical areas. In addition, a new evaluation system is proposed to evaluate the robustness and inference speed of the matching model under different positioning accuracies. The mean absolute error (MAE) and mean square error (MSE) are introduced as supplementary evaluation indicators to quantify the deviation between the predicted position and the actual position, and further comprehensively evaluate the performance of the matching model.

[0046] S2. According to the dataset proposed in S1, the satellite image patch position corresponding to the drone image is first calculated. Then, the local image patches of the drone image and the satellite image are input into the shared weight dual-branch Siamese network for feature extraction, and the network is trained by calculating the similarity and using the contrast loss function.

[0047] S3, extract multiple image blocks from the satellite image through a sliding window and input the drone image into the trained CNN network for feature extraction and calculate the distance between the features of the satellite image block and the drone image to evaluate their similarity. Finally, determine the position of the image block with the highest similarity, so as to achieve accurate positioning of the drone.

[0048] In order to facilitate understanding of the above technical solutions of the present invention, the above technical solutions of the present invention are described in detail below through actual deployment and application.

[0049] Before implementing the present invention, it is necessary to first construct a set of high-resolution satellite images and low-resolution small-field-of-view drone image datasets. Figure 2 The dataset includes 256 high-definition and low-resolution images taken by drones during flight missions, as well as a high-resolution satellite image obtained from Bing Maps that shows the drone's flight path in detail.

[0050] The present invention uses drone_001, drone_002, ...drone_256 to distinguish drone view images obtained at different locations, including a wide range of scene types, not limited to buildings, but also including roads, grasslands, rivers and composite scenes. In addition, each drone view has corresponding latitude and longitude, GPS altitude, pitch, yaw, roll angle and other posture information.

[0051] In addition, four evaluation indicators are constructed to evaluate the performance of the sliding window-based matching method in UAV image positioning, namely, mean absolute error (MAE), mean square error (MSE), positioning accuracy and inference speed. The contents are as follows:

[0052] The mean absolute error is used to measure the average deviation between the predicted position and the true position, reflecting the size of the overall positioning error. The mean square error is used to evaluate the square average of the prediction error. It is more sensitive to large errors and can better reflect the stability of positioning performance. The specific definitions of MAE and MSE are as follows:

[0053]

[0054] Where N is the total number of test samples, d i It represents the Euclidean distance between the predicted position and the true position of the i-th sample, and the calculation formula is:

[0055]

[0056] MAE directly reflects the average deviation between the predicted position and the true position. A lower MAE indicates better overall positioning performance. Compared to MAE, the square operation of MSE will amplify larger errors, so it can provide higher sensitivity to extreme cases when analyzing model performance. MAE is in kilometers, and MSE is calculated and reported in square kilometers.

[0057] In addition, positioning accuracy is used to evaluate the proportion of samples whose predicted position and true position error is less than the threshold under a given distance threshold. This indicator can reflect the applicability of the system under different accuracy requirements. The definition is as follows:

[0058]

[0059] Where M represents the positioning error d i The number of samples less than the threshold T; N is the total number of test samples; T is the distance threshold, such as 400m, 600m, and 800m. By setting multiple thresholds (@400m, @600m, and @800m), the system performance can be comprehensively evaluated from the perspective of different accuracy requirements.

[0060] Finally, the inference speed is also used as an evaluation indicator to select a baseline model with relatively fast inference performance, and its inference time is used as the benchmark speed. The benchmark is set to 1.00, and the other inference speeds are recorded as 1.00×T(0 <T<+∞)。

[0061] As a specific example, please refer to Figure 3As shown, the cross-view geolocation method of the present invention adopts a Siamese network architecture, and realizes accurate comparison and positioning between drone images and satellite images through image matching technology. The neural network architecture of the method consists of two identical branches, which are mainly used to extract deep features from the input image. Each branch contains a Backbone network, and the network structure is designed to extract the features of two input images (i.e., satellite images and drone images) by sharing weights. This design can ensure that the matching process between the two images is consistent and effective.

[0062] During the training phase, the network input is a pair of images: a satellite image and a corresponding drone image. These images are input into two weighted Backbone networks. After feature extraction, the extracted features are used to calculate the similarity between the image pairs. In this process, the network optimizes the model by minimizing the loss function based on the calculated similarity. In order to guide the model to better learn feature representation, the training process also introduces label information, which is used to indicate whether the image pairs match. The Siamese network is optimized by the contrastive loss function so that the feature distances of similar image pairs are as close as possible, while the feature distances of dissimilar image pairs are as far apart as possible.

[0063] Specifically, the contrast loss function is as follows:

[0064]

[0065] Where: y i Represents the label of the image pair: y i = 0 is a similar image pair, y i = 1 for non-similar image pairs; d i is the characteristic distance of the i-th sample, and the calculation formula is the Euclidean distance:

[0066]

[0067] Through this loss function, the network is continuously optimized during the training process, so that the feature distance of similar image pairs gradually decreases, while the feature distance of dissimilar image pairs increases, and finally the best matching effect is achieved.

[0068] In the test phase, the difference from the training phase is that the test process no longer updates the weights, but evaluates their similarity by calculating the feature distance between the satellite image blocks and the drone images. The input of the test phase is the satellite image and the drone image. Specifically, a sliding window is first used to extract multiple image blocks from the satellite image, and each sliding window selects a local area in the satellite image as the image block to be matched. Next, these satellite image blocks and the drone image are simultaneously input into the trained CNN network. After feature extraction, the feature distance between the satellite image blocks and the drone image is calculated to evaluate their similarity.

[0069] By calculating the distance value, the network can determine which satellite image blocks have a higher similarity with the drone image. Finally, the network determines the location of the most matching satellite image block based on the principle of minimum similarity, thereby achieving accurate positioning between the drone image and the satellite image.

[0070] The sliding window technology in this process is a key step in achieving image block matching. Through the sliding window, multiple image blocks are extracted from the satellite image and compared with the drone image, which greatly improves the matching efficiency and positioning accuracy. The size and step size of the sliding window are usually determined based on geometric principles and camera calibration information. By calculating the camera's orientation, field of view, and related geometric transformations, the size and movement of the window can be accurately adjusted so that each slide can select the most suitable image block for matching.

[0071] During the entire testing process, since the network has learned how to extract effective features and match them in the training phase, the calculations in the testing phase only need to perform feature extraction and distance calculation. This enables the system to quickly perform geolocation in practical applications with high real-time performance. Unlike the backpropagation and weight update in the training phase, the main task of the testing phase is to find the area that best matches the drone image from a large number of satellite image blocks, thereby achieving accurate positioning.

[0072] In order to ensure the accuracy and robustness of positioning, a shared weight mechanism is used in the network structure design, so that satellite images and drone images use the same parameters when extracting features, which can ensure the consistency of feature extraction and thus improve the stability of matching. In addition, the advantage of using the Siamese network architecture is that it can directly determine whether they match by calculating the similarity between image pairs without relying on traditional image classification or regression methods.

[0073] In order to better illustrate the performance of the cross-view geolocation method based on image matching provided by the present invention, the results presented on the geolocation dataset of the present invention are as follows:

[0074] This model evaluates the results of different ACC@T, MSE, and MAE indicators by comparing the performance of different sliding window sizes on the dataset:

[0075]

[0076]

[0077] In summary, with the help of the above technical solutions, the cross-viewpoint geolocation method based on image matching proposed in the present invention firstly improves the adaptability and generalization ability of the model by constructing a data set of low-resolution small-field-of-view drone images and high-resolution satellite images. Then, a double-branch Siamese network with shared weights is used to successfully solve the problems of perspective difference, illumination change, etc. through precise feature extraction and contrast loss function, and efficient matching between images is achieved. In addition, in order to further improve the positioning accuracy, a sliding window technology is designed to make the matching process of satellite images and drone images more detailed and accurate. Finally, through multi-dimensional evaluation indicators (such as MAE, MSE and positioning accuracy), the method of the present invention shows excellent performance in positioning tasks and has strong practical application potential, especially in the fields of drone navigation and environmental monitoring. It has broad application prospects.

[0078] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A cross-view geolocation method based on image matching, characterized in that: The following steps are involved: S1. Construct a dataset of low-resolution small-field-of-view drone images and high-resolution satellite images; the dataset contains multiple scenes in different geographical areas; an evaluation system is also proposed to evaluate the robustness and inference speed of the matching model under different positioning accuracies, and the mean absolute error (MAE) and mean square error (MSE) are introduced as supplementary evaluation indicators to quantify the deviation between the predicted position and the actual position, and comprehensively evaluate the matching model performance; S2. According to the dataset proposed in S1, the location of the satellite image patch corresponding to the drone image is first calculated; then, the local image patches of the drone image and the satellite image are input into a weight-sharing dual-branch Siamese network for feature extraction, and the network is trained by calculating the similarity and using the contrast loss function; S3. Extract multiple image blocks from the satellite image through a sliding window and input the UAV image into the trained CNN network for feature extraction and calculate the distance between the features of the satellite image block and the UAV image to evaluate their similarity; finally, determine the position of the image block corresponding to the highest similarity, so as to achieve accurate positioning of the UAV.

2. The cross-viewing geographic positioning method based on image matching according to claim 1, characterized in that: The specific process of constructing the data set in step S1 is: S11, build a data set; The dataset includes 256 high-resolution images taken by drones and one high-resolution satellite image of the drone’s flight path obtained from Bing Maps; drone_001, drone_002, ...drone_256 are used to distinguish drone view images obtained at different locations, including buildings, roads, grasslands, rivers, and composite scenes; in addition, each drone view has corresponding latitude and longitude, GPS altitude, pitch, yaw, and roll angle posture information; S12. Construct four evaluation indicators to evaluate the performance of sliding window-based matching methods in UAV image positioning, which are as follows: The mean absolute error is used to measure the average deviation between the predicted position and the true position, reflecting the size of the overall positioning error; the mean square error is used to evaluate the square average of the prediction error, which is more sensitive to large errors and can better reflect the stability of positioning performance; MAE and MSE are specifically defined as follows: Where N is the total number of test samples, d i Indicates the predicted position of the i-th sample (x predicted ,y predicted ) and the true position (x groundtruth ,y groundtruth ), the Euclidean distance between them is calculated as: MAE directly reflects the average deviation between the predicted position and the true position. A lower MAE indicates better overall positioning performance. Positioning accuracy is used to evaluate the proportion of samples whose error between the predicted position and the actual position is less than the threshold under a given distance threshold. This indicator reflects the applicability of the system under different accuracy requirements. It is defined as follows: Where M represents the positioning error d i The number of samples less than the threshold T; N is the total number of test samples; T is the distance threshold; Finally, the inference speed is also used as an evaluation indicator to select a baseline model with relatively fast inference performance, and its inference time is used as the benchmark speed; the benchmark is set to 1.00, and the other inference speeds are recorded as 1.00×T,0 <T<+∞。 3. The cross-viewing geolocation method based on image matching according to claim 1, characterized in that: In step S2, the specific structure of the training model based on the Siamese network is: During the training phase, the network consists of two identical branches, each of which is composed of a Backbone network, which aims to extract deep features from the input image; During the training process, the input is a pair of images: one is a satellite image and the other is a drone image. The satellite image and the drone image are directly input into the corresponding Backbone network for feature extraction. The network calculates similarity by comparing the extracted features, and then optimizes the model by minimizing the loss function; the label information is used to calculate the loss to guide the model to learn more effective feature representation; during the entire training process, the two Backbone networks share weights, that is, use the same parameters for feature extraction to ensure the consistency of image matching; the Siamese network is optimized by comparing the loss function, so that the feature distance of similar image pairs is closer, and the feature distance of dissimilar image pairs is as far as possible; it is defined as: Where: y i Represents the label of the image pair: y i = 0 is a similar image pair, y i = 1 for non-similar image pairs; d i is the characteristic distance of the i-th sample, and the calculation formula is the Euclidean distance:

4. The cross-viewing geolocation method based on image matching according to claim 1, characterized in that: The specific process of the sliding window technology in step S3 is as follows: First, multiple image blocks are extracted from the satellite image through a sliding window. Each sliding window selects a local area from the satellite image as the image block to be matched. Then, these image blocks and the drone image are input into two Backbone networks for feature extraction. Unlike the training phase, the test phase does not update the weights. Instead, the distance between the features of the satellite image blocks and the drone images is calculated to evaluate their similarity. Finally, the image block position corresponding to the minimum distance is determined to achieve accurate matching of the satellite image blocks and the drone images. The sliding window is based on geometry and camera calibration principles, and the size of the window is determined by calculating the camera's orientation and field of view.

Images